Reducing E-Waste: Reverse engineering the Lenovo EMC PX4-300D NAS firmware and kernel drivers

#electronic waste #refurbished #reverse engineering #linux #kernel development #lenovo emc #px4-300d #nas #environmentalism

Reducing electronic waste is one of my main priorities. Making use of my devices as long as possible, upgrading, servicing as well as repairing them myself are some of the efforts i take to reduce my electronic waste footprint. However, despite my efforts sometimes its nearly impossible. The following blog article will take you on a journey of myself reverse engineering the firmware of one of my old NAS devices to be able to use it securely again.

Introduction

 

 

The Lenovo EMC PX4-300D is a small, but powerful NAS system.
It features the following specs:

Component Specification
CPU Intel Atom D525 (2 Cores, 4 Threads)
RAM 2 GB DDR3 SODIMM (upgradeable)
Network Dual Realtek RTL8168 GbE NICs
Software Debian based, Customized by Lenovo/Iomega
Storage Four 3.5" hot-swap SATA bays, front accessible
Misc Monochrome front display for status display


Matched with four 2TB disks, its got enough performance and space for my backup needs. However, there is one big problem. The NAS was initially released in January 2012 and has not seen any os updates since a long time. Currently the NAS runs a customized, Linux 3.x Kernel. A secure operation of the device is therefore not possible anymore. To fix this i began to try installing a newer linux operating system. Fortunately, the NAS is based on an x86 architecture as well as 64bit compatible so this should not be very hard.

Getting to know whats inside the box

After dismantling the device, i noticed a few things:

  • The mainboard of the device exposes a standard serial interface with a four pin header
  • The front IO USB3.0 is connected through a usb extension cable, directly to a USB A female connector on the board
  • The Mainboard also features a PCIe slot for expansion (e.g. network interfaces or SCSI cards)

To get an idea on which pins of the serial interface are VCC, GND, TX, RX I pulled out my mulitmeter and started measuring. The following table illustrates the pin layout. The pin numbers are numbered from left to right matching the direction of the "SERIAL_B" description label on the board. (Pin #1 above "S"). You can refer to the cable colors if you are using a PL2303TA USB to TTL cable like mine: Amazon affiliate link

Pin number Description, Cable color
1 VCC (5V, Red)
2 RX (White)
3 GND (Black)
4 TX (Green)


To connect to the serial interface I used PuTTY: a free SSH and Telnet client (greenend.org.uk), a baud rate of 115200, 8 parity bits, 1 stop bit and no parity or flow control. Once the connection was open using PuTTY, I powered on the device. Soon, I was able to see the bios post messages appearing on the terminal, prompting to press F11 for a boot device selection menu, or DEL to enter the bios setup. The interface was similar to other bios setup menus, and allowed me to change the boot order as well as many other settings of the device. Great.

Preparing a live USB stick with a console output using serial

To prepare a live USB stick I installed a minimal debian on a usb stick, plugged in to my computer and making sure its installed in legacy BIOS/MBR mode as the hardware in the NAS does not support UEFI.
Once the installation is finished, I needed to make the console output available using the serial interface. This can be done easily by editing the default grub configuration, found in /etc/default/grub by adding 

GRUB_CMDLINE_LINUX="console=ttyS0,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

to the file. The full file can be found below:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=" usbcore.old_scheme_first=1"
GRUB_CMDLINE_LINUX="console=ttyS0,115200n8"

 

GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

Reverse engineering the hardware and its drivers

Once the USB stick was ready, I plugged it into one of the two ports on the back of the device, powered it on and selected the USB stick using the boot device selection menu. It took a while to boot as the flashdrive is not as fast as a ssd drive, but after a few minutes I was greeted with a login prompt of the Debian OS installed on the flashdrive. Now it was time to investigate further into the hardware of the device. It turned out, that there was another flashdrive, probably directly soldered onto the motherboard, containing the operating system of the NAS. To make sure i can always restore the device to its original glory, I created a 1:1 copy of its' contents using the dd command. Funfact: I was able to get the image to boot in a virtual machine on my proxmox server, just for fun as it does not have any real world use case. Further than that, investigating using the live os had no more information to offer. Therefore I rebooted the device and tried investigating using the original firmware, which fortunately allowed me to login as the root user using the diagnostic features (the link is specific to the small sibling of the px4-300d, but applies to it as well.

Digging deeper, I found that the display was a UMS8485MD, monochrome 128x64 pixels display, controlled via a lcm device at /dev/lcm. Moreover, the two buttons at the front, right to the display, were exposed using a standard input device at /dev/input/by-path/platform-asus_laptop-event allowing me to react to the button presses via scripts and programs. Furthermore, the front leds indicating warning and errors, are controllable using the ich_gpio interface of the Intel Atom processor.

Now i had almost everything i needed to create a custom, fully functional firmware using a recent linux kernel version as well as a recent Debian operating system.

Getting the drivers needed

I now knew which devices were available and which ones need special drivers. By searching the internet, GitHub and many other places, I finally stumbled across a github repository by d235j (David Ryskalczyk) (github.com) who found the drivers needed in the fosskit published by lenovo itself and created out of tree kernel modules. See the repository here: d235j/lenovoEMC-300d (github.com)
Furthermore the only issue of the repository mentioned the download links to the fosskits of the devices firmware. All i had to do now was implementing the drivers into a recent kernel source.

Integrating the drivers into the linux kernel

To integrate the drivers for the display, the sensors and buttons into the linux mainline kernel, I first had to get the kernel sources. I decided to go with the current 5.18 linux kernel sources distributed bv the Debian package repositories. By doing so, I could make sure, the current configuration of the Linux 5.4 kernel - which i know boots and provides serial output on this device - can be reused easily. To download and extract the Linux 5.18 kernel source I simply ran the following commands:

sudo apt install linux-source-5.18
tar xavf /usr/src/linux-source-5.18.tar.xz -C .
cd linux-source-5.18

The download as well as the extraction of the archive may take a while. Afterwards I created a new git repository in the directory of the kernel source to be able to track the changes I made to the kernel source. The process of integrating a driver to the kernel sources can be very complex, depending on the driver and its dependencies. In this case it has been fairly simple. Here is a quick summary of my changes:

  • Create a new directory for the ums8485md driver in the kernel source tree at drivers/misc
  • Copy all driver source files into the newly created directory
  • Grab a Makefile and KConfig file from another driver and modify it to match the current driver
  • Add the driver to the nescessary KConfig and Makefile files in drivers/misc
  • Change the logo that will be displayed once the driver is loaded
  • Add a procfs entry to allow writing and reading from the framebuffer directly as well as resetting/refreshing the display

I have published the kernel and the integrated driver to my GitHub repository: mKenfenheuer/linux-image-5.18-px4-300d (github.com)

Now the driver has been integrated into the kernel sources and is ready to be compiled. To compile the kernel, I copied the current kernel source of the default Debian kernel from /boot/config-5.4 to the current directory as .config and ran make oldconfig to check for changed configuration options of the kernel. Once done I made sure that the driver as well as its dependency, the gpio_ich module for the Intel Atom GPIO interface, were enabled using make menuconfig. Now, the configuration was almost ready. To make sure the kernel will compile without errors, I also removed the reference to the kernels trusted system keys by setting CONFIG_SYSTEM_TRUSTED_KEYS = "" to an empty string. To compile ther kernel source into a .deb package which can be easily installed I had to run the following command:

nice make -j`nproc` bindeb-pkg

After a few minutes, the kernel was compiled successfully. I copied the sources onto the NAS and installed the packages via dpkg and rebooted the device. After the reboot I was able to see the driver loaded successfully in the kernel output via dmesg.

[ 17.092900] ums8485md: lcm driver is registered.

From now on, I was able to write and read to and from the display framebuffer by using the procfs files located in /proc/ums8485md/. Now I only had to write a quick and dirty program which will display the current status of the device on the display. As this is not in scope of the blog article, I will not go into details here, but you can ofcourse find the source-code as always in my GitHub repository: mKenfenheuer/lenovo-px4px6-displayd (github.com)

With the updated kernel, a recent Debian 11 buster and the display, buttons and leds working all hardware functionalities have been restored. To be able to easily manage the nas again, I decided to install openmediavault.

 

Here is a video of the display in action: px4-300d-displayd-demo.mp4

About

Hey, nice to finally meet you! My name is Maximilian, welcome to my blog!