Using a modern 9500-series Broadcom HBA with an old Supermicro Expander Backplane may result in instability. This appears to be fixed in the new(er) backplane firmware, however I found the update process to be a bit non-trivial: Instructions in the firmware Release Notes are designed for factory use, and directly following them will result in USC information getting lost, which is not ideal.

This post will describe the problem, and offer step-by step solution, including specific firmware update instructions suggested by Supermicro support.

Similar approach is described in Art Of The Server’s video on the same topic1; this post aims to augment that, and provide specific commands, that ended up being different from what the video describes. This Supermicro FAQ2 touches on the same topic.

The problem

Single-expander backplane BPN-SAS3-826-EL1-N43 in CSE-826 chassis driven by with AOC-S3008L-L8e HBA in IT mode worked flawlessly for years. As part of idle power consumption improvement effort, I have attempted to replace it with one built on a more modern and efficient chipset, Broadcom 9501-8i (SAS3808), expecting to reduce power consumption from 11W to 4W.

Booting with it immediately resulted in an endless spew of errors on every disk, (resulting in data corruptions on the pool, which were later rectified by scrub) that looked like this:

...
(da1:mpr0:0:17:0): READ(6). CDB: 08 00 00 81 01 00
(da1:mpr0:0:17:0): CAM status: CCB request completed with an error
(da1:mpr0:0:17:0): Retrying command, 3 more tries remain
(da1:mpr0:0:17:0): READ(6). CDB: 08 00 00 81 01 00
(da1:mpr0:0:17:0): CAM status: SCSI Status Error
(da1:mpr0:0:17:0): SCSI status: Check Condition
(da1:mpr0:0:17:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da1:mpr0:0:17:0): Retrying command (per sense data)
mpr0: Controller reported scsi ioc terminated tgt 17 SMID 1284 loginfo 3112010c
(da1:mpr0:0:17:0): READ(10). CDB: 28 00 02 00 00 7e 00 00 01 00
(da1:mpr0:0:17:0): CAM status: CCB request completed with an error
(da1:mpr0:0:17:0): Retrying command, 3 more tries remain
(da1:mpr0:0:17:0): READ(10). CDB: 28 00 02 00 00 7e 00 00 01 00
(da1:mpr0:0:17:0): CAM status: SCSI Status Error
(da1:mpr0:0:17:0): SCSI status: Check Condition
(da1:mpr0:0:17:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da1:mpr0:0:17:0): Retrying command (per sense data)
mpr0: Controller reported scsi ioc terminated tgt 27 SMID 1455 loginfo 3112010c
...

Triage

Replacing cables, an HBA, and updating HBA firmware did not help, so the backplane firmware became next logical suspect. Supermicro support provided me with a package containing the binaries and flashing instructions.

That’s where it got complicated.

The instructions described the flashing procedure using the CLIXTL tool with steps to update all three banks with firmware and then the bank with MFG. However, after reading the documentation for the said tool, it became apparent that following the procedure from release notes will blow off the Unit Specific Configuration (USC), such as SCSI ID, which is quite undesirable. These instructions seem to be written for factory to perform initial flashing, followed by programming of the configuration – and this is absolutely not the right thing to do when we just want to update firmware on the existing already provisioned unit.

The manual also contains the warning against updating the bootloader (residing in bank 0) unless absolutely necessary to avoid risking bricking the device if something was to go wrong, but considering the sheer amount of changes in the firmware, there is a possibility that new firmware would have required a new bootloader anyway:

Flash region 0 is the boot region, and it should be only programmed with initial shipping FW version. Please contact Supermicro window before updating flash region 0.

Obtaining the bits

Obtain firmware from Supermicro Support, obtain the CLIXTL4 tool from https://www.supermicro.com/wdl/utility/ExpanderXtools_Lite/Linux/. At the time of writing, the version was 7.10.C.

Preparing the tools

Unpack everything, and place the tool, and binaries from firmware package’s update subfolder somewhere together, e.g. into a new folder called flash.

CLIXTL is a linux tool, so we’ll need a bootable linux of some sort. I downloaded Ubuntu Live Server, but any shall do. Flash it to USB drive with Etcher, or mount with IPMI directly to the server. Boot from it, when it gets to installer, press “Alt+F2” to get to local console, and set the ubuntu-server’s user password, so you can do the rest from the comfort of your favourite terminal.

Then copy the prepared folder to the booted linux instance (e.g. scp -r flash ubuntu-server@10.0.17.248:) and ssh into it.

Examining configuration

Find out address of each expander chip:

$ sudo ./CLIXTL -l

    PRODUCT ID          SAS ADDRESS             DEVICE NAME
01) SAS3x28             500304801F4AE2FF        /dev/sg13

Dump the information for the one you want to update:

$ sudo ./CLIXTL -i -t 500304801F4AE2FF

UNIT SPECIFIC INFORMATION:
    SAS ADDRESS    - 500304801F4AE2FF
    ENCLOSURE ID   - 500304801F4AE2FF

ENCLOSURE INFORMATION:
    PLATFORM NAME  - SMC826ELSAS3N4P
    SERIAL NUMBER  -
    VENDOR ID      - LSI
    PRODUCT ID     - SAS3x28

VERSION INFORMATION:
    FLASH REGION 0 - 66.06.01.05
    FLASH REGION 1 - 66.06.01.05
    FLASH REGION 2 - 66.06.01.05
    FLASH REGION 3 - 06.05

DEVICE INFORMATION:
    DEVICE NAME    - /dev/sg13
    BMC IP         - NULL

Save that configuration. Just in case.

Updating firmware

Here we will flash the firmware into all three regions. Note the -f all parameter. To skip updating bootloader use -f std.

From the manual:

-f              Flash expander firmware into expander flash
<argument>      0~2–download .fw file to firmware region 0~2
                3 –download .bin file to firmware region 3
                std-download .fw file to region 1 & 2 only
                all–download .fw file to all firmware region

Also note the format of the path to the firmware: it must be a full path or filter to the firmware file.

$ sudo ./CLIXTL -f all -t 500304801F4AE2FF -d /home/ubuntu-server/flash/*.fw
 
Firmware Region 0 - Finished

Firmware Region 1 - Finished

Firmware Region 2 - Finished

Please reset expander to activate once all updates finished

Don’t reset yet.

Updating MFG block

Note the -c std flag that is missing from the instructions in Release Notes. From the 3.2 EXPANDER MANUFACTURING IMAGE DOWNLOAD AND CARRY USC section of the Command-Line Xtool document:

-c              Flash expander manufacturing page and inherit unit-
                specific configuration including current SAS address,
                ELID, vendor ID and product ID
<argument>      std–inherits USC from previous manufacturing image
                nvp–inherits without vendor ID and product ID
                foc–same as argument “std” but no platform check

This flag needs to be passed to preserve the USC:

$ sudo ./CLIXTL -c std -t 500304801F4AE2FF -d /home/ubuntu-server/flash/*.bin

MFG page Region 3 - Finished

New configuration is uploaded successfully

Please reset expander to activate once all updates finished

If all steps succeeded – power cycle the server. If any of the steps failed – do not reset anything, contact Supermicro Support: if you reboot, the backplane may not boot.

TLDR

./CLIXTL -l
./CLIXTL -i -t 500304801F4AE2FF
./CLIXTL -f all -t 500304801F4AE2FF -d /home/ubuntu-server/flash/*.fw
./CLIXTL -c std -t 500304801F4AE2FF -d /home/ubuntu-server/flash/*.bin

References

  1. How to not brick your Supermicro SAS-3 backplane firmware updates by Art of the Server ↩

  2. Supermicro FAQ entry: https://www.supermicro.com/support/faqs/faq.cfm?faq=25202 ↩

  3. BPN-SAS3-826EL1-N4 manual: https://www.supermicro.com/manuals/other/BPN-SAS3-826EL1-N4.pdf ↩

  4. CLIXTL tool download https://www.supermicro.com/wdl/utility/ExpanderXtools_Lite/Linux/ ↩