B200-M4 reboot after autoinstall infrastructure from 2.2.3 to 2.2.6

Walter Dey · ‎02-07-2016

I post this to warn you of a nasty bug, which I didn't know and which seems to be well known (> 100 TAC cases)

One of my customer wanted to firmware upgrade from 2.2.3 to 2.2.6e and brought down his whole environment (Super Gau)

All the B200 M4 blades have rebooted, and went into deep check point instated a shallow discover while upgrade, this the cause of these reboot

BUG: CSCus42584

https://tools.cisco.com/bugsearch/bug/CSCus42584/?reffering_site=dumpcr

Any operation that triggers a shallow discovery could transition to a deep discovery if the Vendor/Model information is missing for the blade's MRAID12G controller.

Common operations that will trigger a shallow discovery:
CIMC Reset
UCS Manager Upgrade
UCS Manager Cluster lead operation
IOM Reboot/Reset
IOM Offline Event due to cable disconnect/reconnect
Fabric Interconnect Reboot

Last Modified:

Feb 5,2016

Status:

Fixed

Severity:

3 Moderate

Product:

Cisco Unified Computing System

Support Cases:

102

Known Affected Releases:

(3)

2.2(1b)A

2.2(2c)A

2.2(3)

Known Fixed Releases:

(5)

2.2(3g)

2.2(4b)

2.2(5a)

3.1(1e)

3.1(1e)T

Lukas Mazur · ‎02-11-2016

Hi Walter,

a second very similar bug is also there - https://tools.cisco.com/bugsearch/bug/CSCut61527

Symptom:
B200 M4 rebooted unexpectedly

A deep discovery is observed from the UCSM tech-support, rather than the intended shallow discovery

Conditions:
Any operation that triggers a shallow discovery could transition to a deep discovery if the Serial number of the blade's MRAID12G RAID controller is blank.

Even after the unexpected reboot and deep discovery, UCSM "show server inventory expand" output will show a blank SN for the MRAID12G RAID controller:
RAID Controller 1:
Type: SAS
Vendor: Cisco Systems Inc
Model: UCSB-MRAID12G
Serial:
HW Revision: C0
PCI Addr: 01:00.0
Raid Support: RAID0, RAID1
OOB Interface Supported: Yes
Rebuild Rate: 30
Controller Status: Optimal

Example operations that will trigger a shallow discovery:
CIMC Reset
UCS Manager Upgrade
UCS Manager Cluster lead operation
IOM Reboot/Reset
IOM Offline Event due to cable disconnect/reconnect
Fabric Interconnect Reboot

experienced your bug aswell some time ago :/ - there seem to be some stuff going on with the SAS controllers in the B200 M4 ... had some issues lately

BR Lukas

Mic Dykstra · ‎08-02-2016

Does anyone know if the vendor and model are showing in UCSM and when using the powershell commands, if it could 'disappear' again? ie, if this bug is suspect as the cause of an unplanned reboot when a shallow discovery was triggered and then transitioned to deep, now that I see a vendor and model, could it happen again?