UCS. Split-transaction-stop errors on B200-M3, VIC 1240, blade

[David Duwe]

MEZZ\obfl\syslog:  Log completely full of hundreds of the following messages:

    151214-07:02:16.294414 kernel: [2131724.390000] paloi2c paloi2c.0: target 0x70: split-transaction-stop error

For every dozen or so of the above messages, one of the following is received:

    151214-07:10:16.719172 bootoptd.i2carb ERROR: Failed to acquire I2C bus (err -2)

Frequency of messages vary with CPU load.  When load is heaviest, messages run about a dozen per minute.  CPU load hasn't exceeded 40% in at least a month.  A log excerpt is attached.

Chassis contains 7 other B200-M3 blades with VIC 1240 NICs.  All run ESXi 5.5.  All operate under very similar loads, hosting XenApp virtual servers.  Other servers show similar CPU and Memory usage patterns and percentages.  None of the other 7 show these messages.  All blades are at identical firmware levels, 2.2(5b).  Infrastructure is at 2.2(5b).

Chassis Power Supplies have been replaced as per TAC suggestion.  Blade has been shutdown/restarted.  Neither was any help.

So far the blade is running ESXi and its virtual guests without any noticeable symptoms.

Relevant bugs:   CSCux21100, CSCul00037

Current TAC status:  researching

Has anyone else seen these messages?  Are they benign or not?  Does anyone have insight as to their cause and eradication?

[Kirk J]

Greetings.

Looking at the related bug notes I don't see where there's any kind of functional impact, so it appears to be benign aside from extra chat in your logs.

For CSCux21100, that was recently filed for another system running same firmware as you.  For that bug, there is only one customer case attached, so doesn't look like a common issue.  You may want to open a TAC case if you have time and upload the Chassis tech support, so your case can be associated with CSCux21100.

Currently do not have a resolution.

Thanks,

Kirk..

[David Duwe]

Thank you for the prompt and helpful response.

I haven't seen any discernable performance or functionality issues with this blade, so I agree that the Split Transaction messages are probably benign.  The extra chat is somewhat problematic, however, in that the entire syslog file is full of them to the exclusion of all other messages that might be important.  I would still like to be rid of them.  Maybe re-install ESXi?  Maybe replace mezzanine card?  Any other ideas?

CSCux21100 was opened by TAC on my behalf (SR636860159).  I am the one customer case attached. 

Thanks again,   David ..... .. . . .... . ..

[David Duwe]

Good news.  I am no longer seeing the syslog file full of split-transaction-stop errors.  The steps taken were: 

1)Update the CIMC of the blade from 2.2(5b) to 2.2(6e).  This was supported by TAC even though the infrastructure remained at 2.2(5b).  The update to 2.2(6e) was actually done to address CSCuv51214 which was causing the CIMC\var\log\messages file to be filled with "Palo in boot block" errors.  The update worked, the "boot block" errors were drastically diminished, but the "split-transaction-stop" errors remained.  However, this allowed TAC to do further diagnosis. 

2)Reseat the blade.  Having reviewed the cleaner logs, TAC recommended the reseat.  I cleared the blade of VMs and decommissioned it.  After the reseat, I brought it back up, returned the VM load and, for the first time in months, did not see one "split-transaction-stop" error.

[David Duwe]

Whoops.  I spoke too soon.  Upon further review, the load was not quite the same as before until the morning progressed and our XenApp VMs spun up to handle the business day.  The split-transaction-stop errors returned.   TAC is consulting with VIC team now.

[David Duwe]

Upon advice from TAC, I replaced the vNIC card (VIC 1240).  This has resulted in syslog being free of split-transaction-stop errors for two days now.  The VM load is normal. 

It appears we were dealing with a bad card all along.