Background
Interrupt remapping is a feature implemented in Intel VT-d or AMD IOMMU that provides isolation and compatibility for the Hypervisor to Direct IO to specific CPU through another mapping layer which allows software to control (mainly) the destination and vector for the Interrupts. This feature is enabled by default on ESXi 4.1 and ESXi 5.0. It's known that some vendors architectures including UCS have compatibility problems with Interrupt Remapping Functionality. The Virtual Machine guess OS will randomly loose access to the underlying storage which will compromise the integrity of the file system. VM OS that gets corrupted and report filesystem inconsistencies due to problems with Interrupt Remapping might need to be reinstalled.
It's therefore requried to disable interrupt Remapping in ESXi for UC applications.
Interrupt remapping usually leads to Virtual Machines going into Read only state, and becoming hang or corrupted. A few examples of common problems identified by CISCO TAC include, but are not limited to:
- Virtual Machines not responding.
- Virtual Machines reporting file system inconsistencies.
- Virtual Machines hang.
- You may receive an alerts about "Degraded path for an Unknown Device" in vCenter Server
- Virtual Machines going into read only mode.
- Virtual Machines stuck at shut down or bootup of the VM indefinetly.
Troubleshooting
Common Symptoms found during troubleshooting:
ESXi host might display errors related to SCSI commands not executing succesfully. Other write related commands might report errors.
Virtual Machines might report file system inconsistencies and might request a forced File system check to run.
Vmware support logs from the virtual Machines might report some of the following error messages:
7:2011 Dec 5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:220:peci: transaction failed: [0x61]
7:2011 Dec 5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:190:peci: bad FCS1, received = 0x0, expected = 0xe7
7:2011 Dec 5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:220:peci: transaction failed: [0x61]
Jan 14 22:23:53 vmkernel: 1:04:45:42.202 cpu2:4098)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f2ce640) to NMP device "naa.600605b0030408c014fe4d241d0d8fb9" failed on physical path "vmhba4:C2:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x
Jan 14 22:23:53 vmkernel: 1:04:45:42.202 cpu2:4098)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600605b0030408c014fe4d241d0d8fb9" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2011 Dec 30 09:00:23 BMC:kernel:- <6>[kcs_enqueue_my_ioctl]:243:Pending transaction for channel[01] forced KCS to drop the current transaction
2011 Dec 30 09:00:22 BMC:kernel:- <6>[kcs_release_head_element_in_list]:309:Warning! Current KCS transaction[03] spent 0000004236 ms in queue
2011 Dec 30 09:00:22 BMC:kernel:- "<6>[kcs_ioctl]:394:Out of order sequence message dropped: Expected[0x69] Actual[0x37] "
The following message will indicate an invalid vector used as part of the Interrupt Remapping.
"vmkernel: 6:01:34:46.970 cpu0:4120)ALERT: APIC: 1823: APICID 0x00000000 - ESR = 0x40"
Root Cause
CSCtw68712
UCS C-series bios fix for VMware esx intel interrupt remapping issue
http://kb.vmware.com/kb/1030265vHBAs and other PCI devices may stop responding in ESX/ESXi 4.1 and ESXi 5.0 when using Interrupt Remapping
Please proceed to disable Interrupt Remapping with the workaruond specified on the above VMware Knowledge base
