Disable Interrupt Remapping UCS for UC Applications

Document

Apr 4, 2012 8:26 AM
Apr 4th, 2012

Background

Interrupt remapping is a feature implemented in Intel VT-d or AMD IOMMU that provides isolation and compatibility for the Hypervisor to Direct IO to specific CPU through another mapping layer which allows software to control (mainly) the destination and vector for the Interrupts.  This feature is enabled by default on ESXi 4.1 and ESXi 5.0. It's known that some vendors architectures including UCS have compatibility problems with Interrupt Remapping Functionality. The Virtual Machine guess OS will randomly loose access to the underlying storage which will compromise the integrity of the file system. VM OS that gets corrupted and report filesystem inconsistencies due to problems with Interrupt Remapping might need to be reinstalled.

It's therefore requried to disable interrupt Remapping in ESXi for UC applications.

Interrupt remapping usually leads to Virtual Machines going into Read only state, and becoming hang or corrupted. A few examples of common problems identified by CISCO TAC include, but are not limited to:

  • Virtual Machines not responding.
  • Virtual Machines reporting file system inconsistencies.
  • Virtual Machines hang.
  • You may receive an alerts about "Degraded path for an Unknown Device" in vCenter Server
  • Virtual Machines going into read only mode.
  • Virtual Machines stuck at shut down or bootup of the VM indefinetly.

Troubleshooting

Common Symptoms found during troubleshooting:

ESXi host might display errors related to SCSI commands not executing succesfully. Other write related commands might report errors.

KVM.PNG

Virtual Machines might report file system inconsistencies and might request a forced File system check to run.

ass46_rep_error.jpg

Vmware support logs from the virtual Machines might report some of the following error messages:

7:2011 Dec  5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:220:peci: transaction failed: [0x61]

7:2011 Dec  5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:190:peci: bad FCS1, received = 0x0, expected = 0xe7

7:2011 Dec  5 18:09:02:BMC:kernel:-:<7>[peci_issue_transaction]:220:peci: transaction failed: [0x61]

Jan 14 22:23:53 vmkernel: 1:04:45:42.202 cpu2:4098)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f2ce640) to NMP device "naa.600605b0030408c014fe4d241d0d8fb9" failed on physical path "vmhba4:C2:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

Jan 14 22:23:53 vmkernel: 1:04:45:42.202 cpu2:4098)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600605b0030408c014fe4d241d0d8fb9" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2011 Dec 30 09:00:23 BMC:kernel:-     <6>[kcs_enqueue_my_ioctl]:243:Pending transaction for channel[01] forced KCS to drop the current transaction

2011 Dec 30 09:00:22 BMC:kernel:-     <6>[kcs_release_head_element_in_list]:309:Warning! Current KCS transaction[03] spent 0000004236 ms in queue

2011 Dec 30 09:00:22 BMC:kernel:-     "<6>[kcs_ioctl]:394:Out of order sequence message dropped: Expected[0x69] Actual[0x37] "

The following message will indicate an invalid vector used as part of the Interrupt Remapping.

"vmkernel: 6:01:34:46.970 cpu0:4120)ALERT: APIC: 1823: APICID 0x00000000 - ESR = 0x40"

Root Cause

CSCtw68712

UCS C-series bios fix for VMware esx intel interrupt remapping issue

http://kb.vmware.com/kb/1030265

vHBAs and other PCI devices may stop responding in ESX/ESXi 4.1 and ESXi 5.0 when using Interrupt Remapping

Please proceed to disable Interrupt Remapping with the workaruond specified on the above VMware Knowledge base


Average Rating: 5 (4 ratings)

Comments

kenperez Mon, 05/21/2012 - 11:05

This now has also been address in codes 1.4(3)2c and above.

For blade servers this is corrected in BIOS 2.0(1m)

So the user can either, disable this option or upgrade to the release above mentioned.

-Kenny

Actions

Login or Register to take actions

This Document

Posted April 4, 2012 at 8:26 AM
Stats:
Comments:1 Avg. Rating:5
Views:10380 Contributors:1
Shares:0

Related Content

Documents Leaderboard