Need help to determine cause of error message?

Unanswered Question
Jun 21st, 2009

%EARL_L2_ASIC-SW1_SPSTBY-4-DBUS_HDR_ERR: EARL L2 ASIC #0: Dbus

Hdr. Error occurred. Ctrl1 0xB08D0E3D

We recently implemented VSS 1440 on a pair of 6509E chasis's. Burned in for 3 weeks and tested for 3 weeks, no errors were reported. Two days after converting all our access switches to the VSS 6509's we started getting this error message constantly. I opened a ticket with TAC and the recommendation was to replace the SUP. After replacing the SUP, the same error is still consistenly being reported about 1 per minute. The ticket has been escalated, yet no solution has been provided. Has anyone seen this error message, or does anyone have any idea what could be causing it? All the hardware is brand new. We are running IOS version 12.2(33)SXI.

Any suggestions?

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Reza Sharifi Sun, 06/21/2009 - 08:44

dgi1,

Do you get the same error message if you fail over to the stand-by box?

We are also running VSS with IOS 12.2(33)SXI in our environment, but I have not seen this error message before. Just make sure all your hardware components are exactly the same. We had an issue with PFCs not being the same on both Sup modules and it took us some times to find the problem. We had to pull the Sups out of both boxes and do a comparison on the components and found out that one Sup had PFC3 and the other one had PFC3-XL.

HTH

Reza

dgj1 Sun, 06/21/2009 - 10:32

Hi Reza,

Yes, we get the same error message when we fail over to the stand-by box. Also another interesting symptom after the supervisor was replaced on switch 1 -

everything came back up normally except for two portchannels. Everything was working as expected on switch 2 during the supervisor change which I replaced on switch 1, however when switch 1 came back online 2 port-channels would not send traffic beyond the VSS switch. I resolved this problem by unplugging the Ethernet cable on one of them and plugging it back in on switch 1. On the other one I resolved it by shutting down the interface and then doing a no shut on the interface on switch 1.

Note before doing this, I could only ping the SVIs located on the VSS switch. I could not ping anything beyond the VSS switch from the access switches associated with these 2 port-channels. The trunks and portchannels showed active with no errors on both sides. Not exactly the failover/stability I was hoping for with VSS. Thanks for the advisement on the PFC3 vs PFC3-XL - I'll check it out.

Reza Sharifi Sun, 06/21/2009 - 11:48

Deanna,

When you say:

Port-channels would not send traffic beyond the VSS switch; do you mean the VSL links or port-channels going to your access box? You may already know this, but the port-channel you create for the VSL link has to be in different port-channel numbers. I.e., if on switch-1 the port-channel is 10, on switch-2 has to be 11 or any other number as long as they are not the same.

Can you post your VSS config?

Also, can you post the output of sh switch virtual role and sh switch virtual link

Thanks,

Reza

dgj1 Sun, 06/21/2009 - 11:58

The port-channels going to my access switches.

I can't post the VSS config, or the output today. I'm suppose to be on vacation, but having trouble getting this off my mind. Although I might be able to pull it out of the sho tech output I posted in my Cisco ticket.

jimmysands73_2 Sun, 06/21/2009 - 09:07

This is a little beyond my knowledge, but when I searched for your error (I do not have access to any Cisco CCO stuff yet) on the internet I found nothing, but when I searched for

EARL_L2_ASIC

I got this

CSCsd08468 cat6000 c7600-mpls SP crash at %EARL_L2_ASIC-SP-4-L2L3_SEQ_ERR due to invalid

packets

on Cisco site:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/release/notes/OL_4164.html (its a 7.1 MB pdf)

&

This fourm seems to have your exact error also...or close to it. He replaced the SUP also, didnt fix error.

http://www.gossamer-threads.com/lists/engine?do=post_view_printable;post=98195;list=cisco

This guy below had the same error you had also,

http://markmail.org/message/ji3g4sjlonxvs4lr

It beyond my skill set to fix your issue, but I can search and maybe find others out there that can.

Again, just a stab but thats how I learn,

Jimmy

dgj1 Sun, 06/21/2009 - 10:47

Jimmy,

Thanks for your input. Please see my recent response to Reza. Also the response I received from Cisco was:

This message indicates that the EARL L2 ASIC detected an invalid header in

the DBUS. This signifies that some malformed packet was received on the data

lookup bus by the supervisor.

Obviously I'm very concerned about things getting worse since I can't seem to find the root cause. If you come across anything I would really appreciate the help. This is a new installation and it has a lot of visibility right now if you know what I mean.

Thanks,

Deanna

jimmysands73_2 Sun, 06/21/2009 - 11:11

From:

http://puck.nether.net/pipermail/cisco-nsp/2008-October/055670.html

Hi

We had a similar error in conjunction with an es20 and a RSP720. We had

to replace the ES20 card because it suddenly stopped to forward L2

traffic.

Jul 11 17:08:17.212: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Jul 12 10:47:29.625: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Jul 12 16:11:52.394: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Jul 12 20:56:26.952: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Jul 13 06:08:40.319: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Jul 13 13:05:10.710: %FABRIC_INTF_ASIC-DFC1-5-FABRICSYNC_DONE: Fabric

ASIC 0 Channel 1: Fabric sync done.

Jul 13 13:05:10.520: %FABRIC-SP-6-TIMEOUT_ERR: Fabric in slot 5 reported

timeout error for channel 0 (Module 1, fabric connection 0)

Jul 13 20:39:35.491: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0:

Dbus Hdr. Error occurred. Ctrl1 0xB08D0EBD

Regards

Benjamin Conconi

Telekom-Ingenieur

jimmysands73_2 Mon, 06/22/2009 - 11:42

I did some more research last night, but I could not find out what the error specifically related too, maybe some of the heavy weights on here can weigh in on this.

Good luck,

Jimmy

dgj1 Thu, 06/25/2009 - 08:42

Cisco's response is that they were unable to locate a known bug in IOS causing the error and believe the problem may be the chassis. However, they are having us do some diagnostic work first; to ensure no other module is causing the error.

Actions

This Discussion