Module Fails

[yllee1314]

Hi all,

I was having issue with a particular module on a WS-C6509-E.

Module 9 just hang and lost connectivity.

Below are the error received during the outage.

Oct  8 15:06:13 MAL: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to down
Oct  8 15:06:13 MAL: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/2, changed state to down
Oct  8 15:06:13 MAL: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/3, changed state to down
Oct  8 15:06:13 MAL: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/1, changed state to down
Oct  8 15:06:13 MAL: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/2, changed state to down
Oct  8 15:06:13 MAL: %LINK-SP-3-UPDOWN: Interface TenGigabitEthernet9/3, changed state to down
Oct  8 15:06:13 MAL: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/1, changed state to down
Oct  8 15:06:13 MAL: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/2, changed state to down
Oct  8 15:06:13 MAL: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface TenGigabitEthernet9/3, changed state to down
Oct  8 15:06:14 MAL: %OIR-SP-6-REMCARD: Card removed from slot 9, interfaces disabled
Oct  8 15:06:41 MAL: %FABRIC-SP-6-TIMEOUT_ERR: Fabric in slot 5 reported timeout error for channel 17 (Module 9, fabric connection 1)
Oct  8 15:06:50 MAL: %OSPF-5-ADJCHG: Process 1, Nbr 172.21.255.4 on Vlan902 from FULL to DOWN, Neighbor Down: Dead timer expired
Oct  8 15:06:50 MAL: %FABRIC-SP-6-TIMEOUT_ERR: Fabric in slot 5 reported timeout error for channel 8 (Module 9, fabric connection 0)
Oct  8 15:06:51 MAL: %OSPF-5-ADJCHG: Process 1, Nbr 172.21.255.3 on Vlan901 from FULL to DOWN, Neighbor Down: Dead timer expired

At the same moment, the devices connected on module 6 was also affected but no logs found.

The issue recover after the module 9 was physically remove and slot in.

Both module 6 and 9 are WS-X6708-10GE.

Any method where we could check why the module just hang and not responding?

[Walter Astori]

Can you try to execute show tech-support ?

[yllee1314]

Show tech-support will be too risky as it's a production core switch

[Walter Astori]

I see that you have the module 6 with port 10 Giga. Can you move the configuration from module 9 to module 6 and verify if there is the same issue ?

[InayathUlla Sharieff]

show tech would be very usefull.

else

show module

show diagnostic result module 6/9

show log complete.

[yllee1314]

Attached is the show command.

However show tech-support will be too risky as it's a production core switch

[InayathUlla Sharieff]

Yllee,

Basically, this error is to inform us that the supervisor could not receive any reply to
any poll he transmitts to module 9, when the timeout is reached, it will cut off the power
supplied to the line card. 

Most of the time the issue gets fixed by either reseting or reseating the card. If the
error occurs again after reseating the card, then it is a hw failure.

Please capture the following output before you reseat the card.

show mod
show logg buff 1023
show fabric channel-counters all
show fabric status
show fabric errors
show fabric switching-mode
show fabric utilization  all

HTH