Strange 2960 crash

cbeswick · ‎12-16-2009

Hi,

One of my engineers issued a command to turn off port security on a number of ports using the range command. The command failed on the first attempt due to a tacacs auth failure which I suspect is due to a low tacacs timeout value. The engineer then reduced the number of ports in the range command and re-issued the config change after which the switch just crashed and rebooted.

The logging buffer on the switch displays the following:

000072: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: System previously crashed with the following message:

000073: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 12.2(50)SE3, RELEASE SOFTWARE (fc1)

000074: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport

000076: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Compiled Wed 22-Jul-09 07:03 by prod_rel_team

000077: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED:

000078: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Data TLB Miss Exception (0x1100)!

000079: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED:

000080: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: SRR0 = 0x00B2F2B0 SRR1 = 0x00029230 SRR2 = 0x0064A544 SRR3 = 0x00021200

000081: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x0D0D1D79 TSR = 0x8C000000 DBSR = 0x00000000

000082: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED:

000083: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: CPU Register Context:

000084: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Vector = 0x00001100 PC = 0x00B2F2B0 MSR = 0x00029230 CR = 0x33005055

000085: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: LR = 0x00B320F0 CTR = 0x0065F470 XER = 0xA0000040

000086: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R0 = 0x00B320F0 R1 = 0x03382450 R2 = 0x00000000 R3 = 0x0345AEE4

000087: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R4 = 0x3C31819D R5 = 0x0000001C R6 = 0x00003C31 R7 = 0x3C31819D

000088: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x0D0D0D0D R10 = 0x019F0000 R11 = 0x00000000

000089: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R12 = 0xA0000000 R13 = 0x00110000 R14 = 0x011F406C R15 = 0x00000000

000090: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R16 = 0x0231477C R17 = 0x0231477C R18 = 0x00000006 R19 = 0xFFFFFFFF

000091: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R20 = 0x00000001 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x01D20000

000092: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x01FC9D44

000093: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: R28 = 0x033824C0 R29 = 0x00000000 R30 = 0x0345AEE4 R31 = 0x0345AEE4

000094: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED:

000095: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Stack trace:

000096: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: PC = 0x00B2F2B0, SP = 0x03382450

000097: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 00: SP = 0x033824A0 PC = 0x59EF102B

000098: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 01: SP = 0x03382500 PC = 0x00B320F0

000099: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 02: SP = 0x03382538 PC = 0x00B310BC

000100: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 03: SP = 0x033825E8 PC = 0x00B3F96C

000101: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 04: SP = 0x03382600 PC = 0x00B40930

000102: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 05: SP = 0x03382610 PC = 0x00B3FEAC

000103: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 06: SP = 0x03382638 PC = 0x00B3078C

000104: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 07: SP = 0x03382648 PC = 0x00B30180

000105: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 08: SP = 0x03382658 PC = 0x00B45358

000106: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 09: SP = 0x033826F0 PC = 0x00B6BE98

000107: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 10: SP = 0x03382718 PC = 0x012022FC

000108: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 11: SP = 0x03382738 PC = 0x011F4334

000109: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 12: SP = 0x03382740 PC = 0x0108EE50

000110: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED: Frame 13: SP = 0x00000000 PC = 0x01085920

000111: *Mar 1 00:03:00 GMT: %PLATFORM-1-CRASHED:

I have done some searching and this could be related to bug CSCsq71492. Has anyone got any idea's where we should go next ?

I have tried using the output interpreter but it is still down.

Chris.

Rick Morris · ‎12-16-2009

I have not seen this.

Is there any config in the switch?

Is there an error file in the directory, or a crash file?

This may sound like something to send to TAC for investigation.

cbeswick · ‎12-16-2009

Hi,

Yes there is a config on the switch, we also have a crashinfo file in flash. Unfortunately this switch isn't under a support contract nor is it under smartnet (unless you get a standard level of support for new switches). So we are stuck unless someone here has any recommendations.

Jerry Ye · ‎12-16-2009

Can you post the crashinfo file?

Regards,

jerry

sachinraja · ‎12-16-2009

Hi

what is the IOS version you are running ? Best thing would be to upgrade the IOS to the latest one...

the bug ID you gave me doesnt correspond to this error... it relates to something as given below:

CSCsq71492 Bug Details

Symptoms: A Cisco IOS device may reload with an address error or have alignment errors and tracebacks such as %ALIGN-3-SPURIOUS or %ALIGN-3-TRACE

Output interpretor shows the following:

%PLATFORM-1-CRASHED: [chars]

Explanation:This message indicate that the system is attempting to display the

failure message from a previous failure. [chars] is the description of the error

message.

Recommended Action: Use the exception crashinfo command to enable 'core dump'

on the switch that crashed. The core dump file will be able to a capture a more

detailed crash and have enough information to determine a root cause. Consider

an IOS upgrade.

Is the switch rebooting continuously ?

Raj

cbeswick · ‎12-16-2009

Crashinfo file attached.

The reason I linked it to bug CSCsq71492 is because we were seeing "command authorisation failures" from tacacs just before the switch crashed, which are usually as a result of the tacacs server timing out (assuming the user has the correct level of authorisation in ACS).

cbeswick · ‎12-16-2009

Crashinfo file attached.

The reason I thought it might be bug CSCsq71492 is because we were seeing alot of "command authorisation failures" as a result of tacacs timing out just before the crash.

The software version is 12.2(50)SE3

sachinraja · ‎12-16-2009

Could just be this Open Caveat:

CSCsx25276

When software is reloaded on a switch stack master that has port security enabled, the switch stack might crash.

There is no workaround.

Have a look at the release notes:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_50_se/release/notes/OL18263.html

Raj

cbeswick · ‎12-16-2009

Is this caveat not specific to a stack, presumable the c3750 / c3750-E ?

We saw the crash after issuing port security related commands, not after reloading a switch that has port security enabled.

Thanks for your help btw

sachinraja · ‎12-16-2009

It says 3750, but it could also be with 2960 since the release notes is common between 3750,3560 & 2960 ! bugs can be between platforms.. In any case,

what is the status of the device now ? hanged ? not accessible via console? accessible ?

Raj

cbeswick · ‎12-16-2009

The switch appears to be running perfectly fine now. I think the best thing we could do is upgrade to the latest release which seems to have the least number of open caveats, i.e. 12.2(53)SE

What do you think ?

sachinraja · ‎12-16-2009

Yes.. thats the way to go ! upgrade it and am sure it will solve your issue... unless your switch is hit with some other bug

Raj

Leo Laohoo · ‎12-16-2009

12.2(53)SE??? You mean 12.2(52)SE.

sachinraja · ‎12-16-2009

12.2(52)SE

c2960-lanbasek9-mz.12.2-52.SE

min flash 32 mb

min dram 64 mb

Hope this helps.. all the best

Raj

Jerry Ye · ‎12-16-2009

Don't believe this is cause by CSCsx25276. In the crashinfo file, I see the interface range command is involved, and could be related to CSCsz13490. However, I am not able to find the excact match.

Regards,

jerry