Catalyst 2960 crash

Unanswered Question
Apr 26th, 2010
User Badges:

Hi All,


My customers Catalyst 2960 had some serious problems few days ago, so I'd be glad if someone helped me with solving this issue...


Catalyst 2960 stopped responding and stopped forwarding traffic on 3 out of 4 used ports. In that time, no console access, neither remote access to switch was possible, so force reload was performed.


After the reload, switch worked for 6 hours, and after that switch crashed again.


Since 2960 has old version of IOS ( /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} c2960-lanbase-mz.122-25.FX), I will first try with IOS upgrade because 12.2 25 FX is old version of IOS.


Here are some sh ver outputs captured after the second crash.


//**************


sh ver


/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

Cisco IOS Software, C2960 Software (C2960-LANBASE-M), Version 12.2(25)FX, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 12-Oct-05 22:05 by yenanh

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 12.2(25r)SEE1, RELEASE SOFTWARE (fc1)

2960switch uptime is 2 minutes
System returned to ROM by error - Data TLB Miss Exception (0x1100)!, PC 0x781900
System image file is "flash:c2960-lanbase-mz.122-25.FX/c2960-lanbase-mz.122-25.FX.bin"

cisco WS-C2960-24TT-L (PowerPC405) processor (revision B0) with 61440K/4088K bytes of memory.
Processor board ID FOC1031Z2N5
Last reset from power-on
2 Virtual Ethernet interfaces
24 FastEthernet interfaces
2 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:19:06:F2:83:80
Motherboard assembly number : 73-10390-03
Power supply part number : 341-0097-02
Motherboard serial number : FOC10304JZ4
Power supply serial number : DCA10253KSX
Model revision number : B0
Motherboard revision number : C0
Model number : WS-C2960-24TT-L
System serial number : FOC1031Z2N5
Top Assembly Part Number : 800-27221-02
Top Assembly Revision Number : B0
Version ID : V02
CLEI Code Number : COM3L00BRA
Hardware Board Revision Number : 0x01


Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 26 WS-C2960-24TT-L 12.2(25)FX C2960-LANBASE-M


Configuration register is 0xF


2. crash info

//**********************************************


Cisco IOS Software, C2960 Software (C2960-LANBASE-M), Version 12.2(25)FX, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 12-Oct-05 22:05 by yenanh


Data TLB Miss Exception (0x1100)!


SRR0 = 0x00781900  SRR1 = 0x00029210  SRR2 = 0x00416978  SRR3 = 0x00021200
ESR = 0x00000000  DEAR = 0x3B60001D  TSR = 0x8C000000  DBSR = 0x00000000


CPU Register Context:
Vector = 0x00001100  PC = 0x00781900  MSR = 0x00029210  CR = 0x53000009
LR = 0x0078106C  CTR = 0x009680D4  XER = 0xE0000067
R0 = 0x0000FFFF  R1 = 0x018316E8  R2 = 0x00000000  R3 = 0x017A1C50
R4 = 0x0AEA0533  R5 = 0x00110000  R6 = 0x00000001  R7 = 0x00000000
R8 = 0x018316F0  R9 = 0x3B600001  R10 = 0x00000000  R11 = 0x00000001
R12 = 0x01114BE6  R13 = 0x00110000  R14 = 0x000C3F1C  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000011
R24 = 0x00000011  R25 = 0xAC19000E  R26 = 0x01831808  R27 = 0x0AEA0533
R28 = 0x00000004  R29 = 0x00000000  R30 = 0x0AEA0533  R31 = 0x00000000


Stack trace:
PC = 0x00781900, SP = 0x018316E8
Frame 00: SP = 0x01831708    PC = 0x00000001
Frame 01: SP = 0x01831720    PC = 0x0077B764
Frame 02: SP = 0x01831738    PC = 0x007A827C
Frame 03: SP = 0x01831740    PC = 0x007A837C
Frame 04: SP = 0x01831748    PC = 0x00024274
Frame 05: SP = 0x01831750    PC = 0x007BBBC4
Frame 06: SP = 0x01831768    PC = 0x007A0334
Frame 07: SP = 0x018317C8    PC = 0x0079EDAC
Frame 08: SP = 0x01831800    PC = 0x007B6FA8
Frame 09: SP = 0x01831828    PC = 0x007E2674
Frame 10: SP = 0x01831848    PC = 0x007E23DC
Frame 11: SP = 0x01831858    PC = 0x000C3CC4
Frame 12: SP = 0x01831898    PC = 0x000C4704
Frame 13: SP = 0x018318A0    PC = 0x009AAE0C
Frame 14: SP = 0x00000000    PC = 0x009A5050



//**********************************************

Interesting thing is output about Translation Lookaside Buffer miss exception...

System returned to ROM  by error - Data TLB Miss Exception (0x1100)!, PC 0x781900

And Data TLB Miss  Exception (0x1100)!


As far as I'm concerned TLB miss is CPU cycle consuming because data is not present in virtual memory, and exception has to be served in order to read page from physical memory and map it to virtual mem etc etc...

But this should not reload the switch...


Any suggestions, except IOS upgrade ?


Thanks in advance !


Best Ragards,
Ana

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Loading.
Giuseppe Larosa Mon, 04/26/2010 - 06:20
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Ana,

I would go on with the IOS upgrade.


Keep us informed on results of upgrade


Hope to help

Giuseppe

Edison Ortiz Mon, 04/26/2010 - 06:32
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

A TLB Miss Exception indicates there was a virtual memory location that was not properly written to and this caused the switch to crash.

If this was just a one time event, I wouldn't upgrade the IOS. If the problem is chronic, I recommend upgrading the IOS as you may be suffering from memory leaks.


You can check for memory leaks by typing show process memory sorted every hour within a 5 hour period. The Free: value should increase and decrease. If it keeps decreasing, you have a memory leak.


Regards


Edison

ana.peric Mon, 04/26/2010 - 07:49
User Badges:

Giuseppe, Edison thank you both for your fast reply !


This was recurring event, so I'll go with an IOS upgrade.

I'll let you know what are the outcomes of this action .



Regards,

Ana

Sundeep Dsouza Wed, 02/02/2011 - 02:04
User Badges:

Hi Ana,


Was this problem solved? What did you do to solve this? One of our Catalyst 500 switch reported with exactly the same error.


Regards

ana.peric Mon, 05/02/2011 - 10:56
User Badges:

Hi Sundeep,


Sorry for my late answer.

Yes, upgrade did solve this issue.


Best regards,

Ana

Actions

This Discussion

Related Content