c3560g-24ts random crash

Unanswered Question
Jul 12th, 2010

Today for some reason the switch just crashed. using cisco Network Assistant see the following info.

System Messages
Device : Switch1
 
Severity,Timestamp,Event Description
1-Alert,Mar 01 00:02:16,R4 = 0x00000009  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x00000000
1-Alert,Mar 01 00:02:16,R8 = 0x00007530  R9 = 0x00000000  R10 = 0x00000000  R11 = 0x00000005
1-Alert,Mar 01 00:02:16,R12 = 0xF321A0F0  R13 = 0x00110000  R14 = 0x00B3A434  R15 = 0x00000000
1-Alert,Mar 01 00:02:16,R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
1-Alert,Mar 01 00:02:16,R20 = 0x00000000  R21 = 0x0263C2A8  R22 = 0x079F8E08  R23 = 0x079F8EF9
1-Alert,Mar 01 00:02:16,R24 = 0x029B0450  R25 = 0x00000000  R26 = 0x079F8E04  R27 = 0x02127A10
1-Alert,Mar 01 00:02:16,R28 = 0x026EBED8  R29 = 0x00000000  R30 = 0x026EBEB0  R31 = 0x00000000
1-Alert,Mar 01 00:02:16,
1-Alert,Mar 01 00:02:16,Stack trace:
1-Alert,Mar 01 00:02:16,PC = 0x00A27A3C, SP = 0x02B41D88
1-Alert,Mar 01 00:02:16,Frame 00SP = 0x02B41D98    PC = 0x00A27A00
1-Alert,Mar 01 00:02:16,Frame 01SP = 0x02B41DD0    PC = 0x014E9AE8
1-Alert,Mar 01 00:02:16,Frame 02SP = 0x02B41DF0    PC = 0x016D76A4
1-Alert,Mar 01 00:02:16,Frame 03SP = 0x02B41E18    PC = 0x00B41FF0
1-Alert,Mar 01 00:02:16,Frame 04SP = 0x02B41EB8    PC = 0x00B41618
1-Alert,Mar 01 00:02:16,Frame 05SP = 0x02B41EF8    PC = 0x00B3B744
1-Alert,Mar 01 00:02:16,Frame 06SP = 0x02B41F38    PC = 0x00B3AB34
1-Alert,Mar 01 00:02:16,Frame 07SP = 0x02B41F40    PC = 0x00A299C8
1-Alert,Mar 01 00:02:16,Frame 08SP = 0x00000000    PC = 0x00A20498
1-Alert,Mar 01 00:02:16,

Have found like 1 post that user had similar error. But advice was just to update IOS. The other switches have been up for 33 weeks with no problems.

C3560G-24TS, ver 12.2(50)SE1

I have 4 other switches using the same IOS and configured exactly the same way. 6th switch is ver SE3.

I cant create a TAC for it as not covered. Any help would be appreciated.

TIA

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3 (1 ratings)
Loading.
jefforamma Tue, 07/13/2010 - 04:02

there were 3 in the dir. it did crash twice yesterday.

crashinfo_1

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x014FF84C  SRR1 = 0x00029230  SRR2 = 0x00694B94  SRR3 = 0x00021000
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x00A27A3C  MSR = 0x00029230  CR = 0x33000053
LR = 0x00A27A00  CTR = 0x014F4BB8  XER = 0xE000004A
R0 = 0x00A27A00  R1 = 0x02EE5068  R2 = 0x00000000  R3 = 0x02C16ED4
R4 = 0x00000000  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x00000000
R8 = 0x00007530  R9 = 0x00000000  R10 = 0x00000000  R11 = 0x00000005
R12 = 0xD4F2A233  R13 = 0x00110000  R14 = 0x0169D8E4  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x0000006A
R24 = 0x00000000  R25 = 0x0000006B  R26 = 0x00000000  R27 = 0x02127A10
R28 = 0x03A6A64C  R29 = 0x00000000  R30 = 0x03A6A624  R31 = 0x00000000

Stack trace:
PC = 0x00A27A3C, SP = 0x02EE5068
Frame 00: SP = 0x02EE5078    PC = 0x00A27A00
Frame 01: SP = 0x02EE50B0    PC = 0x014E9AE8
Frame 02: SP = 0x02EE50D0    PC = 0x016D76A4
Frame 03: SP = 0x02EE5108    PC = 0x00C01FF0
Frame 04: SP = 0x02EE5128    PC = 0x0169DBD4
Frame 05: SP = 0x02EE5130    PC = 0x00A299C8
Frame 06: SP = 0x00000000    PC = 0x00A20498

crashinfo_2

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x014FF84C  SRR1 = 0x00029230  SRR2 = 0x00694B94  SRR3 = 0x00021000
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x00A27A3C  MSR = 0x00029230  CR = 0x33005053
LR = 0x00A27A00  CTR = 0x014F4BB8  XER = 0xE000004A
R0 = 0x00A27A00  R1 = 0x02CE9D88  R2 = 0x00000000  R3 = 0x02BE0F74
R4 = 0x00000009  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x00000000
R8 = 0x00007530  R9 = 0x00000000  R10 = 0x00000000  R11 = 0x00000005
R12 = 0xD38290EE  R13 = 0x00110000  R14 = 0x00E92C78  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x00000000  R26 = 0x00000000  R27 = 0x02127A10
R28 = 0x02B948EC  R29 = 0x00000000  R30 = 0x02B948C4  R31 = 0x00000000

Stack trace:
PC = 0x00A27A3C, SP = 0x02CE9D88
Frame 00: SP = 0x02CE9D98    PC = 0x00A27A00
Frame 01: SP = 0x02CE9DD0    PC = 0x014E9AE8
Frame 02: SP = 0x02CE9DF0    PC = 0x016D76A4
Frame 03: SP = 0x02CE9DF8    PC = 0x00A42D38
Frame 04: SP = 0x02CE9E10    PC = 0x00A420CC
Frame 05: SP = 0x02CE9E28    PC = 0x00A42034
Frame 06: SP = 0x02CE9E38    PC = 0x00A4272C
Frame 07: SP = 0x02CE9E60    PC = 0x0159CE30
Frame 08: SP = 0x02CE9E70    PC = 0x0159D010
Frame 09: SP = 0x02CE9E98    PC = 0x016379C0
Frame 10: SP = 0x02CE9EB8    PC = 0x00E98828
Frame 11: SP = 0x02CE9ED0    PC = 0x00E98BB4
Frame 12: SP = 0x02CE9EE8    PC = 0x00E99000
Frame 13: SP = 0x02CE9EF0    PC = 0x00EA13F4
Frame 14: SP = 0x02CE9F08    PC = 0x00E92B64
Frame 15: SP = 0x02CE9F40    PC = 0x00E93008

crashinfo_3

Switch1#more flash:crashinfo/crashinfo_3
Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x014FF84C  SRR1 = 0x00029230  SRR2 = 0x00694B94  SRR3 = 0x00021000
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x00A27A3C  MSR = 0x00029230  CR = 0x30000053
LR = 0x00A27A00  CTR = 0x014F4BB8  XER = 0xE000004A
R0 = 0x00A27A00  R1 = 0x02B41D88  R2 = 0x00000000  R3 = 0x03A66AF4
R4 = 0x00000009  R5 = 0x00000000  R6 = 0x00000000  R7 = 0x00000000
R8 = 0x00007530  R9 = 0x00000000  R10 = 0x00000000  R11 = 0x00000005
R12 = 0xF321A0F0  R13 = 0x00110000  R14 = 0x00B3A434  R15 = 0x00000000
R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000
R20 = 0x00000000  R21 = 0x0263C2A8  R22 = 0x079F8E08  R23 = 0x079F8EF9
R24 = 0x029B0450  R25 = 0x00000000  R26 = 0x079F8E04  R27 = 0x02127A10
R28 = 0x026EBED8  R29 = 0x00000000  R30 = 0x026EBEB0  R31 = 0x00000000

Stack trace:
PC = 0x00A27A3C, SP = 0x02B41D88
Frame 00: SP = 0x02B41D98    PC = 0x00A27A00
Frame 01: SP = 0x02B41DD0    PC = 0x014E9AE8
Frame 02: SP = 0x02B41DF0    PC = 0x016D76A4
Frame 03: SP = 0x02B41E18    PC = 0x00B41FF0
Frame 04: SP = 0x02B41EB8    PC = 0x00B41618
Frame 05: SP = 0x02B41EF8    PC = 0x00B3B744
Frame 06: SP = 0x02B41F38    PC = 0x00B3AB34
Frame 07: SP = 0x02B41F40    PC = 0x00A299C8
Frame 08: SP = 0x00000000    PC = 0x00A20498

Leo Laohoo Tue, 07/13/2010 - 15:00

Looks like an IOS-caused crash.  Upgrade the IOS to 12.2.50-SE4.

s-lahtinen Tue, 07/13/2010 - 20:14

> Re: c3560g-24ts random  crash

> Looks like an IOS-caused crash.   Upgrade the IOS to 12.2.50-SE4.

I have the same problem on 12.2(50)SE4 on the same platform yesterday. First it happened on 12.2(35)SE5, and then following upgrade, it happened again. Same "Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!" error.. the latest crash was after exiting configure terminal after changing line vty entries/acls associated with them.

Interestingly I also have a separate but simultaneous problem on another 3560 (which I'm trying to resolve with TAC). Both have been running fine for weeks until now; one of them for over a year. We ran rancid before, but as we saw some strange behaviour which we couldn't attribute to an operator, we've disabled it for now.

SL

jimmysands73_2 Wed, 07/14/2010 - 15:29

Have you checked out .... imagine you have, but just in case.

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807ccc79.shtml#deref

Debug Exception (Could be NULL pointer dereference)

A Catalyst 3560 Series Switch that runs Cisco IOS system software reloads with the

Debug Exception

(Could be NULL pointer dereference)

error message in the logs.

The probable causes for the error message are:

Memory leak in CEF background process. For information on how to resolve this issue, see

Unexpected memory consumption in CEF IPC Background process.

·

Powered device detection.

This issue occurs when the powered device is detected or classified as an

overcurrent class. This issue

is documented in Cisco bug ID CSCsa72400 (

registered customers only) .

Also...

https://cisco-support.hosted.jivesoftware.com/docs/DOC-4236?decorator=print

Good luck...

s-lahtinen Wed, 07/14/2010 - 16:08

Thanks for the thoughts.. I have checked those before..

> Have you checked out .... imagine you have, but just in case.

> http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807ccc79.shtml#deref

I read that as (a) only affecting 3750s (although some documents mention 3560 too), (b) being related to PoE and this is a 24TS, not 24PS, and (c) I believe some documents mention it was fixed in 12.2(25)SEA and the oldest IOS we sae this in was 12.2(35)SE ..(I think just SE).. but certainly not 12.2(25)SEA.. although bug toolkit still suggests "Release-Pending".


Also some documentation suggests it happens on 3750s when (a) switches are stacked, (b) IP routing is disabled and (c) when the master switch it changed; Since it's 3560 it's not stacked, and IP routing is enabled so they seem to be contraindicated.

SL

jefforamma Thu, 07/15/2010 - 09:47

thanks for the inputs guys. I looked over that link but from what i saw it did not apply to my 3560's. They dont do POE and the other mentions of 3560 dont really apply either. (though i did setup ntp for them now due to what i read on there about the times.)

there was no console/config connections to the switches that day either as only 2 of us do that here and i know neither of us were on teh switches.

it has not done anything since that day and has been stable so far (knock on wood)

i will look at updating it to a newer ios but where it is a prod switch i just can't update it anytime i want. i have a spare in place and will switch it out if it does it again and continue searching and monitoring it in the meantime.

jimmy i gave u some stars as i did find that time issue in your link and fixed that

s-lahtinen Thu, 07/15/2010 - 21:26

FYI TAC have suggested a bug to me which I have replicated twice causing a crash, so I'm confident that's causing it in my case.

If you're interested it's CSCth30052.

SL

jefforamma Mon, 07/19/2010 - 09:51

thnx SL. I dont use IPv6 at all and was not playing with those settings. Not sure if it would crash it if I did but not going to attempt it right now.

Actions

This Discussion