07-12-2010 12:28 PM - edited 03-06-2019 11:59 AM
Today for some reason the switch just crashed. using cisco Network Assistant see the following info.
System Messages
Device : Switch1
Severity,Timestamp,Event Description
1-Alert,Mar 01 00:02:16,R4 = 0x00000009 R5 = 0x00000000 R6 = 0x00000000 R7 = 0x00000000
1-Alert,Mar 01 00:02:16,R8 = 0x00007530 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00000005
1-Alert,Mar 01 00:02:16,R12 = 0xF321A0F0 R13 = 0x00110000 R14 = 0x00B3A434 R15 = 0x00000000
1-Alert,Mar 01 00:02:16,R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
1-Alert,Mar 01 00:02:16,R20 = 0x00000000 R21 = 0x0263C2A8 R22 = 0x079F8E08 R23 = 0x079F8EF9
1-Alert,Mar 01 00:02:16,R24 = 0x029B0450 R25 = 0x00000000 R26 = 0x079F8E04 R27 = 0x02127A10
1-Alert,Mar 01 00:02:16,R28 = 0x026EBED8 R29 = 0x00000000 R30 = 0x026EBEB0 R31 = 0x00000000
1-Alert,Mar 01 00:02:16,
1-Alert,Mar 01 00:02:16,Stack trace:
1-Alert,Mar 01 00:02:16,PC = 0x00A27A3C, SP = 0x02B41D88
1-Alert,Mar 01 00:02:16,Frame 00SP = 0x02B41D98 PC = 0x00A27A00
1-Alert,Mar 01 00:02:16,Frame 01SP = 0x02B41DD0 PC = 0x014E9AE8
1-Alert,Mar 01 00:02:16,Frame 02SP = 0x02B41DF0 PC = 0x016D76A4
1-Alert,Mar 01 00:02:16,Frame 03SP = 0x02B41E18 PC = 0x00B41FF0
1-Alert,Mar 01 00:02:16,Frame 04SP = 0x02B41EB8 PC = 0x00B41618
1-Alert,Mar 01 00:02:16,Frame 05SP = 0x02B41EF8 PC = 0x00B3B744
1-Alert,Mar 01 00:02:16,Frame 06SP = 0x02B41F38 PC = 0x00B3AB34
1-Alert,Mar 01 00:02:16,Frame 07SP = 0x02B41F40 PC = 0x00A299C8
1-Alert,Mar 01 00:02:16,Frame 08SP = 0x00000000 PC = 0x00A20498
1-Alert,Mar 01 00:02:16,
Have found like 1 post that user had similar error. But advice was just to update IOS. The other switches have been up for 33 weeks with no problems.
C3560G-24TS, ver 12.2(50)SE1
I have 4 other switches using the same IOS and configured exactly the same way. 6th switch is ver SE3.
I cant create a TAC for it as not covered. Any help would be appreciated.
TIA
07-12-2010 03:00 PM
Can you post the "crashinfo" file please?
07-13-2010 04:02 AM
there were 3 in the dir. it did crash twice yesterday.
crashinfo_1
Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x014FF84C SRR1 = 0x00029230 SRR2 = 0x00694B94 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000
CPU Register Context:
Vector = 0x00002000 PC = 0x00A27A3C MSR = 0x00029230 CR = 0x33000053
LR = 0x00A27A00 CTR = 0x014F4BB8 XER = 0xE000004A
R0 = 0x00A27A00 R1 = 0x02EE5068 R2 = 0x00000000 R3 = 0x02C16ED4
R4 = 0x00000000 R5 = 0x00000000 R6 = 0x00000000 R7 = 0x00000000
R8 = 0x00007530 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00000005
R12 = 0xD4F2A233 R13 = 0x00110000 R14 = 0x0169D8E4 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x0000006A
R24 = 0x00000000 R25 = 0x0000006B R26 = 0x00000000 R27 = 0x02127A10
R28 = 0x03A6A64C R29 = 0x00000000 R30 = 0x03A6A624 R31 = 0x00000000
Stack trace:
PC = 0x00A27A3C, SP = 0x02EE5068
Frame 00: SP = 0x02EE5078 PC = 0x00A27A00
Frame 01: SP = 0x02EE50B0 PC = 0x014E9AE8
Frame 02: SP = 0x02EE50D0 PC = 0x016D76A4
Frame 03: SP = 0x02EE5108 PC = 0x00C01FF0
Frame 04: SP = 0x02EE5128 PC = 0x0169DBD4
Frame 05: SP = 0x02EE5130 PC = 0x00A299C8
Frame 06: SP = 0x00000000 PC = 0x00A20498
crashinfo_2
Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x014FF84C SRR1 = 0x00029230 SRR2 = 0x00694B94 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000
CPU Register Context:
Vector = 0x00002000 PC = 0x00A27A3C MSR = 0x00029230 CR = 0x33005053
LR = 0x00A27A00 CTR = 0x014F4BB8 XER = 0xE000004A
R0 = 0x00A27A00 R1 = 0x02CE9D88 R2 = 0x00000000 R3 = 0x02BE0F74
R4 = 0x00000009 R5 = 0x00000000 R6 = 0x00000000 R7 = 0x00000000
R8 = 0x00007530 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00000005
R12 = 0xD38290EE R13 = 0x00110000 R14 = 0x00E92C78 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x02127A10
R28 = 0x02B948EC R29 = 0x00000000 R30 = 0x02B948C4 R31 = 0x00000000
Stack trace:
PC = 0x00A27A3C, SP = 0x02CE9D88
Frame 00: SP = 0x02CE9D98 PC = 0x00A27A00
Frame 01: SP = 0x02CE9DD0 PC = 0x014E9AE8
Frame 02: SP = 0x02CE9DF0 PC = 0x016D76A4
Frame 03: SP = 0x02CE9DF8 PC = 0x00A42D38
Frame 04: SP = 0x02CE9E10 PC = 0x00A420CC
Frame 05: SP = 0x02CE9E28 PC = 0x00A42034
Frame 06: SP = 0x02CE9E38 PC = 0x00A4272C
Frame 07: SP = 0x02CE9E60 PC = 0x0159CE30
Frame 08: SP = 0x02CE9E70 PC = 0x0159D010
Frame 09: SP = 0x02CE9E98 PC = 0x016379C0
Frame 10: SP = 0x02CE9EB8 PC = 0x00E98828
Frame 11: SP = 0x02CE9ED0 PC = 0x00E98BB4
Frame 12: SP = 0x02CE9EE8 PC = 0x00E99000
Frame 13: SP = 0x02CE9EF0 PC = 0x00EA13F4
Frame 14: SP = 0x02CE9F08 PC = 0x00E92B64
Frame 15: SP = 0x02CE9F40 PC = 0x00E93008
crashinfo_3
Switch1#more flash:crashinfo/crashinfo_3
Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(50)SE1, RELE
ASE SOFTWARE (fc2)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 06-Apr-09 08:00 by amvarma
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x014FF84C SRR1 = 0x00029230 SRR2 = 0x00694B94 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000
CPU Register Context:
Vector = 0x00002000 PC = 0x00A27A3C MSR = 0x00029230 CR = 0x30000053
LR = 0x00A27A00 CTR = 0x014F4BB8 XER = 0xE000004A
R0 = 0x00A27A00 R1 = 0x02B41D88 R2 = 0x00000000 R3 = 0x03A66AF4
R4 = 0x00000009 R5 = 0x00000000 R6 = 0x00000000 R7 = 0x00000000
R8 = 0x00007530 R9 = 0x00000000 R10 = 0x00000000 R11 = 0x00000005
R12 = 0xF321A0F0 R13 = 0x00110000 R14 = 0x00B3A434 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x0263C2A8 R22 = 0x079F8E08 R23 = 0x079F8EF9
R24 = 0x029B0450 R25 = 0x00000000 R26 = 0x079F8E04 R27 = 0x02127A10
R28 = 0x026EBED8 R29 = 0x00000000 R30 = 0x026EBEB0 R31 = 0x00000000
Stack trace:
PC = 0x00A27A3C, SP = 0x02B41D88
Frame 00: SP = 0x02B41D98 PC = 0x00A27A00
Frame 01: SP = 0x02B41DD0 PC = 0x014E9AE8
Frame 02: SP = 0x02B41DF0 PC = 0x016D76A4
Frame 03: SP = 0x02B41E18 PC = 0x00B41FF0
Frame 04: SP = 0x02B41EB8 PC = 0x00B41618
Frame 05: SP = 0x02B41EF8 PC = 0x00B3B744
Frame 06: SP = 0x02B41F38 PC = 0x00B3AB34
Frame 07: SP = 0x02B41F40 PC = 0x00A299C8
Frame 08: SP = 0x00000000 PC = 0x00A20498
07-13-2010 03:00 PM
Looks like an IOS-caused crash. Upgrade the IOS to 12.2.50-SE4.
07-13-2010 08:14 PM
> Re: c3560g-24ts random crash
> Looks like an IOS-caused crash. Upgrade the IOS to 12.2.50-SE4.
I have the same problem on 12.2(50)SE4 on the same platform yesterday. First it happened on 12.2(35)SE5, and then following upgrade, it happened again. Same "Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!" error.. the latest crash was after exiting configure terminal after changing line vty entries/acls associated with them.
Interestingly I also have a separate but simultaneous problem on another 3560 (which I'm trying to resolve with TAC). Both have been running fine for weeks until now; one of them for over a year. We ran rancid before, but as we saw some strange behaviour which we couldn't attribute to an operator, we've disabled it for now.
SL
07-14-2010 03:17 PM
I'm currently using 12.2(53)SE2 all across the fleet.
07-14-2010 03:29 PM
Have you checked out .... imagine you have, but just in case.
Debug Exception (Could be NULL pointer dereference)
A Catalyst 3560 Series Switch that runs Cisco IOS system software reloads with the
Debug Exception
(Could be NULL pointer dereference)
error message in the logs.
The probable causes for the error message are:
Memory leak in CEF background process. For information on how to resolve this issue, see
Unexpected memory consumption in CEF IPC Background process.
·
Powered device detection.
This issue occurs when the powered device is detected or classified as an
overcurrent class. This issue
is documented in Cisco bug ID CSCsa72400 (
registered customers only) .
Also...
https://cisco-support.hosted.jivesoftware.com/docs/DOC-4236?decorator=print
Good luck...
07-14-2010 04:08 PM
Thanks for the thoughts.. I have checked those before..
> Have you checked out .... imagine you have, but just in case.
I read that as (a) only affecting 3750s (although some documents mention 3560 too), (b) being related to PoE and this is a 24TS, not 24PS, and (c) I believe some documents mention it was fixed in 12.2(25)SEA and the oldest IOS we sae this in was 12.2(35)SE ..(I think just SE).. but certainly not 12.2(25)SEA.. although bug toolkit still suggests "Release-Pending".
Also some documentation suggests it happens on 3750s when (a) switches are stacked, (b) IP routing is disabled and (c) when the master switch it changed; Since it's 3560 it's not stacked, and IP routing is enabled so they seem to be contraindicated.
SL
07-15-2010 09:47 AM
thanks for the inputs guys. I looked over that link but from what i saw it did not apply to my 3560's. They dont do POE and the other mentions of 3560 dont really apply either. (though i did setup ntp for them now due to what i read on there about the times.)
there was no console/config connections to the switches that day either as only 2 of us do that here and i know neither of us were on teh switches.
it has not done anything since that day and has been stable so far (knock on wood)
i will look at updating it to a newer ios but where it is a prod switch i just can't update it anytime i want. i have a spare in place and will switch it out if it does it again and continue searching and monitoring it in the meantime.
jimmy i gave u some stars as i did find that time issue in your link and fixed that
07-15-2010 08:50 PM
Thank you, glad to of helped....I learn alot that way.
07-15-2010 09:26 PM
FYI TAC have suggested a bug to me which I have replicated twice causing a crash, so I'm confident that's causing it in my case.
If you're interested it's CSCth30052.
SL
07-19-2010 09:51 AM
thnx SL. I dont use IPv6 at all and was not playing with those settings. Not sure if it would crash it if I did but not going to attempt it right now.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide