Solved: Re: unplanned switch reboots since installing Prime 2.0

Gareth Tomlinson · ‎11-28-2013

I'm running LMS 4.2, and decided to try an evaluation copy of Prime 2.0 and run them side by side.

Initial installation and discovery completed, I left all settings at default (I didn't set up alerting, email, etc; I wanted to use it as a test, not deploy it in production)

For the last few nights, 2 of my 3560G switches have been rebooting; sometimes just one, sometimes both AND at the same time every night, 10 p.m.. The crashinfo output doesn't show any details, and the Cisco output interpreter didn't help.

Setting the switches to syslog to another device, I discover a message "configured from SNMP by (Prime IP address) at 10:00 on every switch; I haven't set any tasks up on Prime, and I don't understand what this is, but it is far too much of a coincidence to ignore.

Both switches are running an older code,

c3560-ipbase-mz.122-25.SEB4.bin

and prior to this have run for 2 years continuously.

Any ideas, anyone?

SR1-8-3560 uptime is 1 day, 11 hours, 48 minutes

System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C

System restarted at 22:01:23 GMT Tue Nov 26 2013

System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"

SR1-7-3560 uptime is 11 hours, 51 minutes

System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C

System restarted at 22:01:03 GMT Wed Nov 27 2013

System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"

Message was edited by: Gareth Tomlinson

Leo Laohoo · ‎11-28-2013

Any ideas, anyone?

I'd be thinking about upgrading your IOS. Maybe the IOS of your switch has incompatibility. After all, it is old.

View solution in original post

Sandeep Choudhary · ‎11-28-2013

Hi Gareth,

Can u please paste the crash info file from switch which reboots.

Regards

Sent from Cisco Technical Support iPhone App

Gareth Tomlinson · ‎11-28-2013

SR1-8-3560#more crashinfo_2

Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)

Compiled Tue 30-Aug-05 14:19 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x004A54F4 SRR1 = 0x00029210 SRR2 = 0x004A3F68 SRR3 = 0x00020000

ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000

CPU Register Context:

Vector = 0x00002000 PC = 0x0075707C MSR = 0x00029210 CR = 0x33000005

LR = 0x00756FA4 CTR = 0x001C1794 XER = 0xC0000050

R0 = 0x00000000 R1 = 0x02F71EF8 R2 = 0x00000000 R3 = 0x00000000

R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x02F71ED0 R7 = 0x00000000

R8 = 0x00029210 R9 = 0x01240000 R10 = 0x01552868 R11 = 0x00000000

R12 = 0x0000001C R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000200

R16 = 0x03136940 R17 = 0x01240000 R18 = 0x0315A220 R19 = 0x00000003

R20 = 0x00000001 R21 = 0x00FCB59C R22 = 0x00000000 R23 = 0x0176F7D8

R24 = 0x00000000 R25 = 0x009C902C R26 = 0x00000001 R27 = 0x03048078

R28 = 0x00000001 R29 = 0x0133BBA8 R30 = 0x00000000 R31 = 0x01144EE8

Stack trace:

PC = 0x0075707C, SP = 0x02F71EF8

Frame 00: SP = 0x02F71F08 PC = 0x00756F7C

Frame 01: SP = 0x02F71F38 PC = 0x00BF914C

Frame 02: SP = 0x02F71F68 PC = 0x009C9030

Frame 03: SP = 0x02F72088 PC = 0x009C91D8

Frame 04: SP = 0x02F720C8 PC = 0x009CAB34

Frame 05: SP = 0x02F72200 PC = 0x00833A20

Frame 06: SP = 0x02F72220 PC = 0x00834464

Frame 07: SP = 0x02F72298 PC = 0x0086C494

Frame 08: SP = 0x02F722A0 PC = 0x00758DA8

Frame 09: SP = 0x00000000 PC = 0x00752FEC

SR1-8-3560#more crashinfo_1

Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)

Compiled Tue 30-Aug-05 14:19 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x004A54F4 SRR1 = 0x00029210 SRR2 = 0x004A3F6C SRR3 = 0x00021000

ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000

CPU Register Context:

Vector = 0x00002000 PC = 0x0075707C MSR = 0x00029210 CR = 0x33000005

LR = 0x00756FA4 CTR = 0x001C1794 XER = 0xC0000050

R0 = 0x00000000 R1 = 0x03194C18 R2 = 0x00000000 R3 = 0x00000000

R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x03194BF0 R7 = 0x00000000

R8 = 0x00029210 R9 = 0x01240000 R10 = 0x01552868 R11 = 0x00000000

R12 = 0x0000001C R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000200

R16 = 0x0314ABF0 R17 = 0x01240000 R18 = 0x034112E0 R19 = 0x00000003

R20 = 0x00000001 R21 = 0x00FCB59C R22 = 0x00000000 R23 = 0x0176F7D8

R24 = 0x00000000 R25 = 0x009C902C R26 = 0x00000001 R27 = 0x030E3280

R28 = 0x00000001 R29 = 0x0133BBA8 R30 = 0x00000000 R31 = 0x01144EE8

Stack trace:

PC = 0x0075707C, SP = 0x03194C18

Frame 00: SP = 0x03194C28 PC = 0x00756F7C

Frame 01: SP = 0x03194C58 PC = 0x00BF914C

Frame 02: SP = 0x03194C88 PC = 0x009C9030

Frame 03: SP = 0x03194DA8 PC = 0x009C91D8

Frame 04: SP = 0x03194DE8 PC = 0x009CAB34

Frame 05: SP = 0x03194F20 PC = 0x00833A20

Frame 06: SP = 0x03194F40 PC = 0x00834464

Frame 07: SP = 0x03194FB8 PC = 0x0086C494

Frame 08: SP = 0x03194FC0 PC = 0x00758DA8

Frame 09: SP = 0x00000000 PC = 0x00752FEC

Sandeep Choudhary · ‎11-28-2013

 
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

HI Gareth,

A Cisco Catalyst 3560 switch that runs Cisco IOS Software Release 12.2(20)SE4 or later reloads or resets with the Debug Exception (Could be NULL pointer dereference) Exception (0x2000)! error message.

This problem occurs when the Powered Device (PD) is detected or classified as an overcurrent class.

This issue is documented in Cisco bug ID CSCsa72400.

As a workaround, do not connect 802.3af non-standard class PDs, or even bad or loopback cables, to the switch because the switch can detect the class incorrectly.

In order to resolve this issue, upgrade to Cisco IOS Software Release 12.2(25)SEA on the switch. Alternatively, upgrade to the latest maintenance release, which can be downloaded from Cisco Downloads..

https://supportforums.cisco.com/docs/DOC-4236

Regards

Dont forget to rate helpful posts.

Gareth Tomlinson · ‎11-29-2013

unfortunately it isn't a PoE switch; and it's supplied from a APC UPS and a RPS simultaneously for mains power.

Additionally, why would it happen at exactly the same time every day?

Please believe me, I'm very grateful for all these suggestions but I don't think this is the issue.

Gareth

Rob Johnson · ‎11-28-2013

If you aren't able to get any hints from Sandeep's suggestion::

You might go to Administration> Background tasks and look for tasks that run for switches - There are only 4 or 5 or so then got through a process of elimination by disabling certain jobs to try to narrow down what might be causing the crash-

Gareth Tomlinson · ‎11-28-2013

I've killed the server completely, I can't afford to have switches rebooting randomly. No problem tonmight, I'll let it go another few nights before I'm confident that it WAS Prime, though.

Leo Laohoo · ‎11-28-2013

Any ideas, anyone?

I'd be thinking about upgrading your IOS. Maybe the IOS of your switch has incompatibility. After all, it is old.

Gareth Tomlinson · ‎11-28-2013

I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days

Leo Laohoo · ‎11-28-2013

I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days

Here's my dig of what is going on.

1. 12.2(25)SEB is one of the notorious IOS which a lot of CPU hogs issue.

2. 10:00 pm maybe the start of the LMS 4.2 to start collecting data from your switches, however, you may have also some clients doing high-traffic work at that time as well (like backups). Put those two together, add a dash of CPU hog and you have a fine dish of "crash" the next morning. Served warm.

I'm not sure if the developers of PI 2.0 to have switches running IOS this old but I doubt if they have factored that in into the coding.

Gareth Tomlinson · ‎11-29-2013

Hi Leo

No problems last night, so I suspect you are on the money here. Yes, there are backups running at that time across the network, every night.

I am wondering what process runs at 10 p.m on Prime, I haven't been able to identify it yet.

Gareth