Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

unplanned switch reboots since installing Prime 2.0

I'm running LMS 4.2, and decided to try an evaluation copy of Prime 2.0 and run them side by side.

Initial installation and discovery completed, I left all settings at default (I didn't set up alerting, email, etc; I wanted to use it as a test, not deploy it in production)

For the last few nights, 2 of my 3560G switches have been rebooting; sometimes just one, sometimes both AND at the same time every night, 10 p.m.. The crashinfo output doesn't show any details, and the Cisco output interpreter didn't help.

Setting the switches to syslog to another device, I discover a message "configured from SNMP by (Prime IP address) at 10:00 on every switch; I haven't set any tasks up on Prime, and I don't understand what this is, but it is far too much of a coincidence to ignore.

Both switches are running an older code,

c3560-ipbase-mz.122-25.SEB4.bin

and prior to this have run for 2 years continuously.

Any ideas, anyone?

SR1-8-3560 uptime is 1 day, 11 hours, 48 minutes

System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C

System restarted at 22:01:23 GMT Tue Nov 26 2013

System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"

SR1-7-3560 uptime is 11 hours, 51 minutes

System returned to ROM by error - Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!, PC 0x75707C

System restarted at 22:01:03 GMT Wed Nov 27 2013

System image file is "flash:c3560-ipbase-mz.122-25.SEB4/c3560-ipbase-mz.122-25.SEB4.bin"

Message was edited by: Gareth Tomlinson

1 ACCEPTED SOLUTION

Accepted Solutions
Hall of Fame Super Gold

unplanned switch reboots since installing Prime 2.0

Any ideas, anyone?

I'd be thinking about upgrading your IOS.  Maybe the IOS of your switch has incompatibility.  After all, it is old.

10 REPLIES
VIP Purple

Re: unplanned switch reboots since installing Prime 2.0

Hi Gareth,

Can u please paste the crash info file from switch which reboots.

Regards

Sent from Cisco Technical Support iPhone App

New Member

unplanned switch reboots since installing Prime 2.0

SR1-8-3560#more crashinfo_2

Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2005 by Cisco Systems, Inc.

Compiled Tue 30-Aug-05 14:19 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x004A54F4  SRR1 = 0x00029210  SRR2 = 0x004A3F68  SRR3 = 0x00020000

ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:

Vector = 0x00002000  PC = 0x0075707C  MSR = 0x00029210  CR = 0x33000005

LR = 0x00756FA4  CTR = 0x001C1794  XER = 0xC0000050

R0 = 0x00000000  R1 = 0x02F71EF8  R2 = 0x00000000  R3 = 0x00000000

R4 = 0xFFFFFFFE  R5 = 0x00000000  R6 = 0x02F71ED0  R7 = 0x00000000

R8 = 0x00029210  R9 = 0x01240000  R10 = 0x01552868  R11 = 0x00000000

R12 = 0x0000001C  R13 = 0x00110000  R14 = 0x00000000  R15 = 0x00000200

R16 = 0x03136940  R17 = 0x01240000  R18 = 0x0315A220  R19 = 0x00000003

R20 = 0x00000001  R21 = 0x00FCB59C  R22 = 0x00000000  R23 = 0x0176F7D8

R24 = 0x00000000  R25 = 0x009C902C  R26 = 0x00000001  R27 = 0x03048078

R28 = 0x00000001  R29 = 0x0133BBA8  R30 = 0x00000000  R31 = 0x01144EE8

Stack trace:

PC = 0x0075707C, SP = 0x02F71EF8

Frame 00: SP = 0x02F71F08    PC = 0x00756F7C

Frame 01: SP = 0x02F71F38    PC = 0x00BF914C

Frame 02: SP = 0x02F71F68    PC = 0x009C9030

Frame 03: SP = 0x02F72088    PC = 0x009C91D8

Frame 04: SP = 0x02F720C8    PC = 0x009CAB34

Frame 05: SP = 0x02F72200    PC = 0x00833A20

Frame 06: SP = 0x02F72220    PC = 0x00834464

Frame 07: SP = 0x02F72298    PC = 0x0086C494

Frame 08: SP = 0x02F722A0    PC = 0x00758DA8

Frame 09: SP = 0x00000000    PC = 0x00752FEC

SR1-8-3560#more crashinfo_1

Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB4, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2005 by Cisco Systems, Inc.

Compiled Tue 30-Aug-05 14:19 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x004A54F4  SRR1 = 0x00029210  SRR2 = 0x004A3F6C  SRR3 = 0x00021000

ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x8C000000  DBSR = 0x10000000

CPU Register Context:

Vector = 0x00002000  PC = 0x0075707C  MSR = 0x00029210  CR = 0x33000005

LR = 0x00756FA4  CTR = 0x001C1794  XER = 0xC0000050

R0 = 0x00000000  R1 = 0x03194C18  R2 = 0x00000000  R3 = 0x00000000

R4 = 0xFFFFFFFE  R5 = 0x00000000  R6 = 0x03194BF0  R7 = 0x00000000

R8 = 0x00029210  R9 = 0x01240000  R10 = 0x01552868  R11 = 0x00000000

R12 = 0x0000001C  R13 = 0x00110000  R14 = 0x00000000  R15 = 0x00000200

R16 = 0x0314ABF0  R17 = 0x01240000  R18 = 0x034112E0  R19 = 0x00000003

R20 = 0x00000001  R21 = 0x00FCB59C  R22 = 0x00000000  R23 = 0x0176F7D8

R24 = 0x00000000  R25 = 0x009C902C  R26 = 0x00000001  R27 = 0x030E3280

R28 = 0x00000001  R29 = 0x0133BBA8  R30 = 0x00000000  R31 = 0x01144EE8

Stack trace:

PC = 0x0075707C, SP = 0x03194C18

Frame 00: SP = 0x03194C28    PC = 0x00756F7C

Frame 01: SP = 0x03194C58    PC = 0x00BF914C

Frame 02: SP = 0x03194C88    PC = 0x009C9030

Frame 03: SP = 0x03194DA8    PC = 0x009C91D8

Frame 04: SP = 0x03194DE8    PC = 0x009CAB34

Frame 05: SP = 0x03194F20    PC = 0x00833A20

Frame 06: SP = 0x03194F40    PC = 0x00834464

Frame 07: SP = 0x03194FB8    PC = 0x0086C494

Frame 08: SP = 0x03194FC0    PC = 0x00758DA8

Frame 09: SP = 0x00000000    PC = 0x00752FEC

VIP Purple

Re: unplanned switch reboots since installing Prime 2.0

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

HI Gareth,

A Cisco Catalyst 3560 switch that runs Cisco IOS  Software Release 12.2(20)SE4 or later reloads or resets with the Debug Exception (Could be NULL pointer dereference) Exception (0x2000)! error message.

This problem occurs when the Powered Device (PD) is detected or classified as an overcurrent class.

This issue is documented in Cisco bug ID CSCsa72400.

As a workaround, do not connect 802.3af non-standard class PDs, or even bad or loopback cables, to the switch because the switch can detect the class incorrectly.

In order to resolve this issue, upgrade to Cisco IOS Software Release 12.2(25)SEA on the switch. Alternatively, upgrade to the latest maintenance release, which can be downloaded from Cisco Downloads..

https://supportforums.cisco.com/docs/DOC-4236

Regards

Dont forget to rate helpful posts.

New Member

Re: unplanned switch reboots since installing Prime 2.0

unfortunately it isn't a PoE switch; and it's supplied from a APC UPS and a RPS simultaneously for mains power.

Additionally, why would it happen at exactly the same time every day?

Please believe me, I'm very grateful for all these suggestions but I don't think this is  the issue.

Gareth

Cisco Employee

Re: unplanned switch reboots since installing Prime 2.0

If you aren't able to get any hints from Sandeep's suggestion::

You might go to Administration> Background tasks and look for tasks that run for switches - There are only 4 or 5 or so then got through a process of elimination by disabling certain jobs to try to narrow down what might be causing the crash-

New Member

unplanned switch reboots since installing Prime 2.0

I've killed the server completely, I can't afford to have switches rebooting randomly. No problem tonmight, I'll let it go another few nights before I'm confident that it WAS Prime, though.

Hall of Fame Super Gold

unplanned switch reboots since installing Prime 2.0

Any ideas, anyone?

I'd be thinking about upgrading your IOS.  Maybe the IOS of your switch has incompatibility.  After all, it is old.

New Member

unplanned switch reboots since installing Prime 2.0

I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days

Hall of Fame Super Gold

Re: unplanned switch reboots since installing Prime 2.0

I've upgraded one, the trouble is so many clusters spread across the infrastructure that I have to be careful what switch I can take out of service; Killing Prime temporarily seems to have cured the problem, but I'll confirm that after a few more days

Here's my dig of what is going on. 

1.  12.2(25)SEB is one of the notorious IOS which a lot of CPU hogs issue. 

2.  10:00 pm maybe the start of the LMS 4.2 to start collecting data from your switches, however, you may have also some clients doing high-traffic work at that time as well (like backups).  Put those two together, add a dash of CPU hog and you have a fine dish of "crash" the next morning.  Served warm. 

I'm not sure if the developers of PI 2.0 to have switches running IOS this old but I doubt if they have factored that in into the coding.

New Member

Re: unplanned switch reboots since installing Prime 2.0

Hi Leo

No problems last night, so I suspect you are on the money here. Yes, there are backups running at that time across the network, every night.

I am wondering what process runs at 10 p.m on Prime, I haven't been able to identify it yet.

Gareth

633
Views
0
Helpful
10
Replies
CreatePlease to create content