Router reboot

amar_5664 · ‎12-14-2009

Guys,

One of my routers is having strange issue, it reboots itself after the clock updates through NTP source. Below is the log

*Dec 15 15:15:14.151 AEST: %SYS-6-CLOCKUPDATE: System clock has been updated fro
m 05:15:14 UTC Tue Dec 15 2009 to 15:15:14 AEST Tue Dec 15 2009, configured from
console by console.
*Dec 15 16:15:14.151 EDST: %SYS-6-CLOCKUPDATE: System clock has been updated fro
m 15:15:14 AEST Tue Dec 15 2009 to 16:15:14 EDST Tue Dec 15 2009, configured fro
m console by console.
*Dec 15 16:15:15.123 EDST: %SYS-5-CONFIG_I: Configured from memory by console
*Dec 15 16:15:15.596 EDST: %SYS-5-RESTART: System restarted --
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(18b)
, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Mon 19-May-08 15:43 by prod_rel_team
*Dec 15 16:15:15.600 EDST: %SNMP-5-COLDSTART: SNMP agent on host AVS-RR-01 is un
dergoing a cold start

There are about 30 other routers in our network feeding on same NTP with similar clock configuration and doesnt have any issue but this one does, any assistance would be appreciated.

Leo Laohoo · ‎12-15-2009

Can you provide more information?

1. IOS version;
2. Hardware;
3. Relevant information;
4. Router configuration

Does the router have a crashinfo file/subdirectory?

amar_5664 · ‎12-15-2009

Hey mate,

Router does not have crashfile subdirectory...version is in the log above....

Cisco 2821 (revision 53.51) with 249856K/12288K bytes of memory.
2 Gigabit Ethernet interfaces
2 Serial interfaces
2 Channelized E1/PRI ports
1 Virtual Private Network (VPN) Module

anyways i have monitored the router overnight and havent had any issues as of yet...

Leo Laohoo · ‎12-15-2009

It's hard to determine what the cause is if we don't see the crashinfo file or see the traceback errors.

Ganesh Hariharan · ‎12-21-2009

Hi Amar,

Please check the observation for your error logs

%SYS-5-CONFIG_I (x1): Configured from [chars] by [chars]

Explanation: The router configuration has been changed.

Recommended Action: This is a notification message only. No action is required.
%SYS-5-RESTART (x1): System restarted -- [chars]

Explanation: A request to reload or restart the system has been made.

Recommended Action: This is an informational message only. No action is required.
%SNMP-5-COLDSTART (x1): Cold Start Trap

Explanation: This message indicates that the sending protocol entity is reinitializing
itself so the configuration of the agent or the protocol entity implementation
might be altered.

Recommended Action: No action is necessary. This message is provided for information
only.
%SNMP-5-COLDSTART (x0): SNMP agent on host [chars] is undergoing a cold start

Explanation: The SNMP server has undergone a cold start.

Recommended Action: This is a notification message only. No action is required.
%SNMP-5-COLDSTART (x0): Cold Start Trap

Explanation: This message indicates that the sending protocol entity is reinitializing
itself so the configuration of the agent or the protocol entity implementation
might be altered.

Recommended Action: No action is necessary. This message is provided for information
only.
%SYS-6-CLOCKUPDATE (x2): System clock has been updated from [chars] to [chars],
configured from [chars] by [chars].

Explanation: The system clock has been modified.

Recommended Action: This is an informational message only. No action is required.
%SYS-6-CLOCKUPDATE (x0): System clock has been updated from [chars] to [chars],
configured from [chars] by [chars].

Explanation: The system clock has been modified.

Recommended Action: This is an informational message only. No action is required.
%SYS-6-CLOCKUPDATE (x0): System clock has been updated from [chars] to [chars],
configured from [chars] by [chars].

Explanation: The system clock has been modified.

Recommended Action: This is an informational message only. No action is required.

What is the IOS version and what exactly happen when router synced with ntp server.

Regards

Ganesh.H

hcgov · ‎05-11-2010

We are having the same issue with a Cisco 2821 with the following IOS and Hardware Configuration:

Cisco IOS Software, 2800 Software (C2800NM-ENTBASEK9-M), Version 12.4(12a), RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2007 by Cisco Systems, Inc. Compiled Thu 22-Feb-07 17:13 by prod_rel_team ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1) dorsey uptime is 15 minutes System returned to ROM by power-on System restarted at 06:50:41 EDT Tue May 11 2010 System image file is "flash:c2800nm-entbasek9-mz.124-12a.bin" Cisco 2821 (revision 53.51) with 251904K/10240K bytes of memory. 2 Gigabit Ethernet interfaces DRAM configuration is 64 bits wide with parity enabled. 239K bytes of non-volatile configuration memory. 62720K bytes of ATA CompactFlash (Read/Write)

This router is getting login information from an ACS and NTP from 2 sources.

When reviewing the log. the time stamp is incorrect even though the time on the router is correct.

Any comments on a cause and solution would be appreciated.

Leo Laohoo · ‎05-11-2010

dorsey uptime is 15 minutes

System returned to ROM by power-on

System restarted at 06:50:41 EDT Tue May 11 2010

System image file is "flash:c2800nm-entbasek9-mz.124-12a.bin"

I don't see anything suspicious here.

When reviewing the log. the time stamp is incorrect even though the time on the router is correct.

After a reboot, the appliance will always go to default of "*Mar 1" until the appliance syncs with the NTP source.

Zong Ping Lin · ‎02-27-2012

Hi I am having the same issue... the core router reboot itself.

1. IOS version; Version 12.2(25)EW

2. Hardware; Catalyst 4000 L3 Switch

Last powerfail: never

Current time: 11/17/2011 09:23:14

Last crash: 11/17/2011 08:47:32

Build: 12.2(25)EW
buildversion addr: 11F04C1C

pc=11053EFC lr=11053EFC msr=00029030 vector=00000700
cr=82000044 ctr=11510F98 xer=00000000
r0=11053EFC r1=1247F0C0 r2=0000C000 r3=00000047
r4=00000020 r5=00000000 r6=1247F088 r7=00000001
r8=FFFFFFFF r9=1247F147 r10=1247F020 r11=00000300
r12=82000049 r13=FFFFFFFF r14=10396180 r15=00000000
r16=00000000 r17=11F10000 r18=00000001 r19=00000001
r20=11F10000 r21=12090000 r22=11E044A4 r23=11F54CB0
r24=11F548B0 r25=1247F204 r26=1247F1C0 r27=1247F0CC
r28=11F04A90 r29=1247F0E0 r30=1247F1E0 r31=15B7FAFC
dec=00037238 tbu=000221F0 tbl=4A3FC8A7
dar=00000000 dsisr=00000000 hid0=8000C000

ErrDr1=00 ErrDr2=00 ErrStsCpu=32 ErrStsPci=00
BusErrAddr=E8000900
Single bit ECC count 0

Traceback: 11053EFC 1158A2C4 1105F020 115217C4 1152161C 103C5DB4 103C43FC 10005428 11507578 115B20F0 10FD8D14 10FD8D7C 10395EDC 1039607C 103962B0 10309FF0 103042DC

Stack frames:
Frame 1: pc=1158A2C4 stack=1247F190
Frame 2: pc=1105F020 stack=1247F198
Frame 3: pc=115217C4 stack=1247F228
Frame 4: pc=1152161C stack=1247F238
Frame 5: pc=103C5DB4 stack=1247F240
Frame 6: pc=103C43FC stack=11F56CF8
Frame 7: pc=10005428 stack=11F56D30
Frame 8: pc=11507578 stack=12DB6B98
Frame 9: pc=115B20F0 stack=12DB6BC8
Frame 10: pc=10FD8D14 stack=12DB6BD0
Frame 11: pc=10FD8D7C stack=12DB6BD8
Frame 12: pc=10395EDC stack=12DB6BE0
Frame 13: pc=1039607C stack=12DB6C38
Frame 14: pc=103962B0 stack=12DB6C58
Frame 15: pc=10309FF0 stack=12DB6C88
Frame 16: pc=103042DC stack=12DB6C90

Leo Laohoo · ‎02-28-2012

Sorry Zong, your information posted is not enough. Do you have a crashinfo file(s)? If you do, attach the latest 3 files.

Zong Ping Lin · ‎03-06-2012

Hi Leo,

I only have the show tech.

Anything i info from tech that can help?

Thanks.

Joseph W. Doherty · ‎02-29-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

leolaohoo wrote:

After a reboot, the appliance will always go to default of "*Mar 1" until the appliance syncs with the NTP source.

Some Cisco devices also support an on-board calendar (I think this includes the 2800 series).

If you keep the device's clock sync'ed with NTP, you can keep the calendar sync'ed too using ntp update-calendar.

If such devices with a calendar are restarted, the log will have pretty accurate timestamps until NTP syncs up again.

nkarpysh · ‎03-06-2012

Hi Zong,

Your crash is not related to the one discussed in this thread. Your crash is due to parity error and DDTS which force switch to relaod on parity error occurrence,

DDTS is:

Csa75881 WS-X4515 crashed at onNonFatalParityInterrupt__12K2NetflowMan

Symptom:

Customer supervisor crashes when it sees any NFL parity errors (Fatal or Non-

fatal)

Workaround:

It has been fixed by distinguishing between Fatal and Non-Fatal parity errors.

The new behavior is:

[Non Fatal error]

-The system does not reset for Non-fatal error unless it encounters a number

of non-fatal error that is higher than what we expect to see.

-The system will display following log message every time it sees a non-fatal

error.

"Non fatal NFL Error [NFL] [FDT1] [FDT2] [FLD]"

-But if the system a higher number of non-fatal error than the number we

expect to encounter, then system will reset itself.

[Fatal error]

-The system will reset itself right after displaying following log, if the

system encounters even a single fatal error.

"Fatal NFL Error [NFL] [FDT1] [FDT2] [FLD]"

Recomendation is to upgrade to 12.2(25)EWA2.

AS for a crash for which thread was opened - it seems to be not related to NTP synch - somthing else might have happened triggering it. It seems that NTP was trying to synch after reload as the logs you have are the ones during bootup. All logs before reload should have been cleared. I guess in the "show version" you have smth like "System returned to ROM by power-on".

That would mean that either someone on site power cycle it accidently or on purpose, there was a power outage for that server room or problem just with single power outlet - or switch has a power controller issue. I would recomend to monitor it.

Nik

HTH,
Niko

Zong Ping Lin · ‎03-08-2012

Thanks Nik,

Will post in the correct place in future.