Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

Two SCE boxes has died after upgrade to FW 3.6 (but not immediatelly)

I have experienced SCE error with following switching to System Recovery Mode on two SCE 2000 boxes. I upgraded them two days ago but today they switched to Recovery Mode after unexpected platform reload. Difference between these two boxes reload/crash is only 20 seconds.

Log shows:

SCE2000#>show log | include 10-04-29
2010-04-29 19:27:01 | WARN  | CPU #000 | Not enough disk space.
2010-04-29 19:27:01 | WARN  | CPU #000 | SE Control Module: A problem occurred. Please report to Cisco's customer support
2010-04-29 19:27:01 | WARN  | CPU #000 | The SE status changed to Warning
2010-04-29 19:27:01 | WARN  | CPU #000 | Available disk space at '/tffs0' is 44MB which is under 45MB. Some management operations may fail to execute.
2010-04-29 19:29:56 | INFO  | CPU #000 | A SSH session from 10.48.64.244 was established.
2010-04-29 19:29:56 | INFO  | CPU #000 | File system operation : del  /tffs0/iwf.txt
2010-04-29 19:29:57 | INFO  | CPU #000 | Disk space is OK
2010-04-29 19:29:57 | INFO  | CPU #000 | The SE status changed to Operational
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx9
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx8
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx7
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx6
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx5
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx4
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx3
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx2
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx1
2010-04-29 19:30:16 | INFO  | CPU #000 | File system operation : copy /tffs0/system/config.txt /tffs0/system/prevconf/config.tx1
2010-04-29 19:30:16 | INFO  | CPU #000 | Configuration file '/tffs0/system/config.txt' was saved - files size = 4228.
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx9
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx8
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx7
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx6
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx5
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx4
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx3
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx2
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : del  /tffs0/system/prevconf/config.tx1
2010-04-29 19:30:18 | INFO  | CPU #000 | File system operation : copy /tffs0/system/config.txt /tffs0/system/prevconf/config.tx1
2010-04-29 19:30:18 | INFO  | CPU #000 | Configuration file '/tffs0/system/p3hidden/config/confX1.txt' was saved - files size = 443.
2010-04-29 19:30:24 | ERROR | CPU #000 | SE Watchdog Module: An Error occurred. Please report to Cisco's customer support
2010-04-29 19:30:24 | ERROR | CPU #000 | SE Watchdog Module: An Error occurred. Please report to Cisco's customer support
2010-04-29 19:30:24 | WARN  | CPU #000 | SE Watchdog Module: A problem occurred. Please report to Cisco's customer support
2010-04-29 19:30:24 | INFO  | CPU #000 | Party data base was closed.
******************** User Log File - started at 2010-04-29 17:33:35  - Version 3.6.0 Build 716 ********************
2010-04-29 17:33:35 | INFO  | CPU #000 | SCE Application main process starting init phase.
2010-04-29 17:33:35 | INFO  | CPU #000 | Set connection-mode: connection mode is: inline, box priority is: primary, fail-over mode is: bypass, physically-connected link group is: 0.
2010-04-29 17:33:35 | INFO  | CPU #000 | Flow data-base (FC) initialized
2010-04-29 17:33:35 | INFO  | CPU #000 | RuC specific initialization completed
2010-04-29 17:33:35 | INFO  | CPU #000 | General RuC initialization completed
2010-04-29 17:33:35 | INFO  | CPU #000 | System clock set to 17:33:35  UTC  THU  April  29  2010.
2010-04-29 17:33:35 | INFO  | CPU #000 | Logger task Initialized successfully
2010-04-29 17:33:36 | INFO  | CPU #001 | SCE Application main process starting init phase.
2010-04-29 17:33:36 | INFO  | CPU #001 | Flow data-base (FC) initialized
2010-04-29 17:33:36 | INFO  | CPU #001 | RuC specific initialization completed
2010-04-29 17:33:36 | INFO  | CPU #001 | General RuC initialization completed
2010-04-29 17:33:36 | INFO  | CPU #002 | SCE Application main process starting init phase.
2010-04-29 17:33:36 | INFO  | CPU #002 | Flow data-base (FC) initialized
2010-04-29 17:33:36 | INFO  | CPU #002 | RuC specific initialization completed
2010-04-29 17:33:36 | INFO  | CPU #002 | General RuC initialization completed
2010-04-29 17:33:36 | INFO  | CPU #003 | SCE Application main process starting init phase.
2010-04-29 17:33:36 | INFO  | CPU #003 | Flow data-base (FC) initialized
2010-04-29 17:33:36 | INFO  | CPU #003 | RuC specific initialization completed
2010-04-29 17:33:36 | INFO  | CPU #003 | General RuC initialization completed
2010-04-29 17:33:41 | INFO  | CPU #000 | System clock set to 17:33:41  UTC  THU  April  29  2010.
2010-04-29 17:33:41 | INFO  | CPU #000 | Subscriber data-base manager initialized successfully.
2010-04-29 18:33:44 | INFO  | CPU #000 | Time zone set to CET 60 minutes.
2010-04-29 18:34:02 | INFO  | CPU #000 | state changed from INIT to RECOVERY.
2010-04-29 18:34:22 | INFO  | CPU #000 | SM: DHCP LEASE QUERY LEG 3.6.0 Build 429 started
2010-04-29 18:34:31 | INFO  | CPU #000 | EM Agent: EM Agent started. Version: SCE Agent 3.6.0 Build 429
2010-04-29 18:34:31 | INFO  | CPU #000 | Management agent is up.
2010-04-29 18:34:31 | INFO  | CPU #000 | EM Agent: SM operational state switched from Initialization to Active.
2010-04-29 18:34:31 | INFO  | CPU #000 | Beginning execution of configuration files.
2010-04-29 18:34:32 | INFO  | CPU #000 | System hostname changed to 'sce-2'.
2010-04-29 18:34:32 | INFO  | CPU #000 | A new password was set for level 0.
2010-04-29 18:34:32 | INFO  | CPU #000 | A new password was set for level 10.
2010-04-29 18:34:32 | INFO  | CPU #000 | A new password was set for level 15.
2010-04-29 19:34:33 | INFO  | CPU #000 | Time zone set to CET 60 minutes.
2010-04-29 19:34:33 | INFO  | CPU #000 | SM: RDR Server has been started.
2010-04-29 19:34:33 | INFO  | CPU #000 | EM Agent: PRPC server started on port 14374.
2010-04-29 19:34:33 | INFO  | CPU #000 | Set connection-mode: connection mode is: inline-cascade, box priority is: secondary, fail-over mode is: bypass, physically-connected link group is: 1.
2010-04-29 19:34:33 | INFO  | CPU #000 | state changed from INIT to BOOT.
2010-04-29 19:34:33 | INFO  | CPU #000 | Formatter connection opened: address 127.0.0.1, port 33001.
2010-04-29 19:34:33 | INFO  | CPU #000 | SM: Connection from 127.0.0.1:1025 opened.
2010-04-29 19:34:34 | INFO  | CPU #000 | Formatter active connection opened : address 127.0.0.1, port 33001, category 3.
2010-04-29 19:34:36 | INFO  | CPU #000 | Starting Line Card on slot 0 state change to shutdown
2010-04-29 19:34:36 | INFO  | CPU #000 | Linecard on slot 0 is disabled
2010-04-29 19:34:39 | INFO  | CPU #000 | Formatter connection opened: address 10.48.64.78, port 33000.
2010-04-29 19:34:39 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 0.
2010-04-29 19:34:39 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 1.
2010-04-29 19:34:39 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 2.
2010-04-29 19:34:39 | INFO  | CPU #000 | Starting Line Card on slot 0 state change to no shutdown
2010-04-29 19:34:39 | INFO  | CPU #000 | Linecard on slot 0 is enabled
2010-04-29 19:34:40 | ERROR | CPU #000 | Application configuration file executed with 390 errors.
2010-04-29 19:34:41 | INFO  | CPU #000 | SSH server started.
2010-04-29 19:34:41 | WARN  | CPU #000 | Link on management port #1 is down.
2010-04-29 19:34:41 | INFO  | CPU #000 | The SE status changed to Operational
2010-04-29 19:34:45 | INFO  | CPU #000 | EM Agent: PRPC connection to /10.48.64.113 established, client name is mcn.QuotaManager-10.48.64.74_10.48.64.113_SCE.J.API.PRPC, session name is rpc-0.
2010-04-29 19:34:45 | INFO  | CPU #000 | EM Agent: QuotaManager-10.48.64.74_10.48.64.113_SCE.J.API.PRPC - connect operation was called, registered listeners: quota
2010-04-29 19:34:51 | INFO  | CPU #000 | EM Agent: PRPC connection to /10.48.64.113 established, client name is usr.SM, session name is rpc-1.
2010-04-29 19:34:51 | INFO  | CPU #000 | EM Agent: PRPC connection to /10.48.64.113 established, client name is mcn.SM, session name is rpc-2.
2010-04-29 19:34:52 | INFO  | CPU #000 | Redundancy: SM connection status changed from NA to UP.
2010-04-29 19:34:52 | INFO  | CPU #000 | Subscriber Manager connection is up.
2010-04-29 19:34:52 | INFO  | CPU #000 | EM Agent: SM connection established.
2010-04-29 19:34:52 | INFO  | CPU #000 | EM Agent: VPN synchronization started.
2010-04-29 19:34:52 | INFO  | CPU #000 | EM Agent: VPN synchronization finished successfully.
2010-04-29 19:35:32 | INFO  | CPU #000 | state changed from BOOT to RECOVERY.
2010-04-29 19:35:33 | INFO  | CPU #000 | state changed from RECOVERY to STANDALONE.
2010-04-29 19:35:33 | INFO  | CPU #000 | EM Agent: VPN synchronization started.
2010-04-29 19:35:33 | INFO  | CPU #000 | EM Agent: VPN synchronization finished successfully.
2010-04-29 19:35:33 | INFO  | CPU #000 | EM Agent: Subscriber synchronization started.
2010-04-29 19:35:33 | INFO  | CPU #000 | EM Agent: Subscriber synchronization finished successfully.
2010-04-29 19:36:58 | WARN  | CPU #000 | Not enough disk space.
2010-04-29 19:36:58 | WARN  | CPU #000 | SE Control Module: A problem occurred. Please report to Cisco's customer support
2010-04-29 19:36:58 | WARN  | CPU #000 | The SE status changed to Warning
2010-04-29 19:36:58 | WARN  | CPU #000 | Available disk space at '/tffs0' is 44MB which is under 45MB. Some management operations may fail to execute.
2010-04-29 19:45:07 | INFO  | CPU #000 | Formatter connection closed: address 10.48.64.78, port 33000.
2010-04-29 19:45:07 | INFO  | CPU #000 | No Formatter connections is open on category 0.
2010-04-29 19:45:07 | INFO  | CPU #000 | No Formatter connections is open on category 1.
2010-04-29 19:45:07 | INFO  | CPU #000 | No Formatter connections is open on category 2.
2010-04-29 19:45:07 | INFO  | CPU #000 | Formatter connection opened: address 10.48.64.78, port 33000.
2010-04-29 19:45:08 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 0.
2010-04-29 19:45:08 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 1.
2010-04-29 19:45:08 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 2.
2010-04-29 19:55:07 | INFO  | CPU #000 | Formatter connection closed: address 10.48.64.78, port 33000.
2010-04-29 19:55:07 | INFO  | CPU #000 | No Formatter connections is open on category 0.
2010-04-29 19:55:07 | INFO  | CPU #000 | No Formatter connections is open on category 1.
2010-04-29 19:55:07 | INFO  | CPU #000 | No Formatter connections is open on category 2.
2010-04-29 19:55:07 | INFO  | CPU #000 | Formatter connection opened: address 10.48.64.78, port 33000.
2010-04-29 19:55:09 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 0.
2010-04-29 19:55:09 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 1.
2010-04-29 19:55:09 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 2.
2010-04-29 20:04:38 | INFO  | CPU #000 | A SSH session from 10.96.108.90 was established.
2010-04-29 20:05:37 | INFO  | CPU #000 | Formatter connection closed: address 10.48.64.78, port 33000.
2010-04-29 20:05:37 | INFO  | CPU #000 | No Formatter connections is open on category 0.
2010-04-29 20:05:37 | INFO  | CPU #000 | No Formatter connections is open on category 1.
2010-04-29 20:05:37 | INFO  | CPU #000 | No Formatter connections is open on category 2.
2010-04-29 20:05:37 | INFO  | CPU #000 | Formatter connection opened: address 10.48.64.78, port 33000.
2010-04-29 20:05:38 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 0.
2010-04-29 20:05:38 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 1.
2010-04-29 20:05:38 | INFO  | CPU #000 | Formatter active connection opened : address 10.48.64.78, port 33000, category 2.
2010-04-29 20:08:53 | ERROR | CPU #000 | SE Watchdog Module: An Error occurred. Please report to Cisco's customer support
2010-04-29 20:08:53 | ERROR | CPU #000 | SE Watchdog Module: An Error occurred. Please report to Cisco's customer support
2010-04-29 20:08:53 | WARN  | CPU #000 | SE Watchdog Module: A problem occurred. Please report to Cisco's customer support
2010-04-29 20:08:53 | INFO  | CPU #000 | Party data base was closed.

Everyone's tags (8)
1146
Views
0
Helpful
0
Replies
CreatePlease to create content