Error Detected when CRS-MSC inserted.

ZZZ233333 · ‎05-28-2010

Beside me is a Cisco CRS-1 Series 16 Slots Carrier Routing System/Single, and it's running fine, the software running on the system is Cisco IOS XR Software, Version 3.6.1, after a new pair of PLIM and MSC being inserted, error come out, but when I replace the MSC module, the error message disappeared and the LC boot normally,can anyone tell me what's wrong with it?

SP/0/4/SP:May 27 00:12:23.020 : upgrade_daemon[132]: failed to get eeprom, 'sysdb' detected the 'warning' condition 'A SysDB client tried to access a nonexistent item or list an empty directory'
SP/0/4/SP:May 27 00:12:23.249 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR in state default
RP/0/RP0/CPU0:May 27 00:12:56.993 : tftp_server[351]: %PROTO-CE_TFTP-6-KERNEL_DUMP_MSG : Finished writing to filename: /harddisk:/kernel_core_lc_65.Z
RP/0/RP0/CPU0:May 27 00:13:01.238 : invmgr[203]: %PLATFORM-INV-6-OIRIN : OIR: Card 0/4/* inserted
SP/0/4/SP:May 27 00:13:07.622 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR FAIL in state default
RP/0/RP0/CPU0:May 27 00:13:11.875 : shelfmgr[322]: %PLATFORM-SHELFMGR-3-NODE_RESET_BRINGDOWN : Reset node 0/4/CPU0 due to heartbeat loss
RP/0/RP0/CPU0:May 27 00:13:11.882 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/SP, state: BRINGDOWN
RP/0/RP0/CPU0:May 27 00:13:11.926 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: BRINGDOWN

RP/0/RP0/CPU0:ios(admin)#RP/0/RP0/CPU0:May 27 00:14:20.813 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/SP, state: IOS XR RUN
SP/0/4/SP:May 27 00:14:07.951 : sysmgr[76]: %OS-SYSMGR-5-NOTICE : Card is COLD started
RP/0/RP0/CPU0:May 27 00:15:04.410 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: IOS XR RUN
SP/0/4/SP:May 27 00:15:03.337 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR   in state default
LC/0/4/CPU0:5: wd-critical-mon[78]: HW Watchdog: disabled on this node.
LC/0/4/CPU0:May 27 00:15:00.979 : sysmgr[77]: %OS-SYSMGR-5-NOTICE : Card is COLD started
LC/0/4/CPU0:May 27 00:15:09.546 : /pkg/bin/sysmgr_log[65609]: %OS-SYSMGR-4-CHECK_LOG : prior abnormal shutdown from May 27 00:11. Log: /net/node0_RP0_CPU0/harddisk:/shutdown/node0_4_CPU0.log.gz
LC/0/4/CPU0:May 27 00:15:14.318 : cpuctrl[266]: %PLATFORM-CPUCTRL-3-ASIC_NOT_FOUND : Cpuctrl could not find Fabricq Instance 0,
LC/0/4/CPU0:May 27 00:15:15.469 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=45119) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:15.575 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:15:15.579 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 1) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:15:15.572 : sysmgr[77]: gsp(1) (jid 139) (pid 45119) (fail_count 1) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:25.789 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=147498) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:26.854 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:15:26.857 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 2) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:15:26.852 : sysmgr[77]: gsp(1) (jid 139) (pid 147498) (fail_count 2) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:37.072 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=151594) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:38.139 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:15:38.141 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 3) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:15:38.137 : sysmgr[77]: gsp(1) (jid 139) (pid 151594) (fail_count 3) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:48.348 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=155690) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:49.411 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:15:49.413 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 4) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:15:49.410 : sysmgr[77]: gsp(1) (jid 139) (pid 155690) (fail_count 4) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:50.831 : sysmgr[77]: %OS-SYSMGR-3-ERROR : asic_scan_server[102] (pid 36910) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:50.966 : /pkg/bin/sysmgr_log[65581]: %OS-SYSMGR-4-CHECK_LOG : /pkg/bin/sysmgr_debug_script invoked for: (asic_scan_server) process did not signal EOI. Output is in /tmp/sysmgr_debug/debug.159786
LC/0/4/CPU0:May 27 00:15:51.081 : sysmgr[77]: %OS-SYSMGR-3-ERROR : fabricq_mgr[135] (pid 36918) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:51.579 : sysmgr[77]: %OS-SYSMGR-3-ERROR : pse_driver[178] (pid 45105) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:52.041 : sysmgr[77]: %OS-SYSMGR-3-ERROR : mstats_svr[181] (pid 45118) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:52.929 : sysmgr[77]: %OS-SYSMGR-3-ERROR : plim_1p_oc768[200] (pid 45127) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:53.440 : sysmgr[77]: %OS-SYSMGR-3-ERROR : plumm[204] (pid 45134) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:53.633 : sysmgr[77]: %OS-SYSMGR-3-ERROR : tlumm[280] (pid 45137) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:53.732 : sysmgr[77]: %OS-SYSMGR-3-ERROR : uidb_svr[284] (pid 49232) has not sent proc-ready within 45 seconds
LC/0/4/CPU0:May 27 00:15:59.620 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=237610) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:00.697 : sysmgr[77]: %OS-SYSMGR-2-REBOOT : reboot required, process (gsp) reason (maximum restart attempts exceeded)
LC/0/4/CPU0:May 27 00:16:00.682 : sysmgr[77]: gsp(1) (jid 139) (pid 237610) (fail_count 5) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:01.183 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=241706) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:01.214 : /pkg/bin/sysmgr_log[65599]: %OS-SYSMGR-4-CHECK_LOG : /pkg/bin/shutdown_debug_script invoked by sysmgr. Reason: (gsp) maximum restart attempts exceeded, Compressed output will be saved to: /net/node0_RP0_CPU0/harddisk:/shutdown/node0_4_CPU0.log.gz
LC/0/4/CPU0:May 27 00:16:02.250 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:16:02.252 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 6) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:16:02.248 : sysmgr[77]: gsp(1) (jid 139) (pid 241706) (fail_count 6) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:12.469 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=553029) : -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:13.533 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode
LC/0/4/CPU0:May 27 00:16:13.535 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 7) will be respawned in 10 seconds
LC/0/4/CPU0:May 27 00:16:13.531 : sysmgr[77]: gsp(1) (jid 139) (pid 553029) (fail_count 7) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:23.751 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=1196101) : -

PS: I have serach the Cisco IOS XR System Error Message Reference Guide, but there is no detail explanation on how to fix this problem.

Thanks,

Edwared.

Giuseppe Larosa · ‎05-28-2010

Hello Edwared,

or first time you hadn't inserted the line card perfectly, this is likely the reason for these messages it happened to me too.

or you changed the linecard and the first linecard was faulty (this may happen and happened to me too)

Hope to help

Giuseppe

Serge Krier · ‎06-11-2010

Hi,

The root of the problem is given by this message :

PLATFORM-CPUCTRL-3-ASIC_NOT_FOUND : Cpuctrl could not find Fabricq

the cpu controller on the MSC can't detect somehow one of the fabricq asic (also on the MSC), leading to lack of access of the MSC cpu to fabric and causing the reboot ultimately. It's most likely an hardware issue, you might want to try to reseat it just in case.

Hope it helps