Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Error Detected when CRS-MSC inserted.

Beside me is a Cisco CRS-1 Series 16 Slots Carrier Routing System/Single, and it's running fine, the software running on the system is Cisco IOS XR Software, Version 3.6.1, after a new pair of  PLIM and MSC being inserted, error come out, but when I replace the MSC module, the error message disappeared and the LC boot normally,can anyone tell me what's wrong with it?

SP/0/4/SP:May 27 00:12:23.020 : upgrade_daemon[132]: failed to get eeprom, 'sysdb' detected the 'warning' condition 'A SysDB client tried to access a nonexistent item or list an empty directory'
SP/0/4/SP:May 27 00:12:23.249 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR   in state default 
RP/0/RP0/CPU0:May 27 00:12:56.993 : tftp_server[351]: %PROTO-CE_TFTP-6-KERNEL_DUMP_MSG : Finished writing to filename: /harddisk:/kernel_core_lc_65.Z  
RP/0/RP0/CPU0:May 27 00:13:01.238 : invmgr[203]: %PLATFORM-INV-6-OIRIN : OIR: Card 0/4/* inserted 
SP/0/4/SP:May 27 00:13:07.622 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR FAIL in state default 
RP/0/RP0/CPU0:May 27 00:13:11.875 : shelfmgr[322]: %PLATFORM-SHELFMGR-3-NODE_RESET_BRINGDOWN : Reset node 0/4/CPU0 due to heartbeat loss
RP/0/RP0/CPU0:May 27 00:13:11.882 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/SP, state: BRINGDOWN
RP/0/RP0/CPU0:May 27 00:13:11.926 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: BRINGDOWN

RP/0/RP0/CPU0:ios(admin)#RP/0/RP0/CPU0:May 27 00:14:20.813 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/SP, state: IOS XR RUN
SP/0/4/SP:May 27 00:14:07.951 : sysmgr[76]: %OS-SYSMGR-5-NOTICE : Card is COLD started 
RP/0/RP0/CPU0:May 27 00:15:04.410 : invmgr[203]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: IOS XR RUN
SP/0/4/SP:May 27 00:15:03.337 : alphadisplay[100]: %PLATFORM-ALPHA_DISPLAY-6-CHANGE : Alpha display on node 0/4/SP changed to IOS XR   in state default 
LC/0/4/CPU0:5: wd-critical-mon[78]: HW Watchdog: disabled on this node.
LC/0/4/CPU0:May 27 00:15:00.979 : sysmgr[77]: %OS-SYSMGR-5-NOTICE : Card is COLD started 
LC/0/4/CPU0:May 27 00:15:09.546 : /pkg/bin/sysmgr_log[65609]: %OS-SYSMGR-4-CHECK_LOG : prior abnormal shutdown from May 27 00:11. Log: /net/node0_RP0_CPU0/harddisk:/shutdown/node0_4_CPU0.log.gz
LC/0/4/CPU0:May 27 00:15:14.318 : cpuctrl[266]: %PLATFORM-CPUCTRL-3-ASIC_NOT_FOUND : Cpuctrl could not find Fabricq Instance 0,
LC/0/4/CPU0:May 27 00:15:15.469 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=45119) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:15.575 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:15:15.579 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 1) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:15:15.572 : sysmgr[77]: gsp(1) (jid 139) (pid 45119) (fail_count 1) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:25.789 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=147498) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:26.854 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:15:26.857 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 2) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:15:26.852 : sysmgr[77]: gsp(1) (jid 139) (pid 147498) (fail_count 2) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:37.072 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=151594) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:38.139 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:15:38.141 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 3) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:15:38.137 : sysmgr[77]: gsp(1) (jid 139) (pid 151594) (fail_count 3) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:48.348 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=155690) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:15:49.411 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:15:49.413 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 4) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:15:49.410 : sysmgr[77]: gsp(1) (jid 139) (pid 155690) (fail_count 4) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:15:50.831 : sysmgr[77]: %OS-SYSMGR-3-ERROR : asic_scan_server[102] (pid 36910) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:50.966 : /pkg/bin/sysmgr_log[65581]: %OS-SYSMGR-4-CHECK_LOG : /pkg/bin/sysmgr_debug_script invoked for: (asic_scan_server) process did not signal EOI. Output is in /tmp/sysmgr_debug/debug.159786
LC/0/4/CPU0:May 27 00:15:51.081 : sysmgr[77]: %OS-SYSMGR-3-ERROR : fabricq_mgr[135] (pid 36918) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:51.579 : sysmgr[77]: %OS-SYSMGR-3-ERROR : pse_driver[178] (pid 45105) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:52.041 : sysmgr[77]: %OS-SYSMGR-3-ERROR : mstats_svr[181] (pid 45118) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:52.929 : sysmgr[77]: %OS-SYSMGR-3-ERROR : plim_1p_oc768[200] (pid 45127) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:53.440 : sysmgr[77]: %OS-SYSMGR-3-ERROR : plumm[204] (pid 45134) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:53.633 : sysmgr[77]: %OS-SYSMGR-3-ERROR : tlumm[280] (pid 45137) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:53.732 : sysmgr[77]: %OS-SYSMGR-3-ERROR : uidb_svr[284] (pid 49232) has not sent proc-ready within 45 seconds 
LC/0/4/CPU0:May 27 00:15:59.620 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=237610) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:00.697 : sysmgr[77]: %OS-SYSMGR-2-REBOOT : reboot required, process (gsp) reason (maximum restart attempts exceeded)
LC/0/4/CPU0:May 27 00:16:00.682 : sysmgr[77]: gsp(1) (jid 139) (pid 237610) (fail_count 5) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:01.183 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=241706) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:01.214 : /pkg/bin/sysmgr_log[65599]: %OS-SYSMGR-4-CHECK_LOG : /pkg/bin/shutdown_debug_script invoked by sysmgr. Reason: (gsp) maximum restart attempts exceeded, Compressed output will be saved to: /net/node0_RP0_CPU0/harddisk:/shutdown/node0_4_CPU0.log.gz
LC/0/4/CPU0:May 27 00:16:02.250 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:16:02.252 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 6) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:16:02.248 : sysmgr[77]: gsp(1) (jid 139) (pid 241706) (fail_count 6) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:12.469 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=553029) :  -Traceback= 48241e2c fc1ddfb0
LC/0/4/CPU0:May 27 00:16:13.533 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(1) (jid 139) can not be restarted, entering slow-restart mode  
LC/0/4/CPU0:May 27 00:16:13.535 : sysmgr[77]: %OS-SYSMGR-3-ERROR : gsp(139) (fail count 7) will be respawned in 10 seconds 
LC/0/4/CPU0:May 27 00:16:13.531 : sysmgr[77]: gsp(1) (jid 139) (pid 553029) (fail_count 7) abnormally terminated, restart scheduled
LC/0/4/CPU0:May 27 00:16:23.751 : gsp[139]: %OS-gsp-6-INFO : Fatal internal error binding fabric media:-1 restarting.   : pkg/bin/gsp : (PID=1196101) :  -

PS: I have serach the Cisco IOS XR System Error Message Reference Guide, but there is no detail explanation on how to fix this problem.

Thanks,

Edwared.

Everyone's tags (2)
2 REPLIES
Hall of Fame Super Silver

Re: Error Detected when CRS-MSC inserted.

Hello Edwared,

or first time you hadn't inserted the line card perfectly, this is likely the reason for these messages it happened to me too.

or you changed the linecard and the first linecard was faulty (this may happen and happened to me too)

Hope to help

Giuseppe

Cisco Employee

Re: Error Detected when CRS-MSC inserted.

Hi,

The root of the problem is given by this message :

PLATFORM-CPUCTRL-3-ASIC_NOT_FOUND : Cpuctrl could not find Fabricq

the cpu controller on the MSC can't detect somehow one of the fabricq asic (also on the MSC), leading to lack of access of the MSC cpu to fabric and causing the reboot ultimately. It's most likely an hardware issue, you might want to try to reseat it just in case.

Hope it helps

853
Views
0
Helpful
2
Replies