ASR 1000 ESP-10 errors

[Oleg Volkov]

Dear Sirs!

Approximately once in week, my friends get reload F0 card - ESP-10:

Sep 13 00:50:19.676: %CPPHA-3-FAULT: F0: cpp_ha:  CPP:0.0 desc:INFP_INF_SWASSIST_LEAF_INT_INT_EVENT0 det:DRVR(interrupt) class:OTHER sev:FATAL id:2121 cppstate:RUNNING res:UNKNOWN flags:0x7 cdmflags:0x8
Sep 13 00:50:19.677: %CPPHA-3-FAULTCRASH: F0: cpp_ha:  CPP 0.0 unresolved fault detected, initiating crash dump.
Sep 13 00:50:19.681: %CPPHA-3-FAULTCRASH: F0: cpp_ha:  CPP 0.0 unresolved fault detected, initiating crash dump.
Sep 13 00:50:19.683: %CPPDRV-6-INTR: F0: cpp_driver-0:  CPP10(0) Interrupt : 14-Sep-13 00:50:19.668657 UTC+0800:INFP_INF_SWASSIST_LEAF_INT_INT_EVENT0 
Sep 13 00:50:20.299: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Sep 13 00:50:22.324: %CPPDRV-3-LOCKDOWN: F0: cpp_cp:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.
Sep 13 00:50:22.341: %CPPOSLIB-3-ERROR_NOTIFY: F0: cpp_cp:  cpp_cp encountered an error -Traceback= 1#74fded098ef464b6f57a79773f56a094   errmsg:C5B4000+2230 cpp_common_os:CC6B000+C6D0 cpp_common_os:CC6B000+C4D0 cpp_common_os:CC6B000+19E5C cpp_fnf_svr_lib:FDCB000+209D4 cpp_fnf_svr_lib:FDCB000+2A29C cpp_fnf_svr_lib:FDCB000+167CC cpp_fnf_svr_lib:FDCB000+1B3F4 cpp_fnf_svr_lib:FDCB000+19C64 cpp_common_os:CC6B000+11644 cpp_common_os:CC6B000+11C7C evlib:C98A000+E16C evlib:C98A000+10554 cpp_common_os:CC6B000+133EC :
Sep 13 00:50:23.064: %CPPDRV-3-LOCKDOWN: F0: cpp_ha:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.
Sep 13 00:50:23.143: %CPPOSLIB-3-ERROR_NOTIFY: F0: fman_fp_image:  fman-fp encountered an error -Traceback= 1#d64be460d611c48bfcaa0aaccc425965   errmsg:C58A000+2230 cpp_common_os:90E8000+C6D0 cpp_common_os:90E8000+C4D0 cpp_client_ha:96C0000+4DE4 cpp_client_ha_mib:838E000+2908 cpp_common_os:90E8000+11644 cpp_common_os:90E8000+11C7C evlib:9BF7000+E16C evlib:9BF7000+10554 :10000000+42E6F8 c:7226000+1E938 c:7226000+1EAE0
Sep 13 00:50:23.967: %EVENTLIB-3-CPUHOG: F0: cpp_cp:  undefined: 1467ms, Traceback=1#74fded098ef464b6f57a79773f56a094   binos:C85B000+D9D8 :100000+344 c:996F000+562A8 c:996F000+562A8 btrace:C893000+6FAC btrace:C893000+5B24 btrace:C893000+5D18 infra:A151000+95AC infra:A151000+9748 cpp_common_os:CC6B000+10D40 cpp_common_os:CC6B000+10484
Sep 13 00:50:24.014: %CPPCDM-3-ERROR_NOTIFY: F0: cpp_cdm:  QFP 0 thread 158 encountered an error -Traceback= 1#ecad428d7a76a0c2d30601e2384ed030 8029927F 80291440 802988F4 80299090 802993F4 8029A029 8029AB88 8029B7A0 802A7451 802A00BD 8079020B 8201E701 80674280 8067428A 80020064 80020055 80000000 
Sep 13 00:50:24.290: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/Sibline_BRAS_ESP_0_cpp-mcplo-ucode_091314005024.core.gz
Sep 13 00:50:25.342: %CPPDRV-3-LOCKDOWN: F0: fman_fp_image:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.

they use IOS: bootflash:/asr1000rp1-adventerprisek9.03.11.00.S.154-1.S-std.bin
but they try to use older IOS and gets same errors.

Aug  5 13:00:48.770: %IDMGR-3-INVALID_ID: bad id in id_to_ptr (id: 0xB0D0B0D)
-Traceback= 1#b75fa949d5bbbf210afd90208a9e8534  :10000000+EA0DEC :10000000+EA1198 :10000000+ECB68C :10000000+280D7DC :10000000+282A858 :10000000+3944F90 :10000000+395019C :10000000+2819F38 :10000000+5FC0BB0 :10000000+5FC0F78

Aug 18 07:39:43.714: %CPPHA-3-CDMDONE: F0: cpp_ha:  CPP 0 microcode crashdump creation completed.

Aug 18 07:39:43.718: %IOSXE-6-PLATFORM: F0: cpp_cdm: Shutting down CPP MDM while client(s) still connected 

Aug 18 07:39:43.725: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP MDM while client(s) still connected 

Aug 18 07:39:43.729: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP CDM while client(s) still connected 

Aug 18 07:39:44.195: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_cdm_svr has been helddown (rc 69)

Aug 18 07:39:44.220: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_ha_top_level_server has been helddown (rc 69)

Aug 18 07:39:44.688: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process fman_fp_image has been helddown (rc 134)

Aug 18 07:41:09.867: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0

Aug 18 07:41:45.571: %CPPHA-7-START: F0: cpp_ha:  CPP 0 preparing image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Aug 18 07:41:47.203: %CPPHA-7-START: F0: cpp_ha:  CPP 0 startup init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Aug 18 07:41:53.155: %CPPHA-7-START: F0: cpp_ha:  CPP 0 running init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Aug 18 07:41:53.450: %CPPHA-7-READY: F0: cpp_ha:  CPP 0 loading and initialization complete

Aug 18 07:41:54.982: %IOSXE-6-PLATFORM: F0: cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER

What we can determine type of this errors - hardwre/software?

Thanks!

PS:

Alsow I visualy check memory module on the ESP-10 - it based on Micron chips manufactured in 2008 years - it can be bad.

Please help me.

[Akash Agrawal]

Hi,

could be possible faulty DRAM issue or router is running on low memory.  but since there are tracebacks and core file also has got generated. Will advise you to open TAC case with cisco

Regards,

Akash

[Oleg Volkov]

Thanks!
But they do not have SmartNet.
But I can replace memory module on ESP.
Now, on ESPN, installed module based on Micron IC's - 2008 Year.
I know it is chips may be bad (I frequently replace Micron IC's on Catalyst 2960 and 3750).

I attached new tech-info - I always see %CPPHA-3-FAULT: F0: cpp_ha:  CPP:0.0 desc:INFP_INF_SWASSIST_LEAF_INT_INT_EVENT0 det:DRVR(interrupt) class:OTHER sev:FATAL id:2121 cppstate:RUNNING res:UNKNOWN flags:0x7 cdmflags:0x8

[Oleg Volkov]

Can You tell me true part number for memory module on ESP?

Thanks!