ASR1004-ESP10 crashed

Unanswered Question
Feb 20th, 2012

hi.

my ASR 1004 was crashed and I have loggs from the router.

please help me to know what happend and what could be the solution(software problem or herdware) ?

log :



Feb 17 22:26:34.708: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/bb_ESP_0_cpp-mcplo-ucode_021812002634.core.gz
Feb 17 22:26:35.477: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
Feb 17 22:26:40.569: %CPPDRV-3-LOCKDOWN: F0: cpp_cp:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.
Feb 17 22:26:47.044: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/3) is down: holding time expired
Feb 17 22:26:48.034: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/4) is down: holding time expired
Feb 17 22:27:04.451: %CPPHA-3-CDMDONE: F0: cpp_ha:  CPP 0 microcode crashdump creation completed.
Feb 17 22:27:04.454: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP MDM while client(s) still connected
Feb 17 22:27:04.454: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP CDM while client(s) still connected
Feb 17 22:27:04.491: %IOSXE-6-PLATFORM: F0: cpp_cdm: Shutting down CPP MDM while client(s) still connected
Feb 17 22:27:04.827: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_ha_top_level_server has been helddown (rc 69)
Feb 17 22:27:04.863: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_cdm_svr has been helddown (rc 69)
Feb 17 22:27:13.290: %OSPFv3-5-ADJCHG: Process 378, Nbr .x.x.x.x on GigabitEthernet1/0/2 from FULL to DOWN, Neighbor Down: Dead timer expired
Feb 17 22:27:13.291: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from FULL to DOWN, Neighbor Down: Dead timer expired
Feb 17 22:27:13.835: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Feb 17 22:27:14.258: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired
Feb 17 22:27:34.644: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
Feb 17 22:27:55.678: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/4 non DR
Feb 17 22:28:06.678: %PIM-5-NBRCHG: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/1 non DR
Feb 17 22:28:11.678: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/3 DR
Feb 17 22:28:11.678: %PIM-5-DRCHG: VRF tzfat: DR change from neighbor x.x.x.x to x.x.x.x on interface GigabitEthernet1/0/3
Feb 17 22:28:11.678: %PIM-5-NBRCHG: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/2 non DR
Feb 17 22:28:36.107: %ASR1000_OIR-6-ONLINECARD: Card (fp) online in slot F0
Feb 17 22:28:53.739: %CPPHA-7-START: F0: cpp_ha:  CPP 0 preparing image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10
Feb 17 22:28:54.126: %CPPHA-7-START: F0: cpp_ha:  CPP 0 startup init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10
Feb 17 22:28:59.086: %CPPHA-7-START: F0: cpp_ha:  CPP 0 running init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10
Feb 17 22:28:59.348: %CPPHA-7-READY: F0: cpp_ha:  CPP 0 loading and initialization complete
Feb 17 22:29:00.854: %IOSXE-6-PLATFORM: F0: cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER
Feb 17 22:29:02.032: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/4) is up: new adjacency
Feb 17 22:29:02.457: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/3) is up: new adjacency
Feb 17 22:29:03.366: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from LOADING to FULL, Loading Done
Feb 17 22:29:03.367: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from LOADING to FULL, Loading Done
Feb 17 22:29:03.936: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from LOADING to FULL, Loading Done
Feb 17 22:29:04.635: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from LOADING to FULL, Loading Done
Feb 17 22:29:06.059: %PIM-5-NBRCHG: neighbor x.x.x.x UP on interface GigabitEthernet1/0/1
Feb 17 22:29:06.853: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x UP on interface GigabitEthernet1/0/4
Feb 17 22:29:09.118: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x UP on interface GigabitEthernet1/0/3
Feb 17 22:29:09.120: %PIM-5-DRCHG: VRF tzfat: DR change from neighbor x.x.x.x to x.x.x.x on interface GigabitEthernet1/0/3
Feb 17 22:29:14.645: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

Feb 17 22:26:34.556: %CPPDRV-3-LOCKDOWN: F0: fman_fp_image:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.

Feb 17 22:26:34.583: %CPPOSLIB-3-ERROR_NOTIFY: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  cpp_driver-0 encountered an error -Traceback= 1#18f9da19b6c01995eb8088a2798220bc   errmsg:F5A9000+2160 cpp_common_os:FBE1000+BA30 cpp_common_os:FBE1000+18DF8 cpp_drv_cmn:FD34000+6A3EC cpp_drv_cmn:FD34000+74C4C :10000000+CF04 cpp_drv_cmn:FD34000+6C2C4 cpp_drv_cmn:FD34000+60570 time:F26D000+5744 evlib:F8FF000+DABC evlib:F8FF000+FFC4 :10000000+623C c:EBCB000+1E938 c:EBCB000+1EAE0

Feb 17 22:26:34.588: %CPPHA-3-FAULT: F0: cpp_ha:  CPP:0 desc:DPE5_CPE_CPE_DPE_INT_SET_0_LEAF_INT_INT_PHY_ERROR det:DRVR(interrupt) class:OTHER sev:FATAL id:1493 cppstate:STOPPED res:UNKNOWN flags:0x7 cdmflags:0x1

Feb 17 22:26:34.588: %CPPHA-3-FAULTCRASH: F0: cpp_ha:  CPP 0 unresolved fault detected, initiating crash dump.

Feb 17 22:26:34.602: %CPPHA-3-FAULT: F0: cpp_ha:  CPP:0 desc:DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_MBE det:DRVR(interrupt) class:MBE sev:FATAL id:1522 cppstate:STOPPED res:SUCCESS flags:0x7 cdmflags:0x1

Feb 17 22:26:34.602: %CPPHA-3-FAULTCRASH: F0: cpp_ha:  CPP 0 unresolved fault detected, initiating crash dump.

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : Last 16 Interrupts Since Boot

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 16 01:38:31.143161: :GAL_GAL_CSR_IPM_IF_GAL_IPM_IF_LEAF_INT_INT_IPM_ERR

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt :Â

Feb 17 22:34:34.451807: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt :Â

Feb 17 22:34:34.457928: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt :Â

Feb 17 22:56:58.451815: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt :Â

Feb 17 22:56:58.457335: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt :Â

Feb 17 23:19:22.451792: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.617: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 17 23:19:22.457337: :DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_SBE

Feb 17 22:26:34.618: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 18 00:26:34.587948: :HALT:DPE5_CPE_CPE_DPE_INT_SET_0_LEAF_INT_INT_PHY_ERROR

Feb 17 22:26:34.618: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 18 00:26:34.587948: :HALT:DPE5_CPE_CPE_DPE_DUI_LEAF_INT_INT_DUI_CHN0_DRAM_MBE

Feb 17 22:26:34.618: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 18 00:26:34.587948: :HEDP_HED_HALTED_IN_63_0_LEAF_INT_INT_HALTED10

Feb 17 22:26:34.618: %CPPDRV-6-INTR: F0: /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/cpp_driver[6109]:  CPP10(0) Interrupt : 

Feb 18 00:26:34.587948: :HOT:INFP_INF_IDMU_LEAF_INT_INT_TRANSACTION_TIMEOUT

Feb 17 22:26:34.708: %IOSXE-3-PLATFORM: F0: cpp_cdm: CPP crashed, core file /tmp/corelink/bb_ESP_0_cpp-mcplo-ucode_021812002634.core.gz

Feb 17 22:26:35.477: %ASR1000_OIR-6-OFFLINECARD: Card (fp) offline in slot F0

Feb 17 22:26:40.569: %CPPDRV-3-LOCKDOWN: F0: cpp_cp:  CPP10(0) CPP Driver LOCKDOWN due to fatal error.

Feb 17 22:26:47.044: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/3) is down: holding time expired

Feb 17 22:26:48.034: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/4) is down: holding time expired

Feb 17 22:27:04.451: %CPPHA-3-CDMDONE: F0: cpp_ha:  CPP 0 microcode crashdump creation completed.

Feb 17 22:27:04.454: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP MDM while client(s) still connected

Feb 17 22:27:04.454: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP CDM while client(s) still connected

Feb 17 22:27:04.491: %IOSXE-6-PLATFORM: F0: cpp_cdm: Shutting down CPP MDM while client(s) still connected

Feb 17 22:27:04.827: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_ha_top_level_server has been helddown (rc 69)

Feb 17 22:27:04.863: %PMAN-3-PROCHOLDDOWN: F0: pman.sh:  The process cpp_cdm_svr has been helddown (rc 69)

Feb 17 22:27:13.290: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from FULL to DOWN, Neighbor Down: Dead timer expired

Feb 17 22:27:13.291: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from FULL to DOWN, Neighbor Down: Dead timer expired

Feb 17 22:27:13.835: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired

Feb 17 22:27:14.258: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired

Feb 17 22:27:34.644: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down

Feb 17 22:27:55.678: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/4 non DR

Feb 17 22:28:06.678: %PIM-5-NBRCHG: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/1 non DR

Feb 17 22:28:11.678: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/3 DR

Feb 17 22:28:11.678: %PIM-5-DRCHG: VRF tzfat: DR change from neighbor x.x.x.x to x.x.x.x on interface GigabitEthernet1/0/3

Feb 17 22:28:11.678: %PIM-5-NBRCHG: neighbor x.x.x.x DOWN on interface GigabitEthernet1/0/2 non DR

Feb 17 22:28:36.107: %ASR1000_OIR-6-ONLINECARD: Card (fp) online in slot F0

Feb 17 22:28:53.739: %CPPHA-7-START: F0: cpp_ha:  CPP 0 preparing image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Feb 17 22:28:54.126: %CPPHA-7-START: F0: cpp_ha:  CPP 0 startup init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Feb 17 22:28:59.086: %CPPHA-7-START: F0: cpp_ha:  CPP 0 running init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

Feb 17 22:28:59.348: %CPPHA-7-READY: F0: cpp_ha:  CPP 0 loading and initialization complete

Feb 17 22:29:00.854: %IOSXE-6-PLATFORM: F0: cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER

Feb 17 22:29:02.032: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/4) is up: new adjacency

Feb 17 22:29:02.457: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.x (GigabitEthernet1/0/3) is up: new adjacency

Feb 17 22:29:03.366: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from LOADING to FULL, Loading Done

Feb 17 22:29:03.367: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/2 from LOADING to FULL, Loading Done

Feb 17 22:29:03.936: %OSPF-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from LOADING to FULL, Loading Done

Feb 17 22:29:04.635: %OSPFv3-5-ADJCHG: Process 378, Nbr x.x.x.x on GigabitEthernet1/0/1 from LOADING to FULL, Loading Done

Feb 17 22:29:06.059: %PIM-5-NBRCHG: neighbor x.x.x.x UP on interface GigabitEthernet1/0/1

Feb 17 22:29:06.853: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x UP on interface GigabitEthernet1/0/4

Feb 17 22:29:09.118: %PIM-5-NBRCHG: VRF tzfat: neighbor x.x.x.x UP on interface GigabitEthernet1/0/3

Feb 17 22:29:09.120: %PIM-5-DRCHG: VRF tzfat: DR change from neighbor x.x.x.x to x.x.x.x on interface GigabitEthernet1/0/3

Feb 17 22:29:14.645: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

Feb 17 22:29:21.908: %PIM-5-NBRCHG: neighbor x.x.x.x UP on interface GigabitEthernet1/0/2Â

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
johnlloyd_13 Mon, 02/27/2012 - 16:40

hi,

we had a similar issue just yesterday. kindly post your show platform output.

we had to reboot the router for a temporarily fix and then RMAd the ESP card.

#sh platform

Chassis type: ASR1004           

Slot      Type                State                 Insert time (ago)

--------- ------------------- --------------------- -----------------

0         ASR1000-SIP10       ok                    50w2d       

0/0      SPA-5X1GE-V2        ok                    50w2d       

0/1      SPA-2X1GE-V2        ok                    50w2d       

R0        ASR1000-RP1         ok, active            50w2d       

F0        ASR1000-ESP10       booting               50w2d       

P0        ASR1004-PWR-AC      ok                    50w2d       

P1        ASR1004-PWR-AC      ok                    50w2d    

#sh plat

Chassis type: ASR1004            

Slot      Type                State                 Insert time (ago)

--------- ------------------- --------------------- -----------------

0         ASR1000-SIP10       ok                    17:51:09     

0/0      SPA-5X1GE-V2        ok                    17:49:16     

0/1      SPA-2X1GE-V2        ok                    17:49:16     

R0        ASR1000-RP1         ok, active            17:51:09     

F0        ASR1000-ESP10       ok, active            17:51:09   

P0        ASR1004-PWR-AC      ok                    17:50:21     

P1        ASR1004-PWR-AC      ok                    17:50:20     

Slot      CPLD Version        Firmware Version                       

--------- ------------------- ---------------------------------------

0         07091401            12.2(33r)XN2                       

R0        07062111            12.2(33r)XN2                       

F0        07091401            12.2(33r)XN2 

tuzhilin_andrey Mon, 08/06/2012 - 04:53

Hello!

We've got a similar ESP-crash on our asr1004 with log:

CPPDRV-3-LOCKDOWN: F0: cpp_ha:  CPP10(0) CPP Driver LOCKDOWN due to fatal error

And also it created a core-dump. We assume that this is bug CSCte19641 (http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCte19641), but there is no solution and no workaround. That's why I ask you, why did you do an RMA?

Actions

Login or Register to take actions

This Discussion

Posted February 20, 2012 at 1:41 AM
Stats:
Replies:2 Avg. Rating:
Views:1847 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard