Faulty ESP5 in an ASR1002

Iulian Vaideanu · ‎08-06-2013

This morning the ESP5 in one of our ASR1002 devices failed. Well, not entirely (I managed, for example, to update its firmware to 15.3(1r)S) but, if left enabled, goes through an infinite reset loop logging the lines below. It seems to have issues with the crypto hardware - is there any way to disable just that and maintain the rest of the card's functionality?

While at this, noob question: can an ASR1002 equipped with a SIP10 (equipped with a SPA-2X1GE-V2) work without an ESP? If it can, what would the performance penalty be? We are using these devices as PPPoE concentrators...

Many thanks in advance.

*Aug 6 14:11:20: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0

*Aug 6 14:11:54: %CPPHA-7-START: F0: cpp_ha: CPP 0 preparing image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

*Aug 6 14:11:55: %CPPHA-7-START: F0: cpp_ha: CPP 0 startup init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

*Aug 6 14:12:00: %CPPHA-7-START: F0: cpp_ha: CPP 0 running init image /tmp/sw/fp/0/0/fp/mount/usr/cpp/bin/qfp-ucode-esp10

*Aug 6 14:12:00: %CPPHA-7-READY: F0: cpp_ha: CPP 0 loading and initialization complete

*Aug 6 14:12:03: %CMFP-6-N2_DRV_UPDATE: F0: cman_fp: Hardware crypto device disabled. Unable to setup Nitrox cores: Unknown error 3559

*Aug 6 14:12:03: %EVENTLIB-3-RUNHOG: F0: cman_fp: undefined: 3278ms, Traceback=1#c5d8ecb8adc238082285adeae6e14d96 evlib:FD5A000+D6A0 evlib:FD5A000+A998 cpp_common_os:E4E9000+14FAC cpp_client_ha:E56C000+4A98 cpp_platform_ha:E545000+3578 :10000000+27A40 :10000000+28644 :10000000+1C5A0 :10000000+1C854 binos:FC2F000+C76C :10000000+1C35C :10000000+1F738

*Aug 6 14:12:03: %EVENTLIB-3-HISTSUMM: F0: cman_fp: dispatch start: 83527ms elapsed time: 3358ms detected hog: 0 set count: 1

*Aug 6 14:12:03: %EVENTLIB-3-HISTELEM: F0: cman_fp: elem[1] event: Unset -> BEFORE HOG set: 83527ms unset: 86881ms hog check: 1 caller: 0xe4fa284 app-info: SMC comp_id(0x0054) msgno(23) rx_async_count(0)

*Aug 6 14:12:04: %IOSXE-6-PLATFORM: F0: cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER

*Aug 6 14:12:10: %IMGR-6-FIPS_FMFP_KAT_TEST: F0: fman_fp_image: power up cryptographic self-test didnot run to completion: N2RandomFips failed .

*Aug 6 14:12:11: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0

*Aug 6 14:12:45: %CPPHA-3-FAULT: F0: cpp_ha: CPP:0.0 desc:CPP Client process failed: FMAN-FP det:HA class:CLIENT_SW sev:FATAL id:1 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0

*Aug 6 14:12:45: %IOSXE-6-PLATFORM: F0: cpp_ha: Shutting down CPP MDM while client(s) still connected

*Aug 6 14:12:45: %PMAN-3-PROCHOLDDOWN: F0: pman.sh: The process fman_fp_image has been helddown (rc 134)

*Aug 6 14:12:45: %PMAN-3-PROCHOLDDOWN: F0: pman.sh: The process cpp_ha_top_level_server has been helddown (rc 69)

*Aug 6 14:12:45: %PMAN-0-PROCFAILCRIT: F0: pvp.sh: A critical process fman_fp_image has failed (rc 134)

Iulian Vaideanu · ‎08-07-2013

I just loaded a non-K9 IOS image, hoping that maybe the ASR would skip the crypto part when initializing the ESP - well, it didn't. A replacement ESP5 should arrive on site today, and I guess it's RMA time...

Vignesh Rajendran Praveen · ‎08-07-2013

Hello Iulian,

If the ESP stays continously in someother state than in the Active state from "show platform" output of the device, then the ESP needs to be replaced. Probably you could try to do a hard reseat before replacing it.

Another point is that any ASR1K device will not be able to forward any traffic without the presence of an ESP on it. It is the sole responsiblity of the ESP to take care of packet forwarding on any ASR1K device.

****Plz do rate this post without fail if you found it to be helpful*********

Thanks & Regards,

Vignesh R P

Iulian Vaideanu · ‎08-07-2013

Well, it cycles endlessly through a sequence of states made of "booting", "init, ok", "active, ok", "disconnecting", "unknown" and then all over again, all these lasting for about three minutes and generating a core file. It was during "init, ok" and "active, ok" when I managed to update its firmware to 15.3(1r)S (and later on to 15.3(3r)S), but that didn't change anything.

Thanks for clearing up the part about the ASR1K working without an ESP.

Vignesh Rajendran Praveen · ‎08-07-2013

Hello,

Based on the symptoms provided by you, I can confirm that the ESP 5 has a hardware related fault and needs to be replaced ASAP.

****Plz do rate this post without fail if you found it to be helpful*********

Thanks & Regards,

Vignesh R P

GIANRICO FICHERA · ‎07-21-2023

Thanks for helping me solve my problem.
The system is unable to route packets. You can have a behaviour like this:

ASR1002_A_GENESYS#show platform | include ESP10
F0        ASR1000-ESP10     unknown      01:13:17 

ASR1002_A_GENESYS#show platform | include ESP10
F0        ASR1000-ESP10     booting      01:13:29 

ASR1002_ITESYS#sh run int loopback 1
interface Loopback1
  ip address 1.1.1.1 255.255.255.255
end

ASR1002_ITESYS#show ip route
...snip...
1.0.0.0/32 is subnetted, 1 subnets
C 1.1.1.1 is directly connected, Loopback1

ASR1002_ITESYS#ping 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)