VPN HW-1-PACKET ERROR

Lance Wendel · ‎11-10-2014

Hi all,

my customer is seen lots of errors shown below

%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr=x.x.x.x,size=1392,handle=0x5BA7

%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr==x.x.x.x,size=1392,handle=0x5BA7

%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, Output Authentication error:srcadr=y.y.y.y,dstadr=x.x.x.x,size=1392,handle=0x5AB0

%VPN_HW-1-PACKET_ERROR: slot: 2 Packet Encryption/Decryption error, ESP Pad Length:srcadr=y.y.y.y,dstadr=x.x.x.x,size=96,handle=0x5AB0

I have looked in to the show commands and could not see anything which points me to an issue

===============================================

router#show crypto eli

Hardware Encryption Layer : ACTIVE

Number of crypto engines = 1 .

CryptoEngine-0 (slot-2) details.

Capability-IPSec : IPPCP, 3DES, AES, RSA

IKE-Session : 229 active, 5120 max, 0 failed

DH-Key : 12 active, 5120 max, 0 failed

IPSec-Session : 1034 active, 10230 max, 0 failed

router#Show pas vam interf

VPN Acceleration Module Version II+ in slot : 2

Statistics for Hardware VPN Module since the last clear

of counters 241663 seconds ago

368408630 packets in 368408630 packets out

203515582983 bytes in 202789234970 bytes out

1524 paks/sec in 1524 paks/sec out

6737 Kbits/sec in 6713 Kbits/sec out

0 pkts compressed 0 pkts not compressed

0 bytes before compress 0 bytes after compress

1.0:1 compression ratio 1.0:1 overall

3981449 commands out 3981449 commands acknowledged

Last 5 minutes:

935955 packets in 935955 packets out

3119 paks/sec in 3119 paks/sec out

13198000 bits/sec in 13123845 bits/sec out

Errors:

ppq full errors : 0 ppq rx errors : 5

cmdq full errors : 0 cmdq rx errors : 0

ppq down errors : 0 cmdq down errors : 0

no buffer : 0 replay errors : 471

dest overflow : 0 authentication errors : 252

Other error : 0 Raw Input Underrun : 5

IPSEC Unsupported Option: 0 IPV4 Header Length : 0

ESP Pad Length : 5 IPSEC Decompression : 0

AH ESP seq mismatch : 0 AH Header Length : 0

AH ICV Incorrect : 0 IPCOMP CPI Mismatch : 0

IPSEC ESP Modulo : 0 Unexpected IPV6 Extensio: 0

Unexpected Protocol : 0 Dest Buf overflow : 0

IPSEC Pkt is fragment : 0 IPSEC Pkt src count : 0

Invalid IP Version : 0 Unwrappable : 0

PPTP Duplicate packet : 0 PPTP Exceed max missed p: 0

RNG self test fail : 0 DF Bit set : 0

Hash Miscompare : 0 Unwrappable object : 0

Missing attribute : 0 Invalid attrribute value: 0

Bad Attribute : 0 Verification Fail : 0

Decrypt Failure : 0 Invalid Packet : 0

Invalid Key : 0 Input Overrun : 0

Input Underrun : 0 Output buffer overrun : 0

Bad handle value : 0 Invalid parameter : 0

Bad function code : 0 Out of handles : 0

Access denied : 0 Out of memory : 0

NR overflow : 0 pkts dropped : 257

Warnings:

sessions_expired : 0 packets_fragmented : 0

general : 0 compress_bypassed : 0

HSP details:

hsp_operations : 67829762 hsp_sessions : 1258

----------------------------------------------------------------------------------------------

Slot 2:
        VAM2+ Encryption/Compression engine, Port adapter
        Port adapter is analyzed
        Port adapter insertion time 6w3d ago
        EEPROM contents at hardware discovery:
        Hardware Revision        : 1.0
        PCB Serial Number        : AAAAAAAAAAAAA
        Part Number              : 73-9571-06
        Board Revision           : A0
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Deviation Number         : 0
        Product (FRU) Number     : SA-VAM2+
        Version Identifier       : V02
        Top Assy. Part Number    : 68-2288-06
        CLEI Code                : CNDQASMGAA
        EEPROM format version 4
        EEPROM contents (hex):
          0x00: 04 FF 40 04 B0 41 01 00 C1 8B 4A 41 46 31 33 33
          0x10: 32 42 50 43 44 82 49 25 63 06 42 41 30 03 00 81
          0x20: 00 00 00 00 04 00 88 00 00 00 00 CB 94 53 41 2D
          0x30: 56 41 4D 32 2B 20 20 20 20 20 20 20 20 20 20 20
          0x40: 20 89 56 30 32 20 87 44 08 F0 06 C6 8A 43 4E 44
          0x50: 51 41 53 4D 47 41 41 FF FF FF FF FF FF FF FF FF
          0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
          0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF

----------------------------------------------------------------------

------------------ show pas isa interface ------------------

VPN Acceleration Module Version II+ in slot : 2
        Statistics for Hardware VPN Module since the last clear
         of counters 3204653 seconds ago
     6653099487 packets in                  6653099487 packets out
3413929884758 bytes in                 3416877945028 bytes out
           2076 paks/sec in                       2076 paks/sec out
           8522 Kbits/sec in                      8529 Kbits/sec out
              0 pkts compressed                      0 pkts not compressed
              0 bytes before compress                0 bytes after compress
          1.0:1 compression ratio                1.0:1 overall
       51878047 commands out                  51878047 commands acknowledged
        Last 5 minutes:
         996655 packets in                      996655 packets out
           3322 paks/sec in                       3322 paks/sec out
       13649702 bits/sec in                   13570320 bits/sec out

Errors:
   ppq full errors         :        0   ppq rx errors           :       11
   cmdq full errors        :        0   cmdq rx errors          :        0
   ppq down errors         :        0   cmdq down errors        :        0
   no buffer               :        0   replay errors           :     5548
   dest overflow           :        0   authentication errors   :      872
   Other error             :        0   Raw Input Underrun      :       11
   IPSEC Unsupported Option:        0   IPV4 Header Length      :        0
   ESP Pad Length          :       11   IPSEC Decompression     :        0
   AH ESP seq mismatch     :        0   AH Header Length        :        0
   AH ICV Incorrect        :        0   IPCOMP CPI Mismatch     :        0
   IPSEC ESP Modulo        :        0   Unexpected IPV6 Extensio:        0
   Unexpected Protocol     :        0   Dest Buf overflow       :        0
   IPSEC Pkt is fragment   :        0   IPSEC Pkt src count     :        0
   Invalid IP Version      :        0   Unwrappable             :        0
   PPTP Duplicate packet   :        0   PPTP Exceed max missed p:        0
   RNG self test fail      :        0   DF Bit set              :        0
   Hash Miscompare         :        0   Unwrappable object      :        0
   Missing attribute       :        0   Invalid attrribute value:        0
   Bad Attribute           :        0   Verification Fail       :        0
   Decrypt Failure         :        0   Invalid Packet          :        0
   Invalid Key             :        0   Input Overrun           :        0
   Input Underrun          :        0   Output buffer overrun   :        0
   Bad handle value        :        0   Invalid parameter       :        0
   Bad function code       :        0   Out of handles          :        0
   Access denied           :        0   Out of memory           :        0
   NR overflow             :        0   pkts dropped            :      883

Warnings:
sessions_expired : 0 packets_fragmented : 0
general : 0 compress_bypassed : 0

HSP details:
hsp_operations : 63582808 hsp_sessions : 1270

----------------------------------------------------------------------------------

------------------ show pas isa controller ------------------

Controller Information of Slot 2:
Encryption Mode = IPSec

Addresses of Rings and instance structure:
Low Priority Queue:
    OMQ=0xE3A4CC0, OMQ Shadow = 0x64B69FD8, {139, 139, 0, 256}
    PKQ=0xE3A8D00, PKQ Shadow = 0x64B6EC0C, {105, 105, 0, 256}
    ERQ=0xE3ACD40, ERQ Shadow = 0x64B73840, {0, 0, 0, 256}
Heartbeat info: <Addr, Value> = <0xE3B7EC0, 25CD87A>
Running default HSP (addr=0x63973630, size=809364)
hsp_version: 3.4(3) (PRODUCTION)
History Log location: 0x6464F9A0, end: 0x6465B520

High Priority Rings:
   TX: 0x0E3B0D80 {h=100, t=100, queued=0}
SRC: 0x0E3B3640 TX Shadow: 0x64B78474, {h=196, t=196}
   RX: 0x0E3B1DC0 {100, 100, 256}
   RX Pool: 0x00000000 RX Pool(hsp): 0x0E3B2E00 RX Pool Shadow: 0x64B868A8, {255, 254, 511}
Instance Structure address: 0x64B63548, ce_id=0

Misc registers:
HIFN_REG=0x3D000000
IndexReg = 0xFFFF688A
PPQ registers:
CMD ring: ctl=0x18, head=0x64, tail=0x64 [0xE3B13C0, 0xE3B13C0]
SRC ring: ctl=0x10, head=0xC4, tail=0xC4 [0xE3B3C60, 0xE3B3C60]
RSLT ring: ctl=0x5C, head=0x64, tail=0x64 [0xE3B2400]
FREE ring: ctl=0x2000058, head=0x100, tail=0xFF [0xE3B3200]
PCI registers:
pci_stat=0x60000, pci_intr_err_addr=0x0

hifn_driver_vary: 0x00000100
VAM2+ Active FLASH Image is: (V1) Default Image

Image Info for (V1) FLASH Default Image
Image Token == 0x2FADED2C: Image is Valid
Version [0x03090002,0x03900201]
ChkSum from [0xBFC00000,0xBFC2F618) is 0x00000000 with Comp==0x6C08F152
Image ChkSum is Valid

Image Info for (V2) FLASH Downloaded Image
Image Token == 0xFFFFFFFF: Image is NOT Valid

==========================================================

per Cisco error lookup, I need to escalate this to Cisco TAC, before I do that, does anyone has any idea what's causing it. My initial thoughts were HW issue

thanks in advance

Lance

Lance Wendel · ‎11-11-2014

any one? any idea?