vpn hardware error with packets bigger than 128 Byte payload

Unanswered Question
Jun 28th, 2010
User Badges:

anyone else having the same problem?


000085: Jun 21 18:09:12.635 CEST: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Invalid Packet:srcadr=192.168.x.x,dstadr=172.x.x.x,size=120,handle=0x5801.....
Success rate is 0 percent (0/5)

legacy vpn

crypto map


transform-set esp-sha1 esp-aes-128


(hub)

2811 with aim-module AIM-VPN/EPII-PLUS

c2800nm-advipservicesk9-mz.124-8c.bin
c2800nm-advipservicesk9-mz.124-25c.bin


(spoke)

c1841-advipservicesk9-mz.124-10c.bin

on-board-vpn-chip


small packet passes

bigger packets hangs


Don't think of fragmentation, because we are still below the 576-size!


any hint is appreciated, William

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Marcin Latosiewicz Mon, 06/28/2010 - 06:38
User Badges:
  • Cisco Employee,

William,


Please check if the problem persists with onboard accelarator.


I would be interested in:

"show crypt eli"

"show crypto engine accel stati"



Marcin


edit:

Ad, changing accelarator - please note that it will most likely need to re-establish the tunnels.

9w.boye Mon, 06/28/2010 - 07:42
User Badges:

changing from hardware-module to onboard to software ipsec brought no success

no crypto engine aim

no crypto engine onboard

 

with aim-module:

Show crypto engine accelerator stat

 

xxx#show crypto engine acc stat

Virtual Private Network (VPN) Module in slot : 0

Statistics for Hardware VPN Module since the last clear

of counters 285502 seconds ago

...

Errors:

ppq full errors : 0 ppq rx errors : 33

NR overflow : 0 pkts dropped : 33

Warnings:

sessions_expired : 0 packets_fragmented : 0

general: : 0

HSP details:

hsp_operations : 66775 hsp_sessions : 4

 

xxxx#ping 172.16.9.240 source loop0 size 200 repeat 1

Type escape sequence to abort.

Sending 1, 200-byte ICMP Echos to 172.16.9.240, timeout is 2 seconds:

Packet sent with a source address of x.x.x.x

000112: Jun 21 18:25:26.962 CEST: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Invalid Packet:srcadr=192.168.x.x,dstadr=172.x.x.x size=120,handle=0x5801.

Success rate is 0 percent (0/1)

 

r-zhza1#show crypto engine acc stat

Virtual Private Network (VPN) Module in slot : 0

Statistics for Hardware VPN Module since the last clear

of counters 285543 seconds ago

...

Errors:

ppq full errors : 0 ppq rx errors : 34

..

Decrypt Failure : 0 Invalid Packet : 34

...

NR overflow : 0 pkts dropped : 34

Warnings:

sessions_expired : 0 packets_fragmented : 0

general: : 0


Marcin Latosiewicz Mon, 06/28/2010 - 08:27
User Badges:
  • Cisco Employee,

Did you check that you changed crypto from AIM to software "show crypto ipsec sa | i flow" will tell you.


IF the problem persists on soft and hard crypto then it's time to check:

1) Other side - punt in from software to hardwar (or vice versa) - does it change anything?

2) ISP? Check capture on one side as compared to other side. (I would suggest SPAN if possible)

9w.boye Mon, 06/28/2010 - 09:12
User Badges:

Hello Marcin, I appreciate your efforts very much. But don't you think it is time for an internal answer?


After the first occurences - with the 25c-image on the hub - we made a test inside of our lab: with the same result.


Our second router had a bit older image: 8c. So we thought this might be the cause.


It turned out, that it was not.


What else was different? On the "working" hub-side we used as the transform-set esp-aes-256 AND comp-lzs

We wanted to change it - what makes sense - to esp-aes-128 and esp-sha1


Of course we cleared the crypto sas on both sides.


And if I see a hardware vpn error, severity 1, I am getting a bit courious: ping with size 99 works, ping with size 100 not.




Any ideas?


William

Marcin Latosiewicz Mon, 06/28/2010 - 10:20
User Badges:
  • Cisco Employee,

William,


What would you exactly call "internal answer"?


This error appears mostly in case of those:

- Something mangling packet on the way (get capture on both end on physical interfaces)

- HW error - not likely since issue persist with and without HW encryption

- Misconfig/mismatch/not-supported config

- bug - not very likely.


That's why it was curious for me what happens with different encryption setting on both sides.


Marcin

9w.boye Mon, 06/28/2010 - 11:17
User Badges:

Hm, okay


- Something mangling packet on the way (get capture on both end on physical interfaces)

what would be mangling in the way in my testlab?


- HW error - not likely since issue persist with and without HW encryption

ok, if not hardware, then software?


- Misconfig/mismatch/not-supported config

well, why is it working with the other transform-set?

- bug - not very likely.

not very likely?

In one forum answer on such kind of issue, I found the answer to turn out cef and fast-switching ...


William

Marcin Latosiewicz Mon, 06/28/2010 - 11:53
User Badges:
  • Cisco Employee,

Ah that's the info I missed.


Which transform sets work and which ones do not?


If you suspect a bug in CEF, disable it ... it's a test lab after all...

9w.boye Tue, 06/29/2010 - 00:16
User Badges:

There are enough infos on the table.


Hardware 1 error, internal information: handle= ...

accelerator output - what is the reason for the drop of this packets


anyone else with experiences or more information about possible incompatibilities between the mentioned devices/software?


turning off cef is a joke. I don't have a aim-module on one hand to process-switch on the outer interface.


William

Marcin Latosiewicz Tue, 06/29/2010 - 02:39
User Badges:
  • Cisco Employee,

William,


If turning of CEF would help you would see different behavior when pinging FROM the device itself and through.


Which table are you refering to?


Marcin

Actions

This Discussion