IPSEC #pkts internal err (recv)

Unanswered Question
Jun 5th, 2014
User Badges:

Anyone have any ideas about these "#pkts internal err (recv)"?

Configs match on both sides.  Packets are mostly getting thru fine, have about 3-5% packet loss across the IPSEC tunnel.

It's a tunnel interface with a protection profile. (ipsec ipv4 tunnel)

Router at other end shows no errors!!!

Thanks!

 

Router#sh crypto ipsec sa det

interface: Tunnel0
    Crypto map tag: Tunnel0-head-0, local addr 10.1.1.200

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer 10.100.1.1 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 383492, #pkts encrypt: 383492, #pkts digest: 383492
    #pkts decaps: 132322, #pkts decrypt: 132322, #pkts verify: 132322
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #pkts no sa (send) 1, #pkts invalid sa (rcv) 0
    #pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
    #pkts invalid prot (recv) 0, #pkts verify failed: 0
    #pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
    #pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
    ##pkts replay failed (rcv): 0
    #pkts internal err (send): 0, #pkts internal err (recv) 22562

     local crypto endpt.: 10.1.1.1, remote crypto endpt.: 10.100.1.1
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0x8163F0F(135675663)

     inbound esp sas:
      spi: 0xCE66620D(3462816269)
        transform: esp-256-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2013, flow_id: NETGX:13, crypto map: Tunnel0-head-0
        sa timing: remaining key lifetime (k/sec): (4451102/2801)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 512
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x8163F0F(135675663)
        transform: esp-256-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2014, flow_id: NETGX:14, crypto map: Tunnel0-head-0
        sa timing: remaining key lifetime (k/sec): (4189345/2801)
        IV size: 16 bytes
        replay detection support: Y  replay window size: 512
        Status: ACTIVE

     outbound ah sas:

     outbound pcp sas:
Router#

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Vasilii Mikhail... Fri, 06/06/2014 - 05:05
User Badges:
  • Gold, 750 points or more

Hello.

What hardware do you use and what IOS (sh ver)?

If it's a brand new IOS, then it could be an IOS issue.

If not, I would suggest to check MTU and TCP MSS applied on tunnel (to avoid fragmentation).

Do you have any QoS service-policy applied on Tunnel or respective physical interface?

PS: if doesn't help, please try "debug crypto engine error" (and provide us the output).

mattp0002 Fri, 06/06/2014 - 06:32
User Badges:

Vasilii,

 

Both ends are 2811 routers running ios 124-15.T9 advanced IP services.  Not high end stuff but good enough for this connection I think.

 

I've played around with the tunnel interface MTU and TCP MSS settings.  I've also enabled "tunnel path-mtu-discovery" on both tunnel interfaces.  No help.

 

Personally I don't really understand why those settings would effect the connection.  The routers are connected together by serial interfaces ppp multilink bundled together across a MPLS wan.  Those interfaces show a few crc errors but statistically they are very little.

 

I am getting 3-5% packet loss on a ping going through this tunnel using the default ping size. (small packets).  No QOS enabled anywhere on this network.

 

I tried debug crypto engine error (after doing "term mon" on my ssh session) and I see NOTHING.  So strange!!

 

PS - I have an ip policy on the fast ethernet interfaces (LAN-facing) of both routers stripping DF flags off all inbound packets.  This was to correct problems which appeared after I deployed GRE tunnels across the MPLS WAN. (in order to run OSPF).  To help with the CPU I have also enabled "ip route-cache policy" on those LAN interfaces...

Vasilii Mikhail... Fri, 06/06/2014 - 09:30
User Badges:
  • Gold, 750 points or more

Hello.

I'll try to play with MTU in lab, but I guess this issue might be due to fragmentation.

Do you experience packet loss if link is idle (utilization is below 10% or capacity)? What packet loss do you observing pinging outside the tunnel?

Do you experience the same issue if you keep only one serial link in your bundle?

PS: if you run IPSec with GRE, then you would better set ip mtu 1400 on tunnel interface and TCP MSS 1360. In this case there is no need to clear df-bit.

PS2: could you please share your tunnel and IPSec configuration + sh ver + sh inv?

mattp0002 Fri, 06/06/2014 - 11:30
User Badges:

V,

Now I am very confused.  I put on the ip mtu and TCP MSS settings you recommended, and now the link is working fine.  Some packet errors are still showing up but MUCH fewer than before.  Also a ping thru the tunnel is now 0% packet loss.  Thank you!

I also ran "mturoute" (i.e. tracepath) through the tunnel and it is reporting a Path MTU of 1500.  This is confusing me as I thought the configuration you had me put on the tunnel interface would have lowered the maximum possible MTU across the tunnel to 1400?

 

Here's the tunnel & crypto config. (identical/matching config on other router)

interface Tunnel0
 description IPSEC tunnel
 ip address [ip] 255.255.255.252
 ip mtu 1400
 ip tcp adjust-mss 1360
 tunnel source [ip]
 tunnel destination [ip]
 tunnel mode ipsec ipv4
 tunnel path-mtu-discovery
 tunnel protection ipsec profile TunnelProfile
end

!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp key [key] address [ip]
!
crypto ipsec security-association replay window-size 512
!
crypto ipsec transform-set MyTS esp-aes 256 esp-sha-hmac
!
crypto ipsec profile TunnelProfile
 set transform-set MyTS
 set pfs group2
!

Vasilii Mikhail... Sat, 06/07/2014 - 00:39
User Badges:
  • Gold, 750 points or more

Hello.

I would remove tunnel path-mtu-discovery command, as it's not required, but could do a mess.

I would also suggest to check if you have command crypto isakmp invalid-spirecovery and crypto isakmp keepalive 

Also it's better to add bandwidth statement on the interface if you run any routing protocol (especially EIGRP).

PS: your mturoute could be deceived by route-map removing df-bit. 

Vasilii Mikhail... Sat, 06/07/2014 - 00:47
User Badges:
  • Gold, 750 points or more

Lessen number of errors means that it's some sort of fragmentation issue.

You might still see them, as your configuration adjusts TCP (MSS) flows only, but UDP still trying to use MTU 1500 and your route-map removes df-bit, so fragmentation is still possible.

PS: here is an article about IPSec fragmentation.

PS2: I would say, that fragmentation over WAN should be avoided at any cost, unless there is some strict business requirement.

mattp0002 Mon, 06/09/2014 - 06:49
User Badges:

Vasilii,

 

I think I spoke too soon on Friday.  Errors are still there.  I have implemented the new settings (bandwidth statement, keepalive, siprecovery) and removed the path-mtu-discovery.

 

I have also dropped the crypto down from aes-256 to simple des just in case there was a cpu issue or something.

 

I have also played around with the frag-before-encryption and frag-after-encryption settings.

 

Errors are still happening, ONLY ON ONE SIDE OF THE LINE.  Still 5% packet loss from a normal 32-byte continuous ping through the tunnel.  When I do a ping outside of the tunnel, I get no packet loss. :(

 

Could it be a "dirty" serial WAN line?  I don't see errors reported on any serial interfaces.  Also that unencrypted ping is working great. :( :( :(

 

I am now considering trashing this "new" VTI configuration (crypto profile applied as tunnel protection) and maybe configuring an old-fashioned gre which has ipsec applied with a route-map on the WAN interface.

 

Or, in other words, GRE-over-IPSEC instead of IPSEC-in-Tunnel.  If that makes sense?  It's just more of a pain to configure than this VTI.



What do you think?  I'm so confused because it's only happening on one side and not the other.

mattp0002 Mon, 06/09/2014 - 07:08
User Badges:

I've done some extended-command pings sourced at the router.  With the DF-bit set, 1300 is the largest ping I can get across the WAN to the other router.  Therefore I just changed the tunnel ip mtu to 1200 and the tcp adjust to 1160.  Still getting the errors.

 

But, when I do a sh crypto ipsec sa detail, I see the following line:

path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0

 

This will only change if I change the mtu on my Fa0/0 interface (LAN-facing).  I am using this interface(s)as the source/destination of the tunnel because these routers don't have loopback interfaces. (no idea, previous network guys have been fired that's why I'm here).

 

Could this be related?

Actions

This Discussion