VPN Tunnel Failure on 3945 Routers Running 152-4.M1

[Paul Wishart]

I'm experiencing a failure on headend 3945 routers with VPN tunnels to remote 2901 routers. Essentially, a tunnel a a 3945 will go down/down although the tunnel on the remote router indicates it is up/up.  It happens intermittently and I am not seeing anything in the logs, other than the tunnel goes down.  This seems so much like an IOS bug, but I can't find anything specific in caveats on this version of code.  Any ideas?

Paul Wishart

[oamarneh]

Hi Paul,

so the tunnels are up, but suddenly they go down? when this happens, how do you recover from this situation? do you clear the SAs? or it comes back up on its own?

have you tried to enable isakmp  keepalives? the command is "crypto isakmp keepalive 10 3 periodic"

if that didnt help, we need to have the output 'debug crypto isakmp' and 'debug crypto ipsec' when the issue happens.

hope that helps.

Othman

[Paul Wishart]

Thank you, Othman,  Yes, I forgot to mention that a simple shut/no shut brings the 3945 side of the tunnel back up and functioning.  I see the router was originally configured with "crypto isakmp nat keepalive 5" which I'm confused about as there is no nat configured on the router and the VPN endpoints.  I will add the keepalive command you suggested and see if this helps.

[oamarneh]

thanks for your eamil Paul, you also didnt mention that Tunnel interface is involved with this VPN setup

so is it a GRE over IPSec tunnel? or SVTI? or DMVPN?

do you know the status of the tunnel interface when the issue happens? do you get any logs about tunnel interface going down?

Regards,

Othman

[Paul Wishart]

The tunnels are just virtual tunnel interface, so GRE and no DMVPN.  Here is an example of one of the interface configs:

interface Tunnel1

description Tunnel to XXX

ip address 10.1.1.2 255.255.255.252

tunnel source X.X.X.X

tunnel mode ipsec ipv4

tunnel destination X.X.X.X

tunnel protection ipsec profile Customer_Connect

end

Also, is a keepalive statement on the tunnel interface the same as using the crypto isakmp keepalive command?

All I am seeing in the logs is an entry stating the tunnel went down.

Paul

[oamarneh]

Hello Paul,

since you have the command "tunnel mode ipsec ipv4", this is no longer a GRE tunnel, this is a pure ipsec tunnel that is called SVTI.

is the other side configured with the same command under the tunnel interface? can you provide the log messages you get when the tunnel go down?

keepalives under the tunnel interface are GRE keepalives, and this tunnel is an SVTI tunnel, so we cannot use the keepalives under the tunnel interface, it should be used with the command i provided earlier.

Regards,

Othman

[Paul Wishart]

Othman,

Your input has been invaluable thus far.  The other side is also configured as an SVTI interface.

On a side note, we are simply doing static routing from the spoke routers pointing back to the head-end.  If we wish to move to a dynamic routing protocol, those tunnels will need to be converted to GRE, I believe.  Is that correct?

Paul

[oamarneh]

Hello Paul,

as it is an SVTI tunnel, here is the action plan for now:

- collect 'debug crypto isakmp' and 'debug crypto ipsec' when the issue happens again.

- collect syslog from the router as well.

- collect 'show crypto ipsec sa' and 'show interface tunnel '

for the other question, you can use dynamic routing protocol with SVTI, no need to convert it to GRE.

Regards,

Othman

[Paul Wishart]

Thanks very much.  I will start with this and keep you posted.

[Paul Wishart]

I just had a tunnel drop on one of the 3945 and this is out of the logs:

Mar 14 16:11:06.523 EDT: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
        connection id=2636, sequence number=5078

Mar 14 16:57:32.510 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel11, changed state to down
Mar 14 16:57:39.276 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel11, changed state to up
Mar 14 16:59:26.248 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel11, changed state to down

[oamarneh]

line protocol down on tunnel interface with SVTI means that the IPSec tunnel went down. have you managed to collect the other debugs mentioned?

one more question, what is the the IOS image running on both routers here?

[Paul Wishart]

We are running 152-4.M1 on both routers.  I have collected the debugs as well.  Should I post them in this forum?

[oamarneh]

Hello Paul,

thanks for the email. you can send the debugs to my email address oamarneh@cisco.com

however, i see that this issue is exactly matching the symptoms of the following bug:

CSCub74272    line protocol down during Phase II rekey on VTI

the bug details can be found here:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCub74272

this is affecting version 15.2.4M1 you are currently running, and is fixed in the latest version of 15.2.4M3

please upgrade to that IOS and everything should go fine after that.

Regards,

Othman