IPSEC site-site vpn stopped working after 3 months..

[monkeyboy]

Hello, we have a site-site ipsec vpn set up using pre-shared keys.

We have not changed the config - however can no longer establish ipsec sa's to the remote site - this is after 3 months working correctly.

Below is the debug output from the remote monaco pix - we are using nat-traversal as there is an adsl modem at the monaco end - which has to do nat because it has our only public IP - this is natted to 192.168.1.1 - the monaco pix's real outside address..

It seems that the vpn is failing after nat-t negotiations.. and it seems that it is still looking at port 500 - when all the sa's have previously been on 4500

when I do a 'sh isakmp sa' on the monaco pix it shows:

MM_SA_SETUP

which would indicate a difference in the 2 configs - however there is none - also I've recreated the vpn from scratch - same thing

Monaco debug output:

OAK_MM exchange

ISAKMP (0): processing SA payload. message ID = 0

ISAKMP (0): Checking ISAKMP transform 1 against priority 1 policy

ISAKMP: encryption AES-CBC

ISAKMP: keylength of 256

ISAKMP: hash SHA

ISAKMP: default group 1

ISAKMP: auth pre-share

ISAKMP: life type in seconds

ISAKMP: life duration (basic) of 120

ISAKMP (0): atts are acceptable. Next payload is 0

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T

ISAKMP (0): processing vendor id payload

ISAKMP (0:0): vendor ID is NAT-T

ISAKMP (0): SA is doing pre-shared key authentication using id type ID_IPV4_ADDR

ISAKMP (0:0): sending NAT-T vendor ID - rev 2 & 3

ISAKMP (0:0): Detected port floating

return status is IKMP_NO_ERROR

crypto_isakmp_process_block:src:PIXINLIVERPOOLOUTSIDEADDRESS, dest:192.168.1.1 spt:500 dpt:500

VPN Peer:ISAKMP: Peer Info for PIXINLIVERPOOLOUTSIDEADDRESS/500 not found - peers:0

ISAKMP: larval sa found

ISAKMP (0): retransmitting phase 1 (0)...

crypto_isakmp_process_block:src:PIXINLIVERPOOLOUTSIDEADDRESS, dest:192.168.1.1 spt:500 dpt:500

VPN Peer:ISAKMP: Peer Info for PIXINLIVERPOOLOUTSIDEADDRESS/500 not found - peers:0

ISAKMP: larval sa found

[jmia]

Can you post configuration from both PIX, (take out any sesitive info), do you have L3 connectivity between both peers? i.e. can you ping the outside IP of PIX-B from PIX-A?

Let me know...

Jay

[jackko]

you mentioned no modification of config has been made, how about ios upgrade?

also, it maybe worth to try disabling the crypto map on the interface and re-enable it again. sometimes it helps.

[monkeyboy]

..there hasn't been an ios upgrade either

the two pixes in question have remained the same since installation - I've (since the problem) also ripped the configs out of both & reconfigured - reapplied crypto maps saved to NV - rebooted - you name it..

the config is basically -

no nat for ipsec traffic - the two lan networks

ipsec made interesting

esp tunnels

same transform set - pre shared keys

usual stuff - as I said it was working like a dream for around 3 months...

I'll post the configs tomorrow when I'm in work

..it really seems that the pixes aren't negotiating nat-t properly as the sa is trying to establish on udp port 500..

cheers

mark

[monkeyboy]

I've attached the configs and also a basic diagram.

I really can't see how it would fail after such a long time working

btw the pix in liverpool also peers with warrington pix - that vpn is working fine but it doesn't need to use nat-t

thanks for the help so far..

[jmia]

I had a quick look at both of the configs and it looks OK at first glance, you say that this same config was working for 3 months and then stoped. Can you answer my original question:

From your liverpool-pix can you ping the outside IP of the modem and vice-versa? To check that you have L3 connectivity. Also, have you checked with your ISP that they are not blocking UDP port 4500, used by NAT-T.

Also, have you issued - in config mode - clear cry isakmp sa and clear cry ipsec sa on both pixes and then try to ping from internal client at pix-liverpool to an internal client at modem/pix side. What does sho isakmp sa now show?

Jay

[monkeyboy]

Hello Jay/all - as requested:

enabled icmp on outside of liverpool & monaco pixes

ping from liverpool pix to monaco adsl - fail

ping from monaco adsl to liverpool pix - fail

ping from sprint nap london to monaco adsl - success

ping from sprint nap london to liverpool pix - fail

----------------------------------------------------------

monaco-pix(config)# clear cry isakmp sa

monaco-pix(config)# clear cry ipsec sa

monaco-pix(config)#

monaco-pix# sh isakmp sa

Total : 1

Embryonic : 1

dst src state pending created

XXX.121.200.146 192.168.1.1 MM_NO_STATE 0 0

*****I instigated a telnet session from liverpool to a host inside monaco

monaco-pix# sh isakmp sa

Total : 2

Embryonic : 2

dst src state pending created

192.168.1.1 XXX.121.200.146 MM_SA_SETUP 0 0

XXX.121.200.146 192.168.1.1 MM_NO_STATE 0 0

monaco-pix# sh isakmp sa

Total : 1

Embryonic : 1

dst src state pending created

192.168.1.1 XXX.121.200.146 MM_SA_SETUP 0 0

monaco-pix# sh isakmp sa

Total : 1

Embryonic : 1

dst src state pending created

192.168.1.1 XXX.121.200.146 MM_SA_SETUP 0 0

monaco-pix# sh isakmp sa

Total : 1

Embryonic : 1

dst src state pending created

192.168.1.1 XXX.121.200.146 MM_SA_SETUP 0 0

*****here's liverpool pix's output for the same connection:

liverpool-pix# sh isakmp sa

Total : 1

Embryonic : 1

dst src state pending created

XXX.94.100.22 XXX.121.200.146 MM_NO_STATE 0 0

**I've also mailed our ISP here in UK and also Monaco to make sure that they're not blocking nat-t..

cheers

Mark

[jackko]

as you mentioned, the configs are very tidy and straight forward.

just wondering if the issue is with the adsl router at monaco performing nat. please verify whether the nat is functioning.

[monkeyboy]

Yeah, I was wondering about that - the only thing is I can ssh to the Monaco router from here in Liverpool.. so it must be NATting to the outside port of the firewall successfully for that public IP - also I can see the ipsec traffic coming in on Monaco pix - just that it's trying to use port 500 and getting denied:

VPN Peer:ISAKMP: Peer Info for XXX.121.200.146/500 not found - peers:0

[jmia]

If you can not establish L3 connectivity between the two problamatic peers and also the sho isakmp sa is indicating MM_NO_STATE - This means that there is a problem with communication link between the two peers. I would question your ISP on this.

Jay

[monkeyboy]

Would that be the Monaco ISP, UK or both?

I've looked at the debug and it seems it's failing when it's trying to authenticate using IP address

many thanks

[jmia]

It could be either side, hard to tell check with both providers and make sure that your equipment at both ends are functiong correctly too. What type of service are you running, xDSL?

Jay

[monkeyboy]

Yeah, adsl -

I've been testing making telnet connections from liverpool to monaco and seeing the connections on the monaco pix - but then failing

I wondered whether the monaco pix was receiving the ipsec - going back to liverpool to authenticate via ip address and couldn't get back to liverpool possibly because of the nat on the adsl

so I used a backup isdn to get onto a router on monaco - put debug cry isakmp on liverpool and tried a connection from monaco to liv and sure enough no debug on liverpool

so..

we have another public subnet here in l'pool - so we moved the pix onto this and made all the necessary changes - also cleared isakp/ipsec - reapplied crypto maps - - now we can ping liverpool pix's new ip address

I can still ssh to monaco pix via the adsl

now when we try to telnet from liverpool to monaco - it doesn't show up on Monaco's debug anymore - makes me think that it's the cruddy adsl modem..

[jmia]

Good troubleshooting! I did have my suspicion on the modem at Monaco, and now reading your reply do believe that it might be the modem that is creating your problem but do also check with your ISP that there isn’t any issues with the comms link at both end.

Good luck.

Jay

[jackko]

just wondering how you go with the adsl modem.