Re: Remote access and site to site VPN problem.

C0mpetebv · ‎09-07-2010

Hi all.

This is my network topology:

Remote access VPN client (213.x.x.241/10.200.1.0) => Site1 ASA5520 84.x.x.98/172.16.0.0 <=L2L VPN => Site2 ASA 5505 (82.x.x.139/192.168.198.0).

Remote client must be able to connect to 192.168.198.0 Site2 when he is connected to Site1. This do not work anymore because something in config was changed.

I'm can get it working with one of following steps but this is not a sollution.

1. Clear crypto isakmp (or ipsec) sa. When tunnel is reestablished i'm can connect to 192.168.198.0 for about of couple of hours.

2. If i'm ping something in 192.168.198.0 from 10.200.1.0 i'm do not get a reply. But if i'm ping 10.200.1.0 from 192.168.198.0 i'm get reply from both sides and able to connect.

Here is debug:

gate-a# debug crypto isakmp 254
gate-a# Sep 07 09:26:33 [IKEv1]: Group = 82.x.x.139, IP = 82.x.x.139, QM FSM error (P2 struct &0x74b1f5c8, mess id 0x50edd9d0)!
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, IKE QM Initiator FSM error history (struct &0x74b1f5c8) <state>, <event>: QM_DONE, EV_ERROR-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent-->QM_SND_MSG1, EV_SND_MSG-->QM_SND_MSG1, EV_START_TMR-->QM_SND_MSG1, EV_RESEND_MSG-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, sending delete/delete with reason message
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing blank hash payload
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing IPSec delete payload
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing qm hash payload
Sep 07 09:26:33 [IKEv1]: IP = 82.x.x.139, IKE_DECODE SENDING Message (msgid=2e1fa959) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 68
BEFORE ENCRYPTION
ISAKMP Header
Initiator COOKIE: d3 1b c3 df 4f 3c d2 a7
Responder COOKIE: b1 f2 96 b0 3c 8a 44 ba
Next Payload: Hash
Version: 1.0
Exchange Type: Informational
Flags: (none)
MessageID: 59A91F2E
Length: 469762048
Payload Hash
Next Payload: Delete
Reserved: 00
Payload Length: 24
Data:
87 b5 82 90 cd 25 01 f5 9c 2f 0d a7 e1 96 0a ea
1b d8 ae 4d
Payload Delete
Next Payload: None
Reserved: 00
Payload Length: 16
DOI: IPsec
Protocol-ID: PROTO_IPSEC_ESP
Spi Size: 4
# of SPIs: 1
SPI (Hex dump): fd 47 ac ea
Sep 07 09:26:33 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, IKE Deleting SA: Remote Proxy 192.168.198.0, Local Proxy 10.200.1.0
Sep 07 09:26:33 [IKEv1]: Group = 82.x.x.139, IP = 82.x.x.139, Removing peer from correlator table failed, no match!
Sep 07 09:26:36 [IKEv1]: Group = 82.x.x.139, IP = 82.x.x.139, IKE Initiator: New Phase 2, Intf Outside, IKE Peer 82.x.x.139 local Proxy Address 10.200.1.0, remote Proxy Address 192.168.198.0, Crypto map (Outside_map)
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, Oakley begin quick mode
Sep 07 09:26:36 [IKEv1 DECODE]: Group = 82.x.x.139, IP = 82.x.x.139, IKE Initiator starting QM: msg id = 4dcc800e
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, IKE got SPI from key engine: SPI = 0xe42d0daa
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, oakley constucting quick mode
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing blank hash payload
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing IPSec SA payload
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing IPSec nonce payload
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing pfs ke payload
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing proxy ID
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, Transmitting Proxy Id:
Local subnet: 10.200.1.0 mask 255.255.255.0 Protocol 0 Port 0
Remote subnet: 192.168.198.0 Mask 255.255.255.0 Protocol 0 Port 0
Sep 07 09:26:36 [IKEv1 DEBUG]: Group = 82.x.x.139, IP = 82.x.x.139, constructing qm hash payload
Sep 07 09:26:36 [IKEv1 DECODE]: Group = 82.x.x.139, IP = 82.x.x.139, IKE Initiator sending 1st QM pkt: msg id = 4dcc800e
Sep 07 09:26:36 [IKEv1]: IP = 82.x.x.139, IKE_DECODE SENDING Message (msgid=4dcc800e) with payloads : HDR + HASH (8) + SA (1) + NONCE (10) + KE (4) + ID (5) + ID (5) + NONE (0) total length : 272
BEFORE ENCRYPTION
ISAKMP Header
Initiator COOKIE: d3 1b c3 df 4f 3c d2 a7
Responder COOKIE: b1 f2 96 b0 3c 8a 44 ba
Next Payload: Hash
Version: 1.0
Exchange Type: Quick Mode
Flags: (none)
MessageID: 0E80CC4D
Length: 469762048
Payload Hash
Next Payload: Security Association
Reserved: 00
Payload Length: 24
Data:
2d 83 44 da f9 d5 00 24 9f 13 f7 11 8f cf 02 f0
e2 5b 08 0d
Payload Security Association
Next Payload: Nonce
Reserved: 00
Payload Length: 64
DOI: IPsec
Situation:(SIT_IDENTITY_ONLY)
Payload Proposal
Next Payload: None
Reserved: 00
Payload Length: 52
Proposal #: 1
Protocol-Id: PROTO_IPSEC_ESP
SPI Size: 4
# of transforms: 1
SPI: e4 2d 0d aa
Payload Transform
Next Payload: None
Reserved: 00
Payload Length: 40
Transform #: 1
Transform-Id: ESP_3DES
Reserved2: 0000
Life Type: Seconds
Life Duration (Hex): 70 80
Life Type: Kilobytes
Life Duration (Hex): 00 46 50 00
Encapsulation Mode: Tunnel
Authentication Algorithm: SHA1
Group Description: Group 1
Payload Nonce
Next Payload: Key Exchange
Reserved: 00
Payload Length: 24
Data:
5a a1 1f 84 99 a9 c1 78 10 fd 98 d2 dc a1 a8 c7
62 9b 25 18
Payload Key Exchange
Next Payload: Identification
Reserved: 00
Payload Length: 100
Data:
37 d1 62 53 ba 3f 74 3b 52 11 59 1c ff c3 5a b5
cc 63 f4 1a 0f 7d 14 a0 cf c8 15 c6 5b 1d 76 a2
a2 73 75 e3 99 42 4a cc d9 f2 d1 5a eb e5 0d 78
bf 45 30 58 ef e1 89 f5 44 a7 e3 df 24 6d 81 6b
43 15 ce a4 88 99 e9 55 8d 58 f8 01 e6 ea 2b 38
d5 36 cd 14 46 25 ef d0 9b fb fe 78 b5 5c 76 79
Payload Identification
Next Payload: Identification
Reserved: 00
Payload Length: 16
ID Type: IPv4 Subnet (4)
Protocol ID (UDP/TCP, etc...): 0
Port: 0
ID Data: 10.200.1.0/255.255.255.0
Payload Identification
Next Payload: None
Reserved: 00
Payload Length: 16
ID Type: IPv4 Subnet (4)
Protocol ID (UDP/TCP, etc...): 0
Port: 0
ID Data: 192.168.198.0/255.255.255.0
debug crypto isakmp 254
RECV PACKET from 213.x.x.241
ISAKMP Header
Initiator COOKIE: e2 3a aa 1a 76 e1 7b e9
Responder COOKIE: bf cc 7d e3 0c 21 69 da
Next Payload: Hash
Version: 1.0
Exchange Type: Informational
Flags: (Encryption)
MessageID: A6A0E66B
Length: 84
AFTER DECRYPTION
ISAKMP Header
Initiator COOKIE: e2 3a aa 1a 76 e1 7b e9
Responder COOKIE: bf cc 7d e3 0c 21 69 da
Next Payload: Hash
Version: 1.0
Exchange Type: Informational
Flags: (Encryption)
MessageID: A6A0E66B
Length: 84
Payload Hash
Next Payload: Notification
Reserved: 00
Payload Length: 24
Data:
10 2c 03 90 6e 81 ea b6 07 0b 03 72 60 a2 5b 3a
da c4 6b c2
Payload Notification
Next Payload: None
Reserved: 00
Payload Length: 32
DOI: IPsec
Protocol-ID: PROTO_ISAKMP
Spi Size: 16
Notify Type: R_U_THERE
SPI:
e2 3a aa 1a 76 e1 7b e9 bf cc 7d e3 0c 21 69 da
Data: 88 b1 0f 73
Sep 07 09:26:39 [IKEv1]: IP = 213.x.x.241, IKE_DECODE RECEIVED Message (msgid=a6a0e66b) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 84
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, processing hash payload
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, processing notify payload
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, Received keep-alive of type DPD R-U-THERE (seq number 0x88b10f73)
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, Sending keep-alive of type DPD R-U-THERE-ACK (seq number 0x88b10f73)
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, constructing blank hash payload
Sep 07 09:26:39 [IKEv1 DEBUG]: Group = Remote, Username = user, IP = 213.x.x.241, constructing qm hash payload
Sep 07 09:26:39 [IKEv1]: IP = 213.x.x.241, IKE_DECODE SENDING Message (msgid=fb1af155) with payloads : HDR + HASH (8) + NOTIFY (11) + NONE (0) total length : 84
BEFORE ENCRYPTION
ISAKMP Header
Initiator COOKIE: e2 3a aa 1a 76 e1 7b e9
Responder COOKIE: bf cc 7d e3 0c 21 69 da
Next Payload: Hash
Version: 1.0
Exchange Type: Informational
Flags: (none)
MessageID: 55F11AFB
Length: 469762048
Payload Hash
Next Payload: Notification
Reserved: 00
Payload Length: 24
Data:
63 0a 42 a6 61 5f dd 4f 10 13 31 9d f0 ac c7 d8
56 e9 e8 28
Payload Notification
Next Payload: None
Reserved: 00
Payload Length: 32
DOI: IPsec
Protocol-ID: PROTO_ISAKMP
Spi Size: 16
Notify Type: R_U_THERE_ACK
SPI:
e2 3a aa 1a 76 e1 7b e9 bf cc 7d e3 0c 21 69 da
Data: 88 b1 0f 73

Crypto map tag: SYSTEM_DEFAULT_CRYPTO_MAP, seq num: 65535, local addr: 84.x.x.98

      local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      remote ident (addr/mask/prot/port): (10.200.1.19/255.255.255.255/0/0)
      current_peer: 213.x.x.241, username: user
      dynamic allocated peer ip: 10.200.1.19

      #pkts encaps: 364, #pkts encrypt: 364, #pkts digest: 364
      #pkts decaps: 1017, #pkts decrypt: 1017, #pkts verify: 1017
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 364, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 0
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (rcv): 0,
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: 84.x.x.98/4500, remote crypto endpt.: 213.x.x.241/59829
      path mtu 1500, ipsec overhead 82, media mtu 1500
      current outbound spi: 8B6FF16B
      current inbound spi : 12D184DA

    inbound esp sas:
      spi: 0x12D184DA (315720922)
         transform: esp-aes esp-sha-hmac no compression
         in use settings ={RA, Tunnel, NAT-T-Encaps, }
         slot: 0, conn_id: 2715648, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (sec): 25989
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x8B6FF16B (2339369323)
         transform: esp-aes esp-sha-hmac no compression
         in use settings ={RA, Tunnel, NAT-T-Encaps, }
         slot: 0, conn_id: 2715648, crypto-map: SYSTEM_DEFAULT_CRYPTO_MAP
         sa timing: remaining key lifetime (sec): 25978
         IV size: 16 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

IPsec Global Statistics
-----------------------
Active tunnels: 6
Previous tunnels: 868
Inbound
    Bytes: 0
    Decompressed bytes: 0
    Packets: 470303871
    Dropped packets: 11682
    Replay failures: 5489
    Authentications: 470292189
    Authentication failures: 11678
    Decryptions: 470292192
    Decryption failures: 0
    Decapsulated fragments needing reassembly: 8117
Outbound
    Bytes: 0
    Uncompressed bytes: 0
    Packets: 431900456
    Dropped packets: 1449
    Authentications: 431901476
    Authentication failures: 0
    Encryptions: 431901476
    Encryption failures: 0
    Fragmentation successes: 1025
        Pre-fragmentation successses: 1025
        Post-fragmentation successes: 0
    Fragmentation failures: 0
        Pre-fragmentation failures: 0
        Post-fragmentation failures: 0
    Fragments created: 2050
    PMTUs sent: 0
    PMTUs rcvd: 0
Protocol failures: 0
Missing SA failures: 1444
System capacity failures: 0

Jitendriya Athavale · ‎09-07-2010

when you say you can ping/connect in one direction and not in the other, here is wht i think

check firewall on .198 hosts and disable them, probably that is stopping them from responding to pings

if pings works and tcp connections do not work there might be asymmetric routing

to verify this collect captures on the inside interface of site 2 and see if you see the traffic coming in and going out

also trying pinging inside interface of site 2 asa by issueing the command management-interface inside

C0mpetebv · ‎09-09-2010

There are no any other firewalls.

I'm able to connect to 192.168.198.0 but only if connection first initiated FROM this network.

ping, telnet whatever://10.200.1.my ip from 192.168.198.0 solves problem immidately.

Also clearing SAs solves it.

hdashnau · ‎09-09-2010

Are the crypto access-lists for the L2L tunnel exact mirror images of one another (there should be the same number of acl entries on both sides and the ips and subnet masks need to be consistant accross both)? It could be the tunnel is only coming up in one direction because one side has a more specific crypto acl entry than the other.

C0mpetebv · ‎09-09-2010

Yes they are same:

Site 1:

access-list outside_2_cryptomap; 4 elements; name hash: 0x8d0d4873
access-list outside_2_cryptomap line 1 extended permit ip 192.168.198.0 255.255.255.0 object-group DM_INLINE_NETWORK_2 0xfa10f39a
access-list outside_2_cryptomap line 1 extended permit ip 192.168.198.0 255.255.255.0 Vlan1 255.255.255.0 (hitcnt=19) 0x1d71c630
access-list outside_2_cryptomap line 1 extended permit ip 192.168.198.0 255.255.255.0 Vlan112 255.255.255.0 (hitcnt=68) 0xc2145285
access-list outside_2_cryptomap line 1 extended permit ip 192.168.198.0 255.255.255.0 Vlan107 255.255.255.0 (hitcnt=58) 0x322619bd
access-list outside_2_cryptomap line 1 extended permit ip 192.168.198.0 255.255.255.0 RemoteVPN 255.255.255.0 (hitcnt=9) 0x730769cd

Site 2:

access-list Outside_cryptomap_1; 4 elements; name hash: 0xf92983d
access-list Outside_cryptomap_1 line 1 extended permit ip object-group DM_INLINE_NETWORK_5 192.168.198.0 255.255.255.0 0xb7467309
access-list Outside_cryptomap_1 line 1 extended permit ip 10.200.1.0 255.255.255.0 192.168.198.0 255.255.255.0 (hitcnt=0) 0x418e95c6
access-list Outside_cryptomap_1 line 1 extended permit ip 172.16.x.0 255.255.255.0 192.168.198.0 255.255.255.0 (hitcnt=0) 0x2d8163f5
access-list Outside_cryptomap_1 line 1 extended permit ip 172.16.x.0 255.255.255.0 192.168.198.0 255.255.255.0 (hitcnt=0) 0x8c1cb964
access-list Outside_cryptomap_1 line 1 extended permit ip 172.16.x.0 255.255.255.0 192.168.198.0 255.255.255.0 (hitcnt=0) 0x75dbc647