Re: ASA 5510 Problem

meameame12 · ‎10-08-2010

Hi, I have an issue--basically all of the tunnels terminating on a 2811 router are up/down. They were up earlier today, but we made some changes on teh ASA, tried to revert but cannot seem to find the problem. Posted below is the configuration...

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_A

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_B

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_C

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_D

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_E

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Remote_Site_F

access-list inside_nat0_outbound extended permit ip 10.0.0.0 255.0.0.0 192.168.43.0 255.255.255.0

access-list inside_nat0_outbound extended permit ip 10.0.0.0 255.0.0.0 192.168.44.0 255.255.255.0

access-list inside_nat0_outbound extended permit ip any 10.233.59.0 255.255.255.0

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Loop_A

access-list inside_nat0_outbound extended permit ip host MAIN_SWITCH host Loop_B

access-list Security-PA2 extended permit ip host MAIN_SWITCH host Remote_Site_B

access-list Security-PA3 extended permit gre host MAIN_SWITCH host Remote_Site_A

access-list Security-NJ extended permit gre host MAIN_SWITCH host Remote_Site_C

access-list Security-PA extended permit ip host MAIN_SWITCH host Loop_B

access-list Split_Tunnel_Acl standard permit 10.0.0.0 255.0.0.0

access-list Split_Tunnel_Acl standard permit 192.168.128.0 255.255.255.0

access-list Split_Tunnel_Acl standard permit 192.168.160.0 255.255.255.0

access-list Split_Tunnel_Acl standard permit 192.168.144.0 255.255.255.0

access-list Split_Tunnel_Acl standard permit 192.168.130.0 255.255.255.0

access-list Split_Tunnel_Acl standard permit 192.168.140.0 255.255.255.0

access-list Security-USA extended permit gre host MAIN_SWITCH host Remote_Site_D

access-list ISA-Server extended permit object-group tcpudp host 192.168.44.190 object-group DC_Server_Group object-group DNS_tcp-udp

access-list ISA-Server extended permit object-group tcpudp host 192.168.44.190 object-group DC_Server_Group object-group Kerberos

access-list ISA-Server extended permit tcp host 192.168.44.190 host Exchange_Server eq https

access-list ISA-Server extended permit object-group tcpudp host 192.168.44.190 object-group DC_Server_Group eq 389

access-list ISA-Server extended permit tcp host 192.168.44.190 host 10.233.43.85 eq https

access-list ISA-Server extended permit tcp any any eq https

access-list ISA-Server extended permit icmp any any

access-list ISA-Server extended permit tcp host 192.168.44.190 object-group PatchLink-Servers eq www

access-list ISA-Server extended permit tcp host 192.168.44.190 host touchpointdemo eq www

access-list ISA-Server extended permit tcp host 192.168.44.190 host touchpointdemo2 eq www

access-list extdmz-in extended permit tcp host 192.168.43.189 object-group ALL-SMTP-Servers eq smtp

access-list extdmz-in extended permit tcp host 192.168.43.189 object-group OUTSIDE-CONTRACTOR eq smtp

access-list extdmz-in extended permit tcp host 192.168.33.27 object-group Wyeth_SMTP eq smtp

access-list extdmz-in extended permit tcp host 192.168.43.190 any eq www

access-list extdmz-in extended permit tcp host 192.168.43.189 any eq domain

access-list extdmz-in extended permit tcp host 192.168.43.190 any eq domain

access-list extdmz-in extended permit udp host 192.168.43.189 any eq domain

access-list extdmz-in extended permit udp host 192.168.43.190 any eq domain

access-list extdmz-in extended permit tcp host 192.168.43.189 any eq https

access-list extdmz-in extended permit tcp host 192.168.43.189 any eq ftp

access-list extdmz-in extended permit tcp host 192.168.43.190 any eq ftp

access-list extdmz-in extended permit tcp host 192.168.43.190 any eq https

access-list Security-Calif extended permit gre host MAIN_SWITCH host Remote_Site_F

access-list outside-in extended permit tcp any host ISA_Server_Ext eq https

access-list outside-in extended permit icmp any any object-group ping

access-list outside-in extended permit tcp object-group MSG_LAB host Ext_SMTP_Ext eq smtp

access-list outside-in extended permit tcp object-group MessageLabs_SMTP host Ext_SMTP_Ext eq smtp

access-list outside-in extended permit tcp object-group OUTSIDE-CONTRACTOR host Ext_SMTP_Ext eq smtp

access-list outside-in extended permit tcp any host ISA_Server_Ext eq www

access-list outside-in extended permit udp object-group iNet_Hdw host 10.233.43.251 eq tftp

access-list inside-in extended permit tcp any object-group ExtDMZ-Servers object-group RemoteManagement

access-list inside-in extended permit tcp any host 192.168.44.190 object-group RemoteManagement

access-list inside-in extended permit tcp any object-group ExtDMZ-Servers eq www

access-list inside-in extended permit tcp host Exchange_Server host ISA-Server_Inside eq https

access-list inside-in extended permit tcp object-group Internal_SMTP host Ext_SMTP eq smtp

access-list NJ-VPN extended permit ip host MAIN_SWITCH host Loop_A

global (outside) 1 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 0.0.0.0 0.0.0.0

crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ESP-AES-256_ESP-SHA esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-3DES-MD5-DH7 esp-3des esp-md5-hmac

crypto dynamic-map outside_dyn_map 20 set transform-set ESP-3DES-SHA ESP-AES-256_ESP-SHA

crypto map outside_map 20 match address Security-PA3

crypto map outside_map 20 set peer y.y.y.y

crypto map outside_map 20 set transform-set ESP-AES-256-SHA

crypto map outside_map 30 match address Security-PA2

crypto map outside_map 30 set peer x.x.x.x

crypto map outside_map 30 set transform-set ESP-AES-256-SHA

crypto map outside_map 40 match address Security-NJ

crypto map outside_map 40 set peer x.x.x.x

crypto map outside_map 40 set transform-set ESP-AES-256-SHA

crypto map outside_map 45 set peer y.y.y.y

crypto map outside_map 45 set transform-set ESP-AES-256_ESP-SHA

crypto map outside_map 50 set peer x,x,x,x

crypto map outside_map 50 set transform-set ESP-AES-256-SHA

crypto map outside_map 60 match address Security-USA

crypto map outside_map 60 set peer x,x,xx

crypto map outside_map 60 set transform-set ESP-AES-256-SHA

crypto map outside_map 70 match address Security-Calif

crypto map outside_map 70 set peer x.x.x.x

crypto map outside_map 70 set transform-set ESP-AES-256_ESP-SHA

crypto map outside_map 71 match address Security-Calif

crypto map outside_map 71 set peer x.x.x.x

crypto map outside_map 71 set transform-set ESP-AES-256_ESP-SHA

crypto map outside_map 80 match address NJ-VPN

crypto map outside_map 80 set peer x.x.x.x

crypto map outside_map 80 set transform-set ESP-AES-256_ESP-SHA

crypto map outside_map 90 set peer x.x.x.x

crypto map outside_map 90 set transform-set ESP-AES-256_ESP-SHA

crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map

crypto map outside_map interface outside

isakmp identity address

isakmp enable outside

isakmp policy 4 authentication pre-share

isakmp policy 4 encryption 3des

isakmp policy 4 hash md5

isakmp policy 4 group 7

isakmp policy 4 lifetime 86400

isakmp policy 10 authentication pre-share

isakmp policy 10 encryption aes-256

isakmp policy 10 hash sha

isakmp policy 10 group 5

isakmp policy 10 lifetime 86400

isakmp policy 30 authentication pre-share

isakmp policy 30 encryption 3des

isakmp policy 30 hash sha

isakmp policy 30 group 2

isakmp policy 30 lifetime 86400

isakmp policy 65535 authentication pre-share

isakmp policy 65535 encryption 3des

isakmp policy 65535 hash sha

isakmp policy 65535 group 2

isakmp policy 65535 lifetime 86400

isakmp nat-traversal 20

tunnel-group DefaultRAGroup ipsec-attributes

isakmp keepalive threshold 10

:

pre-shared-key *

tunnel-group a.b.d.h type ipsec-l2l

tunnel-group a.b.d.h ipsec-attributes

pre-shared-key *

tunnel-group VPN type ipsec-ra

tunnel-group VPN general-attributes

address-pool pool1

authentication-server-group radius

default-group-policy VPN

strip-realm

tunnel-group VPN ipsec-attributes

pre-shared-key *

radius-with-expiry

tunnel-group a.b.c.d type ipsec-l2l

tunnel-group a.b.c.d ipsec-attributes

pre-shared-key *

:

Here are the errors:

On the 2811 router, which terminates the tunnel (since the ASA is version 7 and doesnt run EIGRP:

1#show ip int brief

Interface IP-Address OK? Method Status Protocol

FastEthernet0/1 10.x.t.100 YES NVRAM up up

Loopback1 10.x.t.197 YES NVRAM up up

Tunnel11 10.x.t.149 YES NVRAM up down

Tunnel12 10.x.t.157 YES NVRAM up down

Tunnel13 10.x.t.165 YES NVRAM up down

Tunnel15 10.x.t.233 YES NVRAM up down

Tunnel16 10.x.t.245 YES NVRAM up down

Tunnel17 10.x.t.117 YES NVRAM up down

Tunnel19 10.x.t.146 YES NVRAM up down

And on the ASA:

[IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0x35b7bb4c)!

Oct 08 21:36:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

asa# Oct 08 21:37:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0xbfdc6fd1)!

Oct 08 21:37:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

Oct 08 21:37:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0xa61c06a9)!

Oct 08 21:37:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

Oct 08 21:38:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0x35dc0cd4)!

Oct 08 21:38:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

asa# Oct 08 21:38:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0x448cd5f8)!

Oct 08 21:38:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

Oct 08 21:39:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0xe6c95f0f)!

Oct 08 21:39:18 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

asa#

asa# show run | iOct 08 21:39:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, QM FSM error (P2 struct &0x3e747d0, mess id 0x245ada99)!

Oct 08 21:39:48 [IKEv1]: Group = a.b.c.d, IP = a.b.c.d, Removing peer from correlator table failed, no match!

Please assist!

Ive tried for hours but I think the problem is phase 2 and the crypto map....all spokes are down.

Jennifer Halim · ‎10-08-2010

I assume that you are running GRE tunnel on the 2811 behind the ASA, and ASA is establishing the IPSec tunnel.

I didn't see access-group command posted earlier, however, i assume that your 2811 router is connected to your ASA inside interface, and acl assigned is named "inside-in". I did not see ACL line that permits GRE that would allow the traffic through the ASA.

meameame12 · ‎10-08-2010

hi, yes, your assumptions are right--the 2811 is terminating gre tunnel--via eigrp.

Is this what i need to add?

access-list inside-in extended permit gre any any

meameame12 · ‎10-08-2010

i added it--to no avail:

access-list inside-in extended permit gre any any

also, i have:

# show run | in access-group

access-group outside-in in interface outside

access-group extdmz-in in interface extdmz

access-group isadmz-in in interface isadmz

meameame12 · ‎10-08-2010

at one of teh spoke offices:

000624: *Oct 9 00:39:02.471 GMT: ISAKMP:(2365):purging node 904887933

000625: *Oct 9 00:39:09.351 GMT: ISAKMP: DPD received KMI message.

000626: *Oct 9 00:39:09.351 GMT: ISAKMP: set new node 353388376 to QM_IDLE

000627: *Oct 9 00:39:09.351 GMT: crypto_engine: Generate IKE hash

000628: *Oct 9 00:39:09.351 GMT: CryptoEngine0: CRYPTO_ISA_IKE_HMAC(hw)(ipsec)

000629: *Oct 9 00:39:09.355 GMT: ISAKMP:(2768):Sending NOTIFY DPD/R_U_THERE protocol 1

spi 1706225560, message ID = 353388376

000630: *Oct 9 00:39:09.355 GMT: ISAKMP:(2768): seq. no 0x240F5391

000631: *Oct 9 00:39:09.355 GMT: crypto_engine: Encrypt IKE packet

000632: *Oct 9 00:39:09.355 GMT: CryptoEngine0: CRYPTO_ISA_IKE_ENCRYPT(hw)(ipsec)

000633: *Oct 9 00:39:09.355 GMT: ISAKMP:(2768): sending packet to a.b.c.d my_port 500 peer_port 500 (I) QM_IDLE

000634: *Oct 9 00:39:09.355 GMT: ISAKMP:(2768):Sending an IKE IPv4 Packet.

000635: *Oct 9 00:39:09.355 GMT: ISAKMP:(2768):purging node 353388376

000636: *Oct 9 00:39:09.387 GMT: ISAKMP (2768): received packet from a.b.c.d dport 500 sport 500 Global (I) QM_IDLE

000637: *Oct 9 00:39:09.387 GMT: ISAKMP: set new node 1232564680 to QM_IDLE

000638: *Oct 9 00:39:09.387 GMT: crypto_engine: Decrypt IKE packet

000639: *Oct 9 00:39:09.387 GMT: CryptoEngine0: CRYPTO_ISA_IKE_DECRYPT(hw)(ipsec)

000640: *Oct 9 00:39:09.387 GMT: crypto_engine: Generate IKE hash

000641: *Oct 9 00:39:09.387 GMT: CryptoEngine0: CRYPTO_ISA_IKE_HMAC(hw)(ipsec)

000642: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768): processing HASH payload. message ID = 1232564680

000643: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768): processing NOTIFY DPD/R_U_THERE_ACK protocol 1

spi 0, message ID = 1232564680, sa = 66290A80

000644: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768): DPD/R_U_THERE_ACK received from peer a.b.c.d, sequence 0x240F5391

000645: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768):deleting node 1232564680 error FALSE reason "Informational (in) state 1"

000646: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY

000647: *Oct 9 00:39:09.391 GMT: ISAKMP:(2768):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

And:

# show crypto isakmp sa

IPv4 Crypto ISAKMP SA

dst src state conn-id status

NEWTOWN_1841# show crypto isakmp sa

IPv4 Crypto ISAKMP SA

dst src state conn-id status

a.b.c.d y.y.y.y QM_IDLE 2768 ACTIVE

a.b.c.d y.y.y.y QM_IDLE 2365 ACTIVE

AND:

#show crypto ipsec sa | in encaps|decaps|Status|current

current_peer a.v.d.x port 500

#pkts encaps: 48, #pkts encrypt: 48, #pkts digest: 48

#pkts decaps: 47, #pkts decrypt: 47, #pkts verify: 47

current outbound spi: 0xBBF43896(3153344662)

Status: ACTIVE

current_peer a.b.c.d port 500

#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

current outbound spi: 0xB8E117DD(3101759453)

Status: ACTIVE

current_peer y.y.y.y port 500

#pkts encaps: 65, #pkts encrypt: 65, #pkts digest: 65

#pkts decaps: 46, #pkts decrypt: 46, #pkts verify: 46

current outbound spi: 0x46118722(1175553826)

Status: ACTIVE

current_peer z.z.z.z port 500

#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

current outbound spi: 0x167A7CD3(377126099)

Status: ACTIVE

BUT:

Tunnel3 a.b.c.d YES NVRAM up down

Tunnel13 a.b.c.d YES NVRAM up down

And on this router:

crypto isakmp policy 10

encr aes 256

authentication pre-share

group 5

!

crypto isakmp policy 15

encr 3des

authentication pre-share

group 2

c

crypto isakmp keepalive 10

!

crypto isakmp client configuration group jkjljkllkjkl

key ********

dns g.g.g.g.

wins sdfsddf.

pool vpn_pool

acl 108

crypto isakmp profile vpn-clients

match identity group jkjllkklkll

client authentication list vpn

isakmp authorization list vpn

client configuration address respond

!

crypto ipsec transform-set ESP-AES-256_ESP-SHA esp-aes 256 esp-sha-hmac

crypto ipsec transform-set ESP-3DES-SHA_tunnel esp-3des esp-sha-hmac

!

crypto dynamic-map dynmap 10

set transform-set ESP-3DES-SHA_tunnel

set isakmp-profile vpn-clients

reverse-route

!

crypto map cryptomap1 10 ipsec-isakmp

set peer a.a.a.a

set transform-set ESP-AES-256_ESP-SHA

match address h.g.f.d

crypto map cryptomap1 20 ipsec-isakmp

set peer r.r.r.r

set transform-set ESP-AES-256_ESP-SHA

match address x.x.x.x

crypto map cryptomap1 30 ipsec-isakmp

set peer gg.g.g.

set transform-set ESP-AES-256_ESP-SHA

match address b.b.b.b

:

etc .

Jennifer Halim · ‎10-08-2010

Seems like the IPSec tunnel is up, only the GRE tunnel is down.

Try to shut/no shut the GRE tunnel on both ends.

meameame12 · ‎10-09-2010

didnt make a bit of difference...

the problem is on the asa---i just cannot find it...

Jennifer Halim · ‎10-09-2010

From the following output:

current_peer a.b.c.d port 500

#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

current outbound spi: 0xB8E117DD(3101759453)

Status: ACTIVE

current_peer z.z.z.z port 500

#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

current outbound spi: 0x167A7CD3(377126099)

Status: ACTIVE

Assuming that the above is from the spoke router, that means traffic is being encrypted, however, the router does not receive any return traffic.

What does the corresponding show cry ipsec sa shows on the ASA?

You might want to try clearing the IPSEC tunnel SA on the ASA and re-establing the tunnel, and also "clear xlate". Otherwise, feel free to open a TAC case so it can be troubleshot live with an engineer.

meameame12 · ‎10-10-2010

Hi,

Cant open tac case as this device has no contract, and i would have to pay per incident.

i debugged ipsec and isakmp and got a phase 2 error---i had to debug level 200 to even see it.

cant recall the exact error, but i googled it, and basically, under the website, said to check acl--or the access list group is applied wrong.

we did not change a single entry on the remote routers, so it must be a problem on the asa itself--but i cannot find it....

i think its probably the access list group command, but not sure...