Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Weird NAT issue on IPSec Tunnel - 8.4(2)

Helloo All,

This has to be the most weirdest issue I have seen since the past year on my ASA.

I have an ASA 5540 running the 8.4(2) code without any issues until I stumbled upon this problem last week and I have spent sleepless nights with no resolution! So, take a deep breath and here is a brief description of my setup and the problem:

A Simple IPSEC tunnel between my ASA 5540 8.4(2) and a Juniper SSG 140 screen OS 6.3.0r9.0(route based VPN)

The tunnel comes up without any issues but the ASA refuses to encrypt the traffic but decrypts it with GLORY!

below are some debug outputs, show outputs and a packet tracer output which also has an explanation of my WEIRD NAT issue:

my setup - ( I wont get into the tunnel encryption details as my tunnel negotiations are perfect and comes up right off the bat when the ASA is configured as answer only)

CISCO ASA - IPSec networking details

LOCAL NETWORK - 10.2.4.0/28

REMOTE NETWORK - 192.168.171.8/32

JUNIPER SSG 140 - IPSec networking details

PROXY ID:

LOCAL NETWORK - 192.168.171.8/32

REMOTE NETWORK - 10.2.4.0/28

HOSTNAME# sh cry ipsec sa peer <JUNIPER SSG PEER>

peer address: <JUNIPER SSG PEER>

    Crypto map tag: outside_map, seq num: 5, local addr: <local peer>

      access-list outside_cryptomap_4 extended permit ip 10.2.4.0 255.255.255.240 host 192.168.171.8

      local ident (addr/mask/prot/port): (10.2.4.0/255.255.255.240/0/0)

      remote ident (addr/mask/prot/port): (192.168.171.8/255.255.255.255/0/0)

      current_peer: <JUNIPER SSG PEER>

      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

      #pkts decaps: 72, #pkts decrypt: 72, #pkts verify: 72

      #pkts compressed: 0, #pkts decompressed: 0

      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0

      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

      #send errors: 0, #recv errors: 0

      local crypto endpt.: <local peer>/0, remote crypto endpt.: <juniper remote peer>/0

      path mtu 1500, ipsec overhead 58, media mtu 1500

      current outbound spi: 5041C19F

      current inbound spi : 0EC13558

    inbound esp sas:

      spi: 0x0EC13558 (247543128)

         transform: esp-3des esp-sha-hmac no compression

         in use settings ={L2L, Tunnel, }

         slot: 0, conn_id: 22040576, crypto-map: outside_map

         sa timing: remaining key lifetime (sec): 3232

         IV size: 8 bytes

         replay detection support: Y

         Anti replay bitmap:

          0xFFFFFFFF 0xFFFFFFFF

    outbound esp sas:

      spi: 0x5041C19F (1346486687)

         transform: esp-3des esp-sha-hmac no compression

         in use settings ={L2L, Tunnel, }

         slot: 0, conn_id: 22040576, crypto-map: outside_map

         sa timing: remaining key lifetime (sec): 3232

         IV size: 8 bytes

         replay detection support: Y

         Anti replay bitmap:

          0x00000000 0x00000001

VPN CONTEXTS for this IPSEC tunnel:

HOSTNAME# sh asp table vpn-context det

VPN CTX  = 0x0742E6BC

Peer IP  = 192.168.171.8

Pointer  = 0x78C94BF8

State    = UP

Flags    = ENCR+ESP

SA       = 0x9C28B633

SPI      = 0x5041C19D

Group    = 0

Pkts     = 0

Bad Pkts = 0

Bad SPI  = 0

Spoof    = 0

Bad Crypto = 0

Rekey Pkt  = 0

Rekey Call = 0

VPN Filter = <none>

VPN CTX  = 0x07430D3C

Peer IP  = 192.168.1.8

Pointer  = 0x78F62018

State    = UP

Flags    = DECR+ESP

SA       = 0x9C286E3D

SPI      = 0x9B6910C5

Group    = 1

Pkts     = 297

Bad Pkts = 0

Bad SPI  = 0

Spoof    = 0

Bad Crypto = 0

Rekey Pkt  = 0

Rekey Call = 0

VPN Filter = <none>

access-list outside_cryptomap_4 extended permit ip 10.2.4.0 255.255.255.240 host 192.168.171.8

nat (inside,outside) source static Ren-vers Ren-vers destination static Peer-Host Peer-Host no-proxy-arp route-lookup

object network Ren-vers

subnet 10.2.4.0 255.255.255.240

object network Peer-Host

host 192.168.171.8

sh cry ipsec sa

   IKE Peer: <remote peer>

    Type    : L2L             Role    : responder

    Rekey   : no              State   : MM_ACTIVE

packet tracer output extract for a packet being sent from the 10.2.4.0/28 network to 192.168.171.8 host

Phase: 7

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0x7789d788, priority=70, domain=encrypt, deny=false

        hits=2, user_data=0x742e6bc, cs_id=0x7ba38680, reverse, flags=0x0, protocol=0

        src ip/id=10.2.4.0, mask=255.255.255.240, port=0

        dst ip/id=192.168.171.8, mask=255.255.255.255, port=0, dscp=0x0

        input_ifc=any, output_ifc=outside

THE VPN contexts match for the encrytpion+encapsulation and the hits here increment only when I run a packet tracer test from my inside host to the remote peer.

A Full packet tracer output for a packet from the 10.2.4.1 255.255.255.255 network to host 192.168.171.8:

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in  id=0x77ebd1b0, priority=1, domain=permit, deny=false

        hits=3037156, user_data=0x0, cs_id=0x0, l3_type=0x8

        src mac=0000.0000.0000, mask=0000.0000.0000

        dst mac=0000.0000.0000, mask=0100.0000.0000

        input_ifc=inside, output_ifc=any

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   192.168.171.0     255.255.255.0   outside

Phase: 3

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0x77ec1030, priority=0, domain=inspect-ip-options, deny=true

        hits=212950, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=inside, output_ifc=any

Phase: 4

Type:

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in  id=0x7c12cb18, priority=18, domain=flow-export, deny=false

        hits=172188, user_data=0x78b1f438, cs_id=0x0, use_real_addr, flags=0x0,

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=inside, output_ifc=any

Phase: 5

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (inside,outside) source static Ren-vers Ren-vers destination static Peer-Host Peer-Host no-proxy-arp route-lookup

Additional Information:

Static translate 10.2.4.1/2700 to 10.2.4.1/2700

Forward Flow based lookup yields rule:

in  id=0x77e0a878, priority=6, domain=nat, deny=false

        hits=9, user_data=0x7b7360a8, cs_id=0x0, use_real_addr, flags=0x0, proto

        src ip/id=10.2.4.1, mask=255.255.255.240, port=0

        dst ip/id=192.168.171.8, mask=255.255.255.255, port=0, dscp=0x0

        input_ifc=inside, output_ifc=outside


(this is the weird  NAT issue I am seeing. I see the hits count is incrementing only when I run the packet tracer even thugh I have constant pings(traffic) from the 192.168.171.8 host to the 10.2.4.1/28) - please see the packet capture that i have pasted after this section)

Phase: 6

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0x7b8751f8, priority=70, domain=encrypt, deny=false

        hits=3, user_data=0x7432b74, cs_id=0x7ba38680, reverse, flags=0x0, proto

        src ip/id=10.2.4.1, mask=255.255.255.240, port=0

        dst ip/id=192.168.171.8, mask=255.255.255.255, port=0, dscp=0x0

        input_ifc=any, output_ifc=outside

Phase: 7

Type: VPN

Subtype: ipsec-tunnel-flow

Result: ALLOW

Config:

Additional Information:

Reverse Flow based lookup yields rule:

in  id=0x78b0c280, priority=69, domain=ipsec-tunnel-flow, deny=false

        hits=154, user_data=0x7435f94, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=192.168.171.8, mask=255.255.255.255, port=0

        dst ip/id=10.2.4.1, mask=255.255.255.240, port=0, dscp=0x0

        input_ifc=outside, output_ifc=any

Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Reverse Flow based lookup yields rule:

in  id=0x77e7a510, priority=0, domain=inspect-ip-options, deny=true

        hits=184556, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

        src ip/id=0.0.0.0, mask=0.0.0.0, port=0

        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

        input_ifc=outside, output_ifc=any

Phase: 9

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 119880921, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_tcp_normalizer

snp_fp_translate

snp_fp_adjacency

snp_fp_encrypt

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_ipsec_tunnel_flow

snp_fp_translate

snp_fp_tcp_normalizer

snp_fp_adjacency

snp_fp_fragment

snp_ifc_stat

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: allow

HOSTNAME# sh cap cap1

8 packets captured

   1: 12:26:53.376033 192.168.10.252 > 10.2.4.1: icmp: echo request

   2: 12:26:53.376597 10.2.4.1 > 192.168.10.252: icmp: echo reply

   3: 12:26:56.487905 192.168.171.8 > 10.2.4.1: icmp: echo request

   4: 12:27:01.489217 192.168.171.8 > 10.2.4.1: icmp: echo request

   5: 12:27:03.378245 192.168.10.252 > 10.2.4.1: icmp: echo request

   6: 12:27:03.378825 10.2.4.1 > 192.168.10.252: icmp: echo reply

   7: 12:27:06.491597 192.168.171.8 > 10.2.4.1: icmp: echo request

   8: 12:27:11.491856 192.168.171.8 > 10.2.4.1: icmp: echo request

8 packets shown

As you can see, there is no echo reply packet at all as the packet is not being encapsulated while it is being sent back.

I have been going madddd with this. Also, this is a live production multitenant firewall with no issues at all apart from this ipsec tunnel to a juniper!!

Also, the 192.168.10.0/24 is another IPSec tunnel remote network to this 10.2.4.0/28 network and this IPSEC tunnel has a similar Juniper SSG 140 screen os 6.3.0r9.0 at the remote end and this woks like a charm without any issues, but the 171 is not being encrypted by the ASA at all.

If anybody could help me out, that would be greatt and greatly appreciated!!

Thanks heaps.!

Everyone's tags (4)
1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Weird NAT issue on IPSec Tunnel - 8.4(2)

Perfect!! Now you have to find something else indoors to do tomorrow --> weather forecasting rain again

Please kindly mark the post as answered so others can learn from it. Thanks.

24 REPLIES
Cisco Employee

Weird NAT issue on IPSec Tunnel - 8.4(2)

Can you try to change the following command:

FROM:

nat (inside,outside) source static Ren-vers Ren-vers destination static Peer-Host Peer-Host no-proxy-arp route-lookup

TO:

nat (inside,outside) source static Ren-vers Ren-vers destination static Peer-Host Peer-Host

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Hi Jen,

didn't work..:(

HOSTNAME# sh cap cap1

16 packets captured

   1: 16:30:28.888900 192.168.171.8 > 10.2.4.1: icmp: echo request

   2: 16:30:33.890731 192.168.171.8 > 10.2.4.1: icmp: echo request

   3: 16:30:38.891799 192.168.171.8 > 10.2.4.1: icmp: echo request

   4: 16:30:43.893004 192.168.171.8 > 10.2.4.1: icmp: echo request

No replies from the peer at all. I still see the hit count on the NAT rule incrementing only when i run a packet tracer from inside to outside

It replies to the icmp from the other peer(192.168.10.0/24) network though!

PS: have tried reverse route injection as well. doesn't nat/encapsulate the traffic!!

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

OK, how is your routing looking like internally?

If it is responding to 192.168.10.0/24 and not to 192.168.171.0/24 then there might be routing issue whereby it does not know how to route back towards 192.168.171.0/24 subnet, ie: needs to be back towards the ASA.

From the packet capture, it is not even replying back, meaning, the return traffic is not even coming back to the ASA.

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

well, honestly I do not have anything in my route table on the ASA for the 192.168.10.0/24 network.

My, packet tracer for the network 192.168.171.0/24 network is perfectly finee! i can see it hitting the vpn module for encryption.

Let me add this.. I have another VPN tunnel which has the same remote peer network but a different local host subnet.

would that matter? is it the bug which is haunting me on the 8.4(2) code as well?

PS: I just saw the below on the route outside table..

S    192.168.171.0 255.255.255.0 [1/0] via , outside

S*   0.0.0.0 0.0.0.0 [1/0] via , outside

the

S    192.168.171.0 255.255.255.0 [1/0] via , outside shouldn't be there?!??!

but the route doesn't show up in my sh run nor my asdm..

weird!!!!

A reload is it?

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Where is the 10.2.4.0/24 connected? directly connected to the ASA interface, or further down in your network?

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

directly connected..I am actually using VLSM. my inside is a /8 and i have used VLSM for further splitting(no VLANing)

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Also, when I try and delete the route it says:

HOSTNAME(config)# no route outside 192.168.171.0 255.255.255.0 1

%No matching route to delete

but it shows up when i do a sh route outside.

WERID..

edit: this is the route outside. it's not even hitting my outside yet(VPN traffic is all inside traffic and i do not see any capture back from my 10.2.4.1 host to 192.168.171.8) - Just thinking out loud. what else could it be? I wonder. my packet tracer outputs are perfect! can I assume that the packet tracer output in this instance is misleading? i reckon packet tracer is very powerful and very accurate, but it aint in this case!!! I am not able to sleep! become a zombieeeeee.. Jen or any1 else from TAC..please helpp!!!


also, I just realised that my management network is 192.168.171.0/24 so that route is pretty much valid and not to be deleted!

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

HUH? your management network is 192.168.171.0/24???? That is the problem.... You can't have 192.168.171.0/24 exist in both remote end and your management network.

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Pls share your complete config on the ASA to double check. Thanks.

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Hi Jen,

when I mean management network, I mean to say that the mangement interface on the ASA is 192.168.171.0/24

Also, the management NIC(which has this network) has the interface disabled. so for me to "manage" the ASA, I use the inside interface!

let me show you the management interface confis on my ASA:

interface Management0/0

shutdown

nameif management

security-level 100

ip address 192.168.171.88 255.255.255.0

Note that the interface is shutdown.

Also, I have another IPSec tunnel from a different local network(another VLSM range) to the same peer network address(192.168.171.0/24) network and that works perfectly fine.

below is the sh route outside:

Gateway of last resort is to network 0.0.0.0

C    is directly connected, outside

S    192.168.171.0 255.255.255.0 [1/0] via outside  - where did that route come from, I am still wondering. must be a management interface route, but I do not know how to remove it. I cant see it in the sh run and when I do a no route outside for this address, it says not present.

S*   0.0.0.0 0.0.0.0 [1/0] via , outside

I also tried to apply a VPN filter which allowed traffic both ways:

access-list re--acl extended permit ip host 10.2.4.1 host 192.168.171.8

access-list re-acl extended permit ip host 192.168.171.8  host 10.2.4.1

even that didn't work.

Any other ideas please. I have shared all the relevant configs of the asa for this tunnel. Please dis-regard the management network as it is disabled and I can also remove the network if you want me to. A bounce of the ASA is the last resort!

Please help!

Thanks.

edit: i just changed the hosts network , still no joy!!!

I have seen many people haviing similar issues with the tunnel traffic between an asa and a juniper. is it just the way the juniper is handling the packets? let me add this as well... if i make the asa an originate only for this ipsec tunnel, the tunnel negotiations fail miserably on the juniper as the asa sends the proxy id's to the juniper as the public ip's of the tunnel end points whereas the juniper is expecting the remote and local private ip's as the proxy id's. is there anwyay i can make the asa send the proxy id to this peer using the local and remote private ip's instead of the public ip's? i reckon this is hardcoded in the IOS. is there anyway to override it Jen?

  I am just thinking out loud on the forum. Hope I donot get a BAN or anything of that sort

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

also, i have bought down the tunnel numerous times and renegotiated after i made changes.. that didn help.

the other thing that amuses me is that i have initiated a continuos ping from the 10.2.4.1 inside host to the remote host 192.168.171.8 host and I do not see the ASA capturing the echo request at all!!!

it is a win 2008 vm and there are no special persistent routes on the same. i have another ping from the same inside host 10.2.4.1 to another tunnel remote network 192.168.10.14 and the asa catches this packet in the capture that i do..

below is the proof:

  38: 19:32:28.648999 10.2.4.1 > 192.168.10.14: icmp: echo request

  39: 19:32:28.655163 192.168.10.14 > 10.2.4.1: icmp: echo reply

  40: 19:32:29.654690 10.2.4.1 > 192.168.10.14: icmp: echo request

  41: 19:32:29.661220 192.168.10.14 > 10.2.4.1: icmp: echo reply

  42: 19:32:30.653729 10.2.4.1 > 192.168.10.14: icmp: echo request

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

after some more analysis and carefully looking through the sh run, i have the below on the asa as well

ip verify reverse-path interface outside

can the above statement actually cause a problem in this case here as my route outside has that weird 192.168.171.0/24 route which I cannot simply find in my sh run!!!!!! how else coduld i remove that route? i have disabled my management interface and removed the IP from it as well.

>.<

can anyone tell me how to remove a route outside entry from my routing table which is not present in my sh run??

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

okay, so i removed that non relevant route outside statement after adding a reverse route injection statement to my ipsec tunnel group and removing it.

so right now my route outside is perfect and the route lookup is normal, but still i have the same issues

HELP M E P L E A S E..

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

@Jennifer and @CISCO TAC

can i override the ASA's proxy id's to the remote peer tunnel end point?

at the moment the ASA 8.4(2) IOS uses the public ip of the peer end points as the proxy id during tunnel negotiations.. is there anyway i can force it to use specific private ip's(local and remote network addresses) for this specific tunnel group only.??

Thanks.

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

no you can't..

proxy id should be the peer public IP

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Okay, thanks.

Looks like I have hit a road block then!

I am unable to answer this question though.. I have another IPSec tunnel with the same configs on the ASA and the juniper ssg 140 and that works perfectly fine, but just this. I was under the impression that I knew the IPSec tunneling on ASA's down pat, reckon I was mistaken. The ASA is making me sweat...

Just can't seem to think anything else.

Any other ideas Jen??

Thanks.

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

this has to be a bitter pill to swallow:

after a frantic 5 min phone call, i bounced my ASA a while back and guess what!

IT

STILL

DOESN'T

WORK

maybe it is just the way the juniper ssg is the initiator, the ASA being the asnwer only is not suitable for this IPSEC tunnel. even though it works on the other juniper ssg 140, it just doesn't like it in this instance.

I shall resort to route this traffic from another ipsec hop!

For now, this question remains unanswered just like the rest of the others out there on the internet who had the same issue!

PS: I am still open to try something else out if someone has any other suggesstions.

Thanks.

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Do you mind sharing the full show run output?

Maybe we can spot something..

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Hi Jen,

I shall send you a PM with my extracted sh run.

let me know if that is okay. Thanks.

Cisco Employee

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

Just re-reading all the post again, and here is my conclusion:

1) Packet tracer shows that it is going through just fine, meaning that it is not a configuration error on the ASA.

2) Packet capture shows that there is no echo reply, meaning that the return traffic is not even hitting the ASA, hence nothing will be encrypted because there is nothing arriving on the ASA.

3) Management network is on 192.168.171.0/24, meaning there is a possibility (I don't know how your mgmt network is connected), the return traffic is going to your management network, instead of back towards the ASA inside interface.

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

thanks a lot Jen!

The problem is related to my L3 switch.. something must have gone wrong with the control plane on the active one and it is not routing right and it hasn't failed over yet..

i changed the default gateway on my inside peer to point to the inside interface of the ASA and it works with full glory. oh sweet baby jesuss.. feels soo good..i knew i had the asa down pat! thanks a lot for your time Jen!

A good way to spend a gloomy weekend indoors!

Cheers

Cisco Employee

Weird NAT issue on IPSec Tunnel - 8.4(2)

Perfect!! Now you have to find something else indoors to do tomorrow --> weather forecasting rain again

Please kindly mark the post as answered so others can learn from it. Thanks.

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

haha..being from mel, kinda immune to the wet cold weather!

bring on more asa realted issues.so much funn to play around with packets and thinkk.. i so love security and these boards!!

New Member

Re: Weird NAT issue on IPSec Tunnel - 8.4(2)

sorted all of this out now.

my L3 switch was buggy, made it failover and it was all sorted.

Also, as a quick heads up, for an IPSec tunnel between an ASA and a Juniper, it is best to have the ASA in an answer only mode and use proxy id's on the Juniper with VPN monitoring.

2342
Views
0
Helpful
24
Replies
CreatePlease to create content