Site-to-Site VPN drops certain UDP packets

Unanswered Question
Aug 30th, 2007
User Badges:

We have a site to site VPN set up using the PIX 506e. The IKE and IPSec tunnel set up and work great; TCP and ICMP traffic flows through them wonderfully! UDP traffic, however, is a different story.


On the "far" end, we have software that sends a UDP packet to a "near" end server. The server receives it and sends a UDP reply (I know this by using tcpdump). However, that UDP reply never seems to make it back to the far end. This is my problem.


I can respond with various configuration information as needed.


The PIX on the far/client end logs the outgoing UDP packet:

Aug 28 17:16:38 021 %PIX-0-302015: Built outbound UDP connection 5508355 for outside:192.168.0.3/5093 (192.168.0.3/5093) to inside:192.168.4.5/4533 (192.168.4.5/4533)

The PIX on the near/server end logs that UDP packet:

Aug 28 17:16:38 %PIX-0-302015: Built inbound UDP connection 408810 for outside:192.168.4.5/4533 (192.168.4.5/4533) to inside:192.168.0.3/5093 (192.168.0.3/5093)


The near/server PIX never seems to build an outbound connection -- is that because it uses the original incoming one? Maybe I'm barking up the wrong tree there.


Anyway, they both eventually tear down the connection after the timeout:

Far/client PIX:

Aug 28 17:19:27 021 %PIX-0-302016: Teardown UDP connection 5508355 for outside:192.168.0.3/5093 to inside:192.168.4.5/4533 duration 0:02:49 bytes 8640

Near/server PIX:

Aug 28 17:19:28 %PIX-0-302016: Teardown UDP connection 408810 for outside:192.168.4.5/4533 to inside:192.168.0.3/5093 duration 0:02:49 bytes 17424


I notice that the near/server PIX registers almost double the bytes than the far/client PIX, which could account for the replies. They never seem to reach the far/client side though.


Any thoughts would be appreciated. Thanks.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (2 ratings)
Loading.
acomiskey Thu, 08/30/2007 - 10:05
User Badges:
  • Green, 3000 points or more

"The near/server PIX never seems to build an outbound connection -- is that because it uses the original incoming one?"


-Have you attempted to capture the traffic to see if the server ever sends the reply?

willie.gillespie Thu, 08/30/2007 - 10:11
User Badges:

Yes, I have. The server does send the reply, addressed to the client IP address.

Jon Marshall Thu, 08/30/2007 - 10:17
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Okay, assuming you are not blocking it with an access-list on the inside interface of your server side pix can you check that your crypto map on the server side would see this as interesting traffic.


It may be an idea to post a sanitised version of the config so we can hopefully help you.


Jon

Jon Marshall Thu, 08/30/2007 - 10:10
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Hi


Can you ping the same server from the same client ?.


Have you tried debugging packets on the inside interface at the server side pix eg.


debug packet inside src 192.168.0.3


to see if the packets are indeed coming back from the server to your server side pix ?


Jon

willie.gillespie Thu, 08/30/2007 - 10:26
User Badges:

Yes, I can ping the same server from the same client.


I hadn't tried debugging packets. Here is the output showing the packets come back from the server to the server side PIX:

--------- PACKET ---------


-- IP --

IFServ01 ==> 192.168.4.5


ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x5b4

id = 0x6635 flags = 0x0 frag off=0x0

ttl = 0x80 proto=0x11 chksum = 0x49ab


-- UDP --

source port = 0x13e5 dest port = 0x4fb

len = 0x5a0 checksum = 0xc2a9

-- DATA --

00000010: 00 00 00 01 | ....

00000020: 00 00 00 00 09 b5 0b 4c 5a 5b 50 57 65 5c 4d 7b | .......LZ[PWe\M{

00000030: 67 3e 41 45 64 67 5f 29 2a 51 57 52 67 5c 40 25 | g>AEdg_)*QWRg\@%

00000040: 4c 40 31 3a 40 3b 40 31 4c 40 34 47 54 4a 31 24 | [email protected]:@;@[email protected]$

00000050: 4c 40 43 37 44 3a 30 7b 36 2e 34 34 45 3b 40 29 | [email protected]:0{6.44E;@)

00000060: 4c 40 37 37 54 4a 40 25 3c 40 43 47 45 4a 40 31 | [email protected]@%<@[email protected]

00000070: 4c 31 43 47 54 3b 40 24 4c 40 34 47 54 4a 31 7b | L1CGT;@[email protected]


One interesting thing I notice as I look at the debug output is that the length field doesn't match the actual amount of data shown and there is no "END OF PACKET" line. Could this make a difference?

willie.gillespie Thu, 08/30/2007 - 12:46
User Badges:

Just a note:

I looked into the debug packet command a little more and just because it doesn't show the --END OF PACKET-- line doesn't mean that it doesn't go through to the other side.


So, the reply packet is getting to the server side PIX. Any clues on why it doesn't show up on the far/client side PIX? Would either PIX be dropping it for some reason?

willie.gillespie Thu, 08/30/2007 - 11:21
User Badges:

Just for fun, I tried the "debug packet inside src 192.168.0.3" on the far/client side. I saw TCP and ICMP packets going to the client, but not the one on UDP 5093.


I tried "debug packet outside src 192.168.0.3" (again on the far/client side) but it was silent.


Does that seem right?


Unrelated: With IPSec, are the remote IPs considered on the inside or the outside interface after traveling through the tunnel?

Jon Marshall Thu, 08/30/2007 - 11:28
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Willie


Yes it would be silent on the outside because the packets are encrypted within a VPN tunnel so you wouldn't see anything as the src IP address would be the peer address of the pix.


Remote IP's are considered on the inside after travelling through tunnel.


Jon

willie.gillespie Thu, 08/30/2007 - 11:42
User Badges:

I wonder if my problem lies in this area then. I mean, the site to site tunnel seems to work (mostly), but if my IPs should be considered on the inside...


Does that mean that this would be incorrect?

access-list outside_cryptomap_20 line 1 permit ip 192.168.0.0 255.255.255.0 192.168.4.0 255.255.255.0

I thought this would say that traffic from 192.168.0.x to 192.168.4.x would be considered interesting traffic. However, does that mean that it thinks 192.168.4.x is on the outside interface?

Jon Marshall Thu, 08/30/2007 - 11:51
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Willie



No you have your crypto map access-list right, i think we just confused each other with terminology.


The crypto map access tells the device that to send traffic from 192.168.0.0 255.255.255.0 192.168.4.0 255.255.255.0 it must go through a VPN tunnel.


Hope this clarifies it.


Jon

willie.gillespie Thu, 08/30/2007 - 21:01
User Badges:

Update:

I downloaded a UDP packet send/receive program for some diagnostics.


I can always seem to send UDP packets from the far/client end to the near/server end... so that's not the issue.


I can send UDP packets from the near/server to the far/client -IF- the UDP data packet is 1433 bytes or less in size.


Notice that for the same packet, the size of the total packet (tlen) changes on the near end compared to the far end (by 9 bytes). I haven't figured that one out yet.


Anyway, so that's my problem. How do I get it so I can send larger packets from the server side to the client side? (Remember that these 'larger'* packets go fine from the client to the server.)


On the near/server side:

--------- PACKET ---------

-- IP --

IFServ01 ==> 192.168.4.5

ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x5ad

id = 0x1745 flags = 0x0 frag off=0x0

ttl = 0x80 proto=0x11 chksum = 0x98a2

-- UDP --

source port = 0x3039 dest port = 0x3039

len = 0x599 checksum = 0x36a



And on the far/client side:

--------- PACKET ---------

-- IP --

192.168.0.3 ==> Server

ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x5a4

id = 0x1745 flags = 0x20 frag off=0x0

ttl = 0x80 proto=0x11 chksum = 0x78ab

-- UDP --

source port = 0x3039 dest port = 0x3039

len = 0x599 checksum = 0x36a



* larger = 1460 bytes total.

Since it shows up in the near/server side packet trace, I almost wonder if it's the far side that is just dropping them.


Anyway, any thoughts or suggestions are welcome at this point!

willie.gillespie Fri, 08/31/2007 - 08:13
User Badges:

Got it!


Solution: on the near/server side PIX, the outside interface had an MTU of 9000. Every other interface in the picture had an MTU of 1500. So, the server side PIX was sending this large packet to the client side, which it just dropped.


Matching MTUs across the board (to 1500) worked perfectly.

Jon Marshall Fri, 08/31/2007 - 09:06
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Willie


Great news. Thanks for getting back and letting us know and sorry couldn't be more help.


Jon

willie.gillespie Fri, 08/31/2007 - 09:22
User Badges:

Jon,


No worries. Talk about an obscure problem!

Thanks for all your help and willingness. I was surprised at the quickness of the response.


Thanks again.


Willie

Actions

This Discussion