Weird MTU issue

Unanswered Question
Aug 29th, 2009

We're having a somewhat strange issue with MTU. We have a 50Mbps ethernet handoff from our provider (which, apparently, comes off of a fiber ring) which terminates directly on our ASA 5510. Speed and duplex are set to 100/full on both sides. The MTU for the outside interface is set to 1500 and our provider reports that the MTU on their router is set to 2450. From a consumer-class DSL connection at my home, I show the path MTU to the provider's router as 1500, which is expected. However, the path MTU to my ASA drops to 1020, which, theoretically, is only one hop back from the ISP's router. From inside the LAN behind the ASA, the path MTU to any site on the Internet is 1020, but the path MTU to anywhere on my LAN is 1500. We have noticed some latency to Internet sites which makes me wonder about packet fragmentation. Also, we have been testing a video conference unit that streams HD video using UDP packets and we are seeing a lot of loss. I'm wondering if the MTU change is part of the equation with the streaming as well. Any thoughts or places to check would be very much appreciated. Thanks!

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Peter Paluch Sat, 08/29/2009 - 07:12

Hello Trey,

This is an interesting issue. How do you measure the path MTU? Is it perhaps a traceroute or a similar command? My idea is that the path MTU discovery is based on ICMP messages sent by a router that needs you to decrease the packet size. Personally, the first thing I would look after is catching these ICMP messages and having a look what is the sender IP. Most probably, that machine would be the best place to start because, after all, it is precisely that machine that forces the MTU to be smaller.

Best regards,

Peter

trey.thompson Mon, 08/31/2009 - 05:53

Thanks for the reply. I used a small windows utility called mturoute.exe which, as I understood it, used ICMP packets of various sizes to try and determine the path MTU. I have not done a capture. Would I capture the packets on the machine that's sending the ICMP packets? Thanks!

Peter Paluch Mon, 08/31/2009 - 06:11

Hi Trey,

Nice utility :) I haven't had the pleasure of knowing it until now (being an ardent Linux user, the Windows just pass by me :)

Yes, sniff the packets on your computer. I suggest using Wireshark for that purpose. Then, look for all ICMP packet that are sent back to you and watch for 'Fragmentation needed' ICMP messages.

Best regards,

Peter

milan.kulik Mon, 08/31/2009 - 06:39

Hi,

sometimes 'Fragmentation needed' ICMP messages are not passing the network (dropped by FWs or some L3 devices not supporting the feature).

So the only way working any time is setting DF-bit in your Pings to the destination and watching the replies.

Example:

Ping x.x.x.x -l 1472 -f

(in Windows world, corresponds to "Ping x.x.x.x size 1500 df-bit" in Cisco world).

When no reply comes back, decrease the packet size

Ping x.x.x.x -l 1200 -f

If a reply comes back, try to increase again.

In several steps, you can find the maximum packet size passing the network without fragmentation.

(Which, I believe, is exactly what mturoute does?)

BR,

Milan

Paolo Bevilacqua Mon, 08/31/2009 - 06:53

I don't know mturoute but in software it's easy to combine mtu probing and traceroute so that it can tell you on which link over a path the MTU gets reduced.

Peter Paluch Mon, 08/31/2009 - 07:34

Hi Milan,

Yes, you are right. But I assume that if Trey already got that software running and it did produce some information for him then the ICMP messages were probably permitted. But, yes, that command could have also tried to interpolate that value. We'll see what Trey has to say.

Best regards,

Peter

milan.kulik Mon, 08/31/2009 - 23:01

Hi Peter,

yes, I agree.

It would be interesting to see Trey's output of mturoute -t ...

in both directions (from his LAN to the Internet and from the Internet to his LAN).

I think I'll test mturoute.exe in my network "when I have some time free".

One thing I don't understand:

Looking to that tool author's web pages

(http://www.elifulkerson.com/projects/mturoute.php), there's an example:

"C:\mturoute-src\bin>mturoute -t www.slashdot.org

mturoute to www.slashdot.org, 30 hops max, variable sized packets

* ICMP Fragmentation is not permitted. *

* Maximum payload is 10000 bytes. *

1 --+---+++-+++-++ host: 192.168.2.1 max: 1500 bytes

2 --+---+++-+++-++ host: 10.90.160.1 max: 1500 bytes

3 --+---+++-+++-++ host: 68.87.56.165 max: 1500 bytes

4 --+---+++-+++-++ host: 68.87.56.213 max: 1500 bytes

5 --+---+++-+++-++ host: 68.87.56.209 max: 1500 bytes

6 --+---+++-+++-++ host: 68.87.56.201 max: 1500 bytes

7 --+---+-+++++++ host: 68.87.56.197 max: 1472 bytes

8 --+---+++-+++-++ host: 68.87.56.249 max: 1500 bytes

"

If router 7 permitted only 1472 Byte packet length, how could the tool recognize router 8 permitting 1500 again?

BR,

Milan

Peter Paluch Mon, 08/31/2009 - 23:36

Hello Milan,

That has left me wondering as well until I had a better look at the author's page which states:

Due to the nature of the test, it is sensitive to packets dropped for non-MTU related reasons. Packets that are lost for other reasons will result in an incorrect MTU reading.

Apparently, the mturoute utility does not use the elicited ICMP Fragmentation Needed responses as the normal Path MTU discovery does, but simply reacts to all lost packets as being "too big" for that hop and uses a heuristic approach to determine the packet size.

This really lefts me wondering now if the mturoute is the good utility for this job. Trey, do you have a recent Linux handy with the traceroute or tracepath commands? They are based on real ICMP messaging to find out the MTU along a path.

Best regards,

Peter

trey.thompson Tue, 09/01/2009 - 02:15

I have access to a linux box. Is there a particular set of traceroute switches I should use to get the best output? I'll provide the output and let you all take a look. Many thanks for looking into this for me!

Peter Paluch Tue, 09/01/2009 - 02:45

Hello Trey,

The traceroute v2.0.12 has a --mtu switch that tries to find out the path MTU. Try running it as:

traceroute -n --mtu DESTINATION

or

traceroute -I -n --mtu DESTINATION

The -I uses ICMP packets for traceroute operation (as opposed to using UDP), the -n is for numerical output (no IP to DNS name translation).

I am very interested in seeing what it produces.

Best regards,

Peter

trey.thompson Tue, 09/01/2009 - 05:40

Here are the results from outside our network. The first traceroute is to the default route from the firewall (the first hop on the ISPs network). The second is to the outside interface of our firewall.

[email protected]:~$ sudo traceroute -I -n --mtu 64.158.200.141

traceroute to 64.158.200.141 (64.158.200.141), 30 hops max, 65000 byte packets

1 69.69.224.1 57.646 ms F=1500 55.600 ms 55.678 ms

2 69.34.35.89 55.597 ms 55.616 ms 64.525 ms

3 4.79.18.209 66.992 ms 66.715 ms 66.981 ms

4 4.68.17.194 67.433 ms 70.662 ms 66.955 ms

5 209.247.11.89 69.391 ms 69.428 ms 69.431 ms

6 64.158.200.141 73.535 ms 73.750 ms 73.630 ms

[email protected]:~$ sudo traceroute -I -n --mtu 64.158.200.142

traceroute to 64.158.200.142 (64.158.200.142), 30 hops max, 65000 byte packets

1 69.69.224.1 57.578 ms F=1500 55.671 ms 55.343 ms

2 69.34.35.93 55.872 ms 55.152 ms 55.731 ms

3 4.59.146.41 67.395 ms 67.194 ms 67.267 ms

4 4.68.17.2 67.980 ms 67.670 ms 67.534 ms

5 209.247.11.89 69.929 ms 69.435 ms 69.481 ms

6 209.244.22.2 69.676 ms 69.697 ms 69.408 ms

7 * * *

8 * * *

9 * * *

10 * * *

11 * * *

12 * * *

13 * * *

14 * * *

15 * * *

16 * * *

17 * * *

18 * * *

19 * * *

20 * * *

21 * * *

22 * * *

23 * * *

24 * * *

25 * * *

26 * * *

27 * * *

28 * * *

29 * * *

30 * * *

Peter Paluch Tue, 09/01/2009 - 07:09

Hello Trey,

This is interesting. First of all, the MTU is, according to your output, 1500 bytes and does not change further. As for the traceroute to 64.158.200.142, ASA firewalls often to not respond to ICMP so that might be a little problem here but nevertheless, the Linux traceroute did not confirm the MTU issue.

Where did you do these tests from? Were you inside that network which allegedly had the MTU issues?

Best regards,

Peter

f-cagica Mon, 07/25/2011 - 01:56

Hi Trey,

i know this is a 2 year old issue, but perhaps you still remember... I am facing a similar problem at a Customer: 2x ASA5510 8.0(5) connected through a VPN tunnel  (outside-outside), drops packets greater than 1020byte (incl. ping between the outside interfaces). Could you tell me how did you solve your problem? IOS upgrade to which version, workaround, etc..

Thank you in advance

Fernando

lgijssel Mon, 07/25/2011 - 04:29

Old or not, for anyone who stumbles over a similar issue: The ASA will block all incoming icmp traffic by default.

This was mentioned in the thread but probably the poster has not taken sufficient notice of this.

If this is omitted, you will never see any icmp messages from sources on the Internet.

So you need to explicitly allow icmp from outside. I am using an acl like this:

access-list icmp-out-in extended permit icmp any any echo-reply
access-list icmp-out-in extended permit icmp any any time-exceeded
access-list icmp-out-in extended permit icmp any any unreachable

access-group icmp-out-in in interface outside

regards,

Leo

f-cagica Mon, 07/25/2011 - 04:34

Leo,

icmp is already allowed in my case... I can ping the outside interface with packet size till 1020byte. Abobe 1020byte packet size it doesn't answer anymore.

Thanks

Fernando

lgijssel Mon, 07/25/2011 - 13:39

Hi Fernando,

This is presumably an Internet connection?

Are both sides of the VPN connected to the same ISP? If yes, then please contact them.

Otherwise, try to establish the path using tools like ping and tracrt, and determine where the traffic is dropped.

This is most likely not on the ASA's. The resolution method is already described in this thread.

Be sure to test properly and verify your results via several methods.

You will not be the first one to be tricked by a tool which is misinterpreted or simply provides incorrect information.

regards,

Leo

f-cagica Mon, 07/25/2011 - 15:23

Hi Leo,

i guess my description was not accurate... It is a connection between two site susing a service provider. The connection is done at layer 2 (the SP uses some Optical P2P link). When i ping the neighbor Firewall with packets bigger than 1020bytes they are not answered. I also thought it could be some problem in the Optical link (as i don't know was lays there in the middle), but then i cheched the Interface counters on the receiving FW, and noticed that the Drop counters increase with my long Pings... so either the optical equipment introduces some error on packets bigger than 1020bytes or the interface doesn't like the size. What do you think? When i saw this tread, i started blaming the FW.

Br,

Fernando

Actions

This Discussion