cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
31028
Views
5
Helpful
25
Replies

Need help with MTU issue (GRE + IPSec VPN tunnel)

Alen Danielyan
Level 1
Level 1

Hi,

I have some questions about the best strategy for MSS / MTU definition and PMTUD activation.

What I read:

Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and IPSEC.pdf (I read this one twice )

MTU, TCP MSS, PMTUD MSS Adjust.doc

MTU Tuning for L2TP.pdf

Adjusting IP MTU, TCP MSS, and PMTUD on Windows Systems.pdf

Path MTU Discovery in Windows.mht

Step-by-step instructions for tuning TCP under Windows XP.mht

The idea is clear for me, the first source is a full ABC for the issue, but I still have some questions.

1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?

2. If I have DMVPN (GRE + IPSec) tunnels with my branches and I also have ACL (for incoming traffic) on the head office router WAN interface, which permits isakmp, esp and icmp traffic from the branch router only, and I enable PMTUD together with ip unreachables on that interface, what should I add in the ACL for PMTUD function to work normally? I mean should I add a filter to permit ICMP packets from the next hop? Or should I permit ICMP from any hop?

3. As I understand in case of GRE + IPSec (obviously transport mode) which works over ADSL line we have MSS = 1500 - 24 (GRE) - 38 (IPSec transport mode) - 20 (IP) - 20 (TCP) - 8 (for DSL) = 1390. Is this correct?

4. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec)?

I know I can use "ping -f" but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU (according to my calcs it would be ~ 1438.)

And my main question:

I have DM-VPN (as I understand it means GRE + IPSec in transport mode?) with my local regional branches.

My target is:

- to provide the best performance of the connections between hosts in the head office and each branch

- do not touch hosts settings

- provide invisibility for my routers in public networks.

(AFAIK, the latter means ACLs on our routers WAN interfaces for incoming traffic denying everything except traffic from our another respective router WAN interfaces plus "no ip unreachables". Am I right?)

Now what we have by default:

- in the offices we have Ethernet networks and Windows based PCs, which have PMTUD enabled, MTU = 1500, ICMP unreachables enabled. => All is ready for PMTUD (possibly except personal firewalls, which could block ICMP packets from a router and thus prevent PMTUD from functioning properly)

- on the offices Cisco routers: PMTUD is disabled by default, MTU on physical interfaces = 1500, on GRE tunnel = 1476, ip unreachables on WAN interfaces we are turning off (to provide invisibility).

Now, what is the best strategy?

My version:

for our routers LAN interfaces:

- set "ip tcp adjust-mss 1390"

- enable PMTUD and make sure "ip unreachables" option is activated (AFAIK, it is by default)

- leave MTU by default

for WAN interfaces:

- do nothing (which means PMTUD is disabled, MSS and MTU by default)

for tunnel interfaces:

- do nothing.

Please help me to clarify the question, because I can not handle the information I read, I am feeling giddy...

P.S. Why I am thinking of PMTUD together with MSS adjustment? - Because MSS works for TCP traffic only...

1 Accepted Solution

Accepted Solutions

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

HTH

Rick

View solution in original post

25 Replies 25

andrew.prince
Level 10
Level 10

PMTDU does not work, windows systems do not react or change the MTU of an interface when a router send an icmp "fragementation needed"

Forget about PMTUD.

What you need to remember is the process - TCP hasnshake, the sender looks at it's MTU and subtracts 20 bytes to IP header, and 20 bytes for the TCP header.  Default MTU is 1500 - so the sender will send his MSS as 1460, the remote end will do the same.

Sadly they will send this with the DF set.  If you are unlucky and the application you are using sends ALL packets with the DF bit set, you have an issue.

Personally when using GRE & Encryption I do the following, work out my overhead then configure MSS Adjust:-

TCP Header - 20 Bytes

IP Header - 20 Bytes (UDP is 8 bytes)

GRE Header - 24 Bytes

So far 64 bytes..

Ethernet Frame - 26 Bytes

IPSEC Header - 56 Bytes

Total 146 Bytes - round it up 150 bytes of overhead BEFORE you think about the data!  Leaves you with 1350 bytes for the data IF the packets has the DF bit set.  So I then configure my mss adjust to 1300 for good measure - everything works fine, includes ADSL WAN as well.

1300 is my golden number.

HTH>

andrew.prince@monster.com

PMTDU does not work, windows systems do not react or change the MTU of an interface when a router send an icmp "fragementation needed"

Forget about PMTUD.

Something new for me. As far as I can see it is working. But anyway, if you read my post you will see that I am going to use MSS as the main mean.

What you need to remember is the process - TCP hasnshake, the sender looks at it's MTU and subtracts 20 bytes to IP header, and 20 bytes for the TCP header.  Default MTU is 1500 - so the sender will send his MSS as 1460, the remote end will do the same.

Sadly they will send this with the DF set.  If you are unlucky and the application you are using sends ALL packets with the DF bit set, you have an issue.

Personally when using GRE & Encryption I do the following, work out my overhead then configure MSS Adjust:-

TCP Header - 20 Bytes

IP Header - 20 Bytes (UDP is 8 bytes)

GRE Header - 24 Bytes

So far 64 bytes..

Ethernet Frame - 26 Bytes

IPSEC Header - 56 Bytes

Total 146 Bytes - round it up 150 bytes of overhead BEFORE you think about the data!  Leaves you with 1350 bytes for the data IF the packets has the DF bit set.  So I then configure my mss adjust to 1300 for good measure - everything works fine, includes ADSL WAN as well.

1300 is my golden number.

HTH>

Well, I know it all, and you can see my own calculations. And you did not mention where to set MSS = 1300 (obviously on the LAN interface).

P.S. I think you should not subtract Ethernet frame size, because, AFAIK, MTU=1500 (in fact it is Default IP MTU for Ethernet = Default media MTU -  L2 encapsulation overhead = 1514 - 14) does not include it.

In the implementations i have done and have seen (mostly in ipsec scenarios) the value of the mss adjust used is 1360.

ISometimes it helps to clear the df-bit also (its dangerous because of cpu use).

This link give the basic ideias for this type of implementations http://www.cisco.com/en/US/tech/tk827/tk369/technologies_tech_note09186a0080093f1f.shtml

Alen Danielyan
Level 1
Level 1

Yesterday after long thinking I want to change my solution for tunnel interfaces (I don't think it will make much changes as tunnel PMTUD is enabled by default on tunnel interfaces, but anyway - that was wrong, tunnel PMTUD is also disabled by default):

New version:

for our routers LAN interfaces:

- set "ip tcp adjust-mss 1390"

- enable PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is by default)

- set ip MTU 1430 (this is for non TCP traffic, as MSS influence on TCP only)

for WAN  interfaces:

- do nothing (which means PMTUD is disabled, MSS and  MTU by default)

for tunnel interfaces:

- enable tunnel PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is by default)

- set ip MTU 1430.

I am still hoping someone will answer all of my questions and not just recommend the MTU or MSS values...

Alen

Right now I wish to address the issue of PMTUD. You have enabled it and you observe that it seems to be working. And likely it will work for some (perhaps many) connections. But it will most certainly fail for some. The issue is not at your end but may be at the other end or may be somewhere in the middle. If the network with which you communicate at the remote end has disabled ip unreachables, or if the remote network is filtering and denying outbound ICMP (as some do) or if somewhere on the path in the middle a router is filtering and denying ICMP, then PMTUD for you will not work.

HTH

Rick

HTH

Rick

I understand it, that is why I am also going to decrease MTU and MSS from default settings to our case-specific.

Can you please answer the following questions:

1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?

2. Do you think it is reasonable to add “permit icmp any any packet-too-big” in my ACLs for incoming traffic on our routers WAN interfaces?

Would you do that?

3. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec), if we do not know anything about the line (equipment, technology, etc.) between?

I know I can use "ping -f" (or mturoute util) but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU?! Or not?

4. I can not enable PMTUD on LAN interface?! I am trying "ip tcp path-mtu-discovery" command in the interface config mode, IOS does not say anything, just exits interface config mode to global config!? Why?

5. How can I define if my GRE + IPSec tunnels work in IPSec tranport, not tunnel mode?

6. Let's suppose we have links which provide "pure tunneling" only (our  branch routers make just tunneling between branches and head  office, nothing else).

Is it enough in that case to lower ip MTU on our routers (branches and head office) tunnel interfaces down to required minimum, or we have to do the same on the routers LAN interfaces? Is the latter reasonable in any case?

The question is especially important because our head office router provides also other connections, and among them is connection to Internet. So I doubt lowering MTU on its LAN interface is a good idea...

Thanks in advance.

I'll try to answer some of my own questions. Comments, disproof or proof are very welcomed.

Can you please answer the following questions:



1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?


2. Do you think it is reasonable to add “permit icmp any any packet-too-big” in my ACLs for incoming traffic on our routers WAN interfaces?

Would you do that?


3. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec), if we do not know anything about the line (equipment, technology, etc.) between?

I know I can use "ping -f" (or mturoute util) but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU?! Or not?


4. I can not enable PMTUD on LAN interface?! I am trying "ip tcp path-mtu-discovery" command in the interface config mode, IOS does not say anything, just exits interface config mode to global config!? Why?


5. How can I define if my GRE + IPSec tunnels work in IPSec tranport, not tunnel mode?


6. Let's suppose we have links which provide "pure tunneling" only (our  branch routers make just tunneling between branches and head  office, nothing else).

Is it enough in that case to lower ip MTU on our routers (branches and head office) tunnel interfaces down to required minimum, or we have to do the same on the routers LAN interfaces? Is the latter reasonable in any case?


The question is especially important because our head office router provides also other connections, and among them is connection to Internet. So I doubt lowering MTU on its LAN interface is a good idea...



Thanks in advance.


1. According to the examples in "Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC", when using GRE+IPSec tunnels we have the following order of the procedures: fragmentation -> encapsulation -> encryption.


I am not sure this is correct information, but I have no other source for the info.



2. In this very case, when we are going to manually set defined path MTU, it is not required to permit "packet-too-big" ICMP packets or send ip unreachables ourselves, but that will be a static solution. Once MTU change (decrease) we will have to change it either...



3. As I understand from the same source, (when  using GRE+IPSec tunnels) if we send packet with DF bit set it should not be fragmented even when is being sent via tunnel (neither while being handled by GRE, nor IPSec) => we can define real maximum path MTU.


I am not sure this info is correct for 100%, especially on the stage of IPSec...



4. PMTUD is globally configured, not in interface config mode.



Still can't answer on questions 5. and 6.

About question 5., I do not see "transport mode" in show run output, does it mean IPSec is in Tunnel mode?

The latter would be superfluous, as we would have double encapsulation...

Alen Danielyan
Level 1
Level 1

I found one more good source about the problem: "The never-ending story of ip fragmentation", by Ivan Pepelnjak.

http://www.nil.si/ipcorner/IP_Fragmentation/

As I understand the source text, it is enough to activate tunnel PMTUD or just set ip MTU (and MSS) on tunnel interfaces.

Nevertheless, finally I decided to do all together: activate tunnel PMTUD and set ip MTU (and also MSS) on tunnel interfaces.This will provide decreased (down to calculated) MTU and MSS, plus possibility to dynamically decrease it even more if/when needed.

Thus, if I understand things correctly, I'll make my packets to go without fragmentation even if PMTUD would not work on some segment (if we count MTUs were defined correctly).

Yesterday with no MTU settings set (but tunnel PMTUD activated on tunnel interface) I defined that my lines MTU = 1400, so I am setting ip MTU 1400 and MSS 1360 on my tunnel interfaces). Of course I am still not sure I defined it correctly (for automation I used mturoute utility - very useful stuff), because I don't know what is happening in tunnel, specifically: if packet with DF bit set is fragmented when it goes to GRE+IPSec tunnel?

If no, then I got correct result. If yes, then real MTU is even less than I determined, but I don't think this is the case...

So, as all my branch routers work only to provide connectivity with the head office via DM-VPN (GRE + IPSec), I am going to make the same settings on the routers LAN interfaces.

Head office router also provides connection to Internet, but I think it will be ok if I decrease MTU for Internet packets too.

Now, I know this is superfluous, but I decided to use these settings:

for our routers LAN  interfaces:

- set "ip tcp adjust-mss 1360"

- enable  PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is  by default)

- set ip MTU 1400

for WAN  interfaces:

- do nothing (which  means PMTUD is disabled, MSS and MTU by default), except "no ip unreachables" to provide routers "invisibility" in  public networks

for tunnel  interfaces:

- set "ip tcp adjust-mss 1360"

- enable tunnel PMTUD and make sure "ip unreachables"  option is activated  (AFAIK, it is by default)

- set ip MTU 1400.

Friends, I can not get one thing: if we have multiple destinations and some of them have small MTUs, others - high, how can we make connections to work properly in case ICMP unreachables are closed on somewhere on the paths?

I mean for small MTU destinations our end hosts and routers have to be able to be informed that their MTU is small. And vise versa, if we have small MTU ourselves or if we decrease our MTU to conform to small MTU destinations, our end hosts and routers have to be able to inform destinations with high MTU that our MTU is small. How is this possible in case ICMP unreachables can not reach their targets?

As I understand it is just impossible to provide that with any settings done in one end, isn't it?

P.S. In light of above mentioned, I think in case of links which provide "pure tunneling" only (for example our branch routers which make tunneling only between branches and head office) solution exists - just lower MTU on both (tunnel) ends until packets can go without fragmentation.

But in case router have to provide Internet connection we have the issue...

Please anyone answer my questions.

Alen

The question of how to optimize the MTU for various destinations becomes complex, especially for a router providing connectivity to multiple Internet destinations. The best solution for that was PMTUD which would allow a large MTU if it worked and would set a smaller MTU if necessary. And in the early days of the Internet this was an effective solution. But as we have said in our current environment it sometimes does not work.

So we frequently use the solution for set the MTU on the router (perhaps using ip mtu, or adjust mss). When we are setting the MTU we can not set a large one for some destinations and a smaller one for other destinations. So we must set it to a size that will work for the most destinations, which is to set it small. This results in using a smaller MTU to destinations where a larger one could have worked and makes the transmission less efficient. But it is the solution that works.

HTH

Rick

HTH

Rick

Thank you Richard,

I realize that. But my main question is more specific. Here is the fully formulated question:

- we have connections between branches networks and head office network via Cisco routers (DM-VPNs - GRE + IPSec tunnels, all is made on a single device)

- our branch routers tranfer VPN traffic between branches and head office only

- our head office router also provides connection to Internet

- we know the minimum path MTU between branches and head office and it is 1400bytes

- we want to be invisible in public networks (=> no ip unreachables on external interfaces)

- we have ACLs on external interfaces for incoming traffic which permits tunneled traffic from another tunnel end only (permit ISAKMP and ESP from specific ip only)

- we do not want to touch our Windows hosts settings in local networks

- we want to achieve the best perfomance for connections between head office and branches.

What is the best solution in that case?

IMHO, that would be to set up our Cisco routers like this:

for our routers LAN  interfaces:

- set "ip tcp adjust-mss 1360"

- set ip MTU 1400

- enable  PMTUD and make sure "ip  unreachables" option is activated  (AFAIK, it is  by default)

for  WAN  interfaces:

- do nothing (which  means PMTUD is  disabled, MSS and MTU by default), except "no ip unreachables" to  provide routers "invisibility" in  public networks

for tunnel  interfaces:

-  set "ip tcp adjust-mss 1360"

- set ip MTU  1400

- enable  tunnel PMTUD and make sure "ip unreachables"  option is activated   (AFAIK, it is by default).

I choose to set MTU and MSS on tunnel interfaces, because, IMHO, it will provide the same effect for tunnel traffic, but on head office router it will also still allow Internet traffic to use default MTU.

Now. Am I right?

And does setting MTU and MSS on LAN interfaces give us anything more, than setting them on tunnel interfaces  (in this very case)?

Please anyone, read my last post and answer the question. I just can not calm down till get an answer.

P.S. And a couple of small questions:

1.  When using GRE+IPSec tunnels in what order are the following procedures  done: fragmentation / encapsulation / encryption?

2. Do you think it  is reasonable to add “permit  icmp any any packet-too-big” in my ACLs for incoming traffic on the  routers WAN interfaces?

3. Does using ping with "-f" option show the real MTU when pinging from one LAN host to another LAN host (LANs are connected via GRE+IPSec tunnel)?

I am not sure GRE or IPSec will copy DF flag from the original packet and it will not be fragmented over the whole path.

4. How can I define if my GRE + IPSec  tunnels  work in IPSec tranport, not tunnel mode?

Please answer.

Mohamed Sobair
Level 7
Level 7

Hello Alen;

you wrote:

((( Please anyone, read my last post and answer the question. I just can  not calm down till get an answer.

P.S. And a couple of small questions:

1.   When using GRE+IPSec tunnels in what order are the following procedures   done: fragmentation / encapsulation / encryption?

2. Do you think it  is  reasonable to add “permit  icmp any any packet-too-big” in my ACLs for  incoming traffic on the  routers WAN interfaces?

3. Does using ping with "-f" option show the real MTU  when pinging from one LAN host to another LAN host (LANs are connected  via GRE+IPSec tunnel)?

I am not sure GRE or  IPSec will copy DF flag from the original packet and it will not be  fragmented over the whole path.

4. How can I define if  my GRE + IPSec  tunnels  work in IPSec tranport, not tunnel mode? )))

Answer,

Fragmentation is not desired and recommended , fragmentation could lead to performance degradation on a network , it also not required for certain security concern. Fragmentation cause high load on routers as well as if a detination doesnt recieve all the fragmented packet (fragmented packet get lost on the middle), the packet needs to be resended again.

Now, coming to your previous question, the best common methods as follows:

1- The Tunnel interface should be at least set to 1400 , the attribute is not arbitary, its devided as follows:

     -- GRE Header  = 4 Byte

     -- IP Header      =  20 Byte

     -- TCP Header   =  20 Byte

     -- IPsec Header = 56 Byte

Total is 100 Byte substracting it from 1500 , as such the tunnel should be at least set with 1400.

2- The TCP maximum segment size MSS should be adjusted to 40 byte lower than than the 1400 byte of the GRE tunnel and should be set on the LAN segment.

Coming to your 1st question:

1- The packet will first be encrypted and then Encapsulated with a GRE and then fragmented if it need fragmentation, so the order is  Encryption > Encapsulation  > Fragmentation.

2- You dont need this command, if the ICMP packet is too big, icmp packet will be fragmented if the Dont fragment bit is not set on the router. if the router for any reason has DF bit set, an icmp (destination host unreachable) message will be sent to the original host dictating packet needs to be fragmented but DF is set.

3- Yes, it will do the same function and will set the DF bit on the icmp.

4- you can either have IPsec tunnel profile or IPsec Crypto map based on your IPsec configuration.

HTH

Mohamed

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco