Need help with MTU issue (GRE + IPSec VPN tunnel)

Answered Question
Jul 2nd, 2010

Hi,

I have some questions about the best strategy for MSS / MTU definition and PMTUD activation.

What I read:

Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and IPSEC.pdf (I read this one twice )

MTU, TCP MSS, PMTUD MSS Adjust.doc

MTU Tuning for L2TP.pdf

Adjusting IP MTU, TCP MSS, and PMTUD on Windows Systems.pdf

Path MTU Discovery in Windows.mht

Step-by-step instructions for tuning TCP under Windows XP.mht

The idea is clear for me, the first source is a full ABC for the issue, but I still have some questions.

1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?

2. If I have DMVPN (GRE + IPSec) tunnels with my branches and I also have ACL (for incoming traffic) on the head office router WAN interface, which permits isakmp, esp and icmp traffic from the branch router only, and I enable PMTUD together with ip unreachables on that interface, what should I add in the ACL for PMTUD function to work normally? I mean should I add a filter to permit ICMP packets from the next hop? Or should I permit ICMP from any hop?

3. As I understand in case of GRE + IPSec (obviously transport mode) which works over ADSL line we have MSS = 1500 - 24 (GRE) - 38 (IPSec transport mode) - 20 (IP) - 20 (TCP) - 8 (for DSL) = 1390. Is this correct?

4. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec)?

I know I can use "ping -f" but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU (according to my calcs it would be ~ 1438.)

And my main question:

I have DM-VPN (as I understand it means GRE + IPSec in transport mode?) with my local regional branches.

My target is:

- to provide the best performance of the connections between hosts in the head office and each branch

- do not touch hosts settings

- provide invisibility for my routers in public networks.

(AFAIK, the latter means ACLs on our routers WAN interfaces for incoming traffic denying everything except traffic from our another respective router WAN interfaces plus "no ip unreachables". Am I right?)

Now what we have by default:

- in the offices we have Ethernet networks and Windows based PCs, which have PMTUD enabled, MTU = 1500, ICMP unreachables enabled. => All is ready for PMTUD (possibly except personal firewalls, which could block ICMP packets from a router and thus prevent PMTUD from functioning properly)

- on the offices Cisco routers: PMTUD is disabled by default, MTU on physical interfaces = 1500, on GRE tunnel = 1476, ip unreachables on WAN interfaces we are turning off (to provide invisibility).

Now, what is the best strategy?

My version:

for our routers LAN interfaces:

- set "ip tcp adjust-mss 1390"

- enable PMTUD and make sure "ip unreachables" option is activated (AFAIK, it is by default)

- leave MTU by default

for WAN interfaces:

- do nothing (which means PMTUD is disabled, MSS and MTU by default)

for tunnel interfaces:

- do nothing.

Please help me to clarify the question, because I can not handle the information I read, I am feeling giddy...

P.S. Why I am thinking of PMTUD together with MSS adjustment? - Because MSS works for TCP traffic only...

I have this problem too.
0 votes
Correct Answer by Richard Burts about 6 years 4 months ago

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.

PMTDU does not work, windows systems do not react or change the MTU of an interface when a router send an icmp "fragementation needed"

Forget about PMTUD.

What you need to remember is the process - TCP hasnshake, the sender looks at it's MTU and subtracts 20 bytes to IP header, and 20 bytes for the TCP header.  Default MTU is 1500 - so the sender will send his MSS as 1460, the remote end will do the same.

Sadly they will send this with the DF set.  If you are unlucky and the application you are using sends ALL packets with the DF bit set, you have an issue.

Personally when using GRE & Encryption I do the following, work out my overhead then configure MSS Adjust:-

TCP Header - 20 Bytes

IP Header - 20 Bytes (UDP is 8 bytes)

GRE Header - 24 Bytes

So far 64 bytes..

Ethernet Frame - 26 Bytes

IPSEC Header - 56 Bytes

Total 146 Bytes - round it up 150 bytes of overhead BEFORE you think about the data!  Leaves you with 1350 bytes for the data IF the packets has the DF bit set.  So I then configure my mss adjust to 1300 for good measure - everything works fine, includes ADSL WAN as well.

1300 is my golden number.

HTH>

Alen Danielyan Fri, 07/02/2010 - 06:10

[email protected]

PMTDU does not work, windows systems do not react or change the MTU of an interface when a router send an icmp "fragementation needed"

Forget about PMTUD.

Something new for me. As far as I can see it is working. But anyway, if you read my post you will see that I am going to use MSS as the main mean.

What you need to remember is the process - TCP hasnshake, the sender looks at it's MTU and subtracts 20 bytes to IP header, and 20 bytes for the TCP header.  Default MTU is 1500 - so the sender will send his MSS as 1460, the remote end will do the same.

Sadly they will send this with the DF set.  If you are unlucky and the application you are using sends ALL packets with the DF bit set, you have an issue.

Personally when using GRE & Encryption I do the following, work out my overhead then configure MSS Adjust:-

TCP Header - 20 Bytes

IP Header - 20 Bytes (UDP is 8 bytes)

GRE Header - 24 Bytes

So far 64 bytes..

Ethernet Frame - 26 Bytes

IPSEC Header - 56 Bytes

Total 146 Bytes - round it up 150 bytes of overhead BEFORE you think about the data!  Leaves you with 1350 bytes for the data IF the packets has the DF bit set.  So I then configure my mss adjust to 1300 for good measure - everything works fine, includes ADSL WAN as well.

1300 is my golden number.

HTH>

Well, I know it all, and you can see my own calculations. And you did not mention where to set MSS = 1300 (obviously on the LAN interface).

P.S. I think you should not subtract Ethernet frame size, because, AFAIK, MTU=1500 (in fact it is Default IP MTU for Ethernet = Default media MTU -  L2 encapsulation overhead = 1514 - 14) does not include it.

Alen Danielyan Fri, 07/02/2010 - 23:11

Yesterday after long thinking I want to change my solution for tunnel interfaces (I don't think it will make much changes as tunnel PMTUD is enabled by default on tunnel interfaces, but anyway - that was wrong, tunnel PMTUD is also disabled by default):

New version:

for our routers LAN interfaces:

- set "ip tcp adjust-mss 1390"

- enable PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is by default)

- set ip MTU 1430 (this is for non TCP traffic, as MSS influence on TCP only)

for WAN  interfaces:

- do nothing (which means PMTUD is disabled, MSS and  MTU by default)

for tunnel interfaces:

- enable tunnel PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is by default)

- set ip MTU 1430.

I am still hoping someone will answer all of my questions and not just recommend the MTU or MSS values...

Richard Burts Sat, 07/03/2010 - 10:54

Alen

Right now I wish to address the issue of PMTUD. You have enabled it and you observe that it seems to be working. And likely it will work for some (perhaps many) connections. But it will most certainly fail for some. The issue is not at your end but may be at the other end or may be somewhere in the middle. If the network with which you communicate at the remote end has disabled ip unreachables, or if the remote network is filtering and denying outbound ICMP (as some do) or if somewhere on the path in the middle a router is filtering and denying ICMP, then PMTUD for you will not work.

HTH

Rick

Alen Danielyan Mon, 07/05/2010 - 04:18

I understand it, that is why I am also going to decrease MTU and MSS from default settings to our case-specific.

Can you please answer the following questions:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?

2. Do you think it is reasonable to add “permit icmp any any packet-too-big” in my ACLs for incoming traffic on our routers WAN interfaces?

Would you do that?

3. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec), if we do not know anything about the line (equipment, technology, etc.) between?

I know I can use "ping -f" (or mturoute util) but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU?! Or not?

4. I can not enable PMTUD on LAN interface?! I am trying "ip tcp path-mtu-discovery" command in the interface config mode, IOS does not say anything, just exits interface config mode to global config!? Why?

5. How can I define if my GRE + IPSec tunnels work in IPSec tranport, not tunnel mode?

6. Let's suppose we have links which provide "pure tunneling" only (our  branch routers make just tunneling between branches and head  office, nothing else).

Is it enough in that case to lower ip MTU on our routers (branches and head office) tunnel interfaces down to required minimum, or we have to do the same on the routers LAN interfaces? Is the latter reasonable in any case?

The question is especially important because our head office router provides also other connections, and among them is connection to Internet. So I doubt lowering MTU on its LAN interface is a good idea...

Thanks in advance.

Alen Danielyan Mon, 07/12/2010 - 03:44

I'll try to answer some of my own questions. Comments, disproof or proof are very welcomed.

Can you please answer the following questions:



1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?


2. Do you think it is reasonable to add “permit icmp any any packet-too-big” in my ACLs for incoming traffic on our routers WAN interfaces?

Would you do that?


3. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec), if we do not know anything about the line (equipment, technology, etc.) between?

I know I can use "ping -f" (or mturoute util) but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU?! Or not?


4. I can not enable PMTUD on LAN interface?! I am trying "ip tcp path-mtu-discovery" command in the interface config mode, IOS does not say anything, just exits interface config mode to global config!? Why?


5. How can I define if my GRE + IPSec tunnels work in IPSec tranport, not tunnel mode?


6. Let's suppose we have links which provide "pure tunneling" only (our  branch routers make just tunneling between branches and head  office, nothing else).

Is it enough in that case to lower ip MTU on our routers (branches and head office) tunnel interfaces down to required minimum, or we have to do the same on the routers LAN interfaces? Is the latter reasonable in any case?


The question is especially important because our head office router provides also other connections, and among them is connection to Internet. So I doubt lowering MTU on its LAN interface is a good idea...



Thanks in advance.


1. According to the examples in "Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC", /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";} when using GRE+IPSec tunnels we have the following order of the procedures: fragmentation -> encapsulation -> encryption.


I am not sure this is correct information, but I have no other source for the info.



2. In this very case, when we are going to manually set defined path MTU, it is not required to permit "packet-too-big" ICMP packets or send ip unreachables ourselves, but that will be a static solution. Once MTU change (decrease) we will have to change it either...



3. As I understand from the same source, (when  using GRE+IPSec tunnels) if we send packet with DF bit set it should not be fragmented even when is being sent via tunnel (neither while being handled by GRE, nor IPSec) => we can define real maximum path MTU.


I am not sure this info is correct for 100%, especially on the stage of IPSec...



4. PMTUD is globally configured, not in interface config mode.



Still can't answer on questions 5. and 6.

About question 5., I do not see "transport mode" in show run output, does it mean IPSec is in Tunnel mode?

The latter would be superfluous, as we would have double encapsulation...

Alen Danielyan Mon, 07/05/2010 - 03:48

I found one more good source about the problem: "The never-ending story of ip fragmentation", by Ivan Pepelnjak.

http://www.nil.si/ipcorner/IP_Fragmentation/

As I understand the source text, it is enough to activate tunnel PMTUD or just set ip MTU (and MSS) on tunnel interfaces.

Nevertheless, finally I decided to do all together: activate tunnel PMTUD and set ip MTU (and also MSS) on tunnel interfaces.This will provide decreased (down to calculated) MTU and MSS, plus possibility to dynamically decrease it even more if/when needed.

Thus, if I understand things correctly, I'll make my packets to go without fragmentation even if PMTUD would not work on some segment (if we count MTUs were defined correctly).

Yesterday with no MTU settings set (but tunnel PMTUD activated on tunnel interface) I defined that my lines MTU = 1400, so I am setting ip MTU 1400 and MSS 1360 on my tunnel interfaces). Of course I am still not sure I defined it correctly (for automation I used mturoute utility - very useful stuff), because I don't know what is happening in tunnel, specifically: if packet with DF bit set is fragmented when it goes to GRE+IPSec tunnel?

If no, then I got correct result. If yes, then real MTU is even less than I determined, but I don't think this is the case...

So, as all my branch routers work only to provide connectivity with the head office via DM-VPN (GRE + IPSec), I am going to make the same settings on the routers LAN interfaces.

Head office router also provides connection to Internet, but I think it will be ok if I decrease MTU for Internet packets too.

Now, I know this is superfluous, but I decided to use these settings:

for our routers LAN  interfaces:

- set "ip tcp adjust-mss 1360"

- enable  PMTUD and make sure "ip unreachables" option is activated  (AFAIK, it is  by default)

- set ip MTU 1400

for WAN  interfaces:

- do nothing (which  means PMTUD is disabled, MSS and MTU by default), except "no ip unreachables" to provide routers "invisibility" in  public networks

for tunnel  interfaces:

- set "ip tcp adjust-mss 1360"

- enable tunnel PMTUD and make sure "ip unreachables"  option is activated  (AFAIK, it is by default)

- set ip MTU 1400.

Alen Danielyan Mon, 07/05/2010 - 21:49

Friends, I can not get one thing: if we have multiple destinations and some of them have small MTUs, others - high, how can we make connections to work properly in case ICMP unreachables are closed on somewhere on the paths?

I mean for small MTU destinations our end hosts and routers have to be able to be informed that their MTU is small. And vise versa, if we have small MTU ourselves or if we decrease our MTU to conform to small MTU destinations, our end hosts and routers have to be able to inform destinations with high MTU that our MTU is small. How is this possible in case ICMP unreachables can not reach their targets?

As I understand it is just impossible to provide that with any settings done in one end, isn't it?

P.S. In light of above mentioned, I think in case of links which provide "pure tunneling" only (for example our branch routers which make tunneling only between branches and head office) solution exists - just lower MTU on both (tunnel) ends until packets can go without fragmentation.

But in case router have to provide Internet connection we have the issue...

Richard Burts Sat, 07/10/2010 - 06:47

Alen

The question of how to optimize the MTU for various destinations becomes complex, especially for a router providing connectivity to multiple Internet destinations. The best solution for that was PMTUD which would allow a large MTU if it worked and would set a smaller MTU if necessary. And in the early days of the Internet this was an effective solution. But as we have said in our current environment it sometimes does not work.

So we frequently use the solution for set the MTU on the router (perhaps using ip mtu, or adjust mss). When we are setting the MTU we can not set a large one for some destinations and a smaller one for other destinations. So we must set it to a size that will work for the most destinations, which is to set it small. This results in using a smaller MTU to destinations where a larger one could have worked and makes the transmission less efficient. But it is the solution that works.

HTH

Rick

Alen Danielyan Mon, 07/12/2010 - 02:58

Thank you Richard,

I realize that. But my main question is more specific. Here is the fully formulated question:

- we have connections between branches networks and head office network via Cisco routers (DM-VPNs - GRE + IPSec tunnels, all is made on a single device)

- our branch routers tranfer VPN traffic between branches and head office only

- our head office router also provides connection to Internet

- we know the minimum path MTU between branches and head office and it is 1400bytes

- we want to be invisible in public networks (=> no ip unreachables on external interfaces)

- we have ACLs on external interfaces for incoming traffic which permits tunneled traffic from another tunnel end only (permit ISAKMP and ESP from specific ip only)

- we do not want to touch our Windows hosts settings in local networks

- we want to achieve the best perfomance for connections between head office and branches.

What is the best solution in that case?

IMHO, that would be to set up our Cisco routers like this:

for our routers LAN  interfaces:

- set "ip tcp adjust-mss 1360"

- set ip MTU 1400

- enable  PMTUD and make sure "ip  unreachables" option is activated  (AFAIK, it is  by default)

for  WAN  interfaces:

- do nothing (which  means PMTUD is  disabled, MSS and MTU by default), except "no ip unreachables" to  provide routers "invisibility" in  public networks

for tunnel  interfaces:

-  set "ip tcp adjust-mss 1360"

- set ip MTU  1400

- enable  tunnel PMTUD and make sure "ip unreachables"  option is activated   (AFAIK, it is by default).

I choose to set MTU and MSS on tunnel interfaces, because, IMHO, it will provide the same effect for tunnel traffic, but on head office router it will also still allow Internet traffic to use default MTU.

Now. Am I right?

And does setting MTU and MSS on LAN interfaces give us anything more, than setting them on tunnel interfaces  (in this very case)?

Alen Danielyan Sun, 07/18/2010 - 04:01

Please anyone, read my last post and answer the question. I just can not calm down till get an answer.

P.S. And a couple of small questions:

1.  When using GRE+IPSec tunnels in what order are the following procedures  done: fragmentation / encapsulation / encryption?

2. Do you think it  is reasonable to add “permit  icmp any any packet-too-big” in my ACLs for incoming traffic on the  routers WAN interfaces?

3. Does using ping with "-f" option show the real MTU when pinging from one LAN host to another LAN host (LANs are connected via GRE+IPSec tunnel)?

I am not sure GRE or IPSec will copy DF flag from the original packet and it will not be fragmented over the whole path.

4. How can I define if my GRE + IPSec  tunnels  work in IPSec tranport, not tunnel mode?

Please answer.

Mohamed Sobair Sun, 07/18/2010 - 06:18

Hello Alen;

you wrote:

((( Please anyone, read my last post and answer the question. I just can  not calm down till get an answer.

P.S. And a couple of small questions:

1.   When using GRE+IPSec tunnels in what order are the following procedures   done: fragmentation / encapsulation / encryption?

2. Do you think it  is  reasonable to add “permit  icmp any any packet-too-big” in my ACLs for  incoming traffic on the  routers WAN interfaces?

3. Does using ping with "-f" option show the real MTU  when pinging from one LAN host to another LAN host (LANs are connected  via GRE+IPSec tunnel)?

I am not sure GRE or  IPSec will copy DF flag from the original packet and it will not be  fragmented over the whole path.

4. How can I define if  my GRE + IPSec  tunnels  work in IPSec tranport, not tunnel mode? )))

Answer,

Fragmentation is not desired and recommended , fragmentation could lead to performance degradation on a network , it also not required for certain security concern. Fragmentation cause high load on routers as well as if a detination doesnt recieve all the fragmented packet (fragmented packet get lost on the middle), the packet needs to be resended again.

Now, coming to your previous question, the best common methods as follows:

1- The Tunnel interface should be at least set to 1400 , the attribute is not arbitary, its devided as follows:

     -- GRE Header  = 4 Byte

     -- IP Header      =  20 Byte

     -- TCP Header   =  20 Byte

     -- IPsec Header = 56 Byte

Total is 100 Byte substracting it from 1500 , as such the tunnel should be at least set with 1400.

2- The TCP maximum segment size MSS should be adjusted to 40 byte lower than than the 1400 byte of the GRE tunnel and should be set on the LAN segment.

Coming to your 1st question:

1- The packet will first be encrypted and then Encapsulated with a GRE and then fragmented if it need fragmentation, so the order is  Encryption > Encapsulation  > Fragmentation.

2- You dont need this command, if the ICMP packet is too big, icmp packet will be fragmented if the Dont fragment bit is not set on the router. if the router for any reason has DF bit set, an icmp (destination host unreachable) message will be sent to the original host dictating packet needs to be fragmented but DF is set.

3- Yes, it will do the same function and will set the DF bit on the icmp.

4- you can either have IPsec tunnel profile or IPsec Crypto map based on your IPsec configuration.

HTH

Mohamed

Alen Danielyan Mon, 07/19/2010 - 00:37

Hi Mohamed, thank you for your answers.

msobier123 wrote:

Hello Alen;

you wrote:

((( Please anyone, read my last post and answer the question. I just can  not calm down till get an answer.

Answer,

Fragmentation is not desired and recommended , fragmentation could lead to performance degradation on a network , it also not required for certain security concern. Fragmentation cause high load on routers as well as if a detination doesnt recieve all the fragmented packet (fragmented packet get lost on the middle), the packet needs to be resended again.

Now, coming to your previous question, the best common methods as follows:

1- The Tunnel interface should be at least set to 1400 , the attribute is not arbitary, its devided as follows:

     -- GRE Header  = 4 Byte

     -- IP Header      =  20 Byte

     -- TCP Header   =  20 Byte

     -- IPsec Header = 56 Byte

Total is 100 Byte substracting it from 1500 , as such the tunnel should be at least set with 1400.

2- The TCP maximum segment size MSS should be adjusted to 40 byte lower than than the 1400 byte of the GRE tunnel and should be set on the LAN segment.

Thank you, but I read the theory many times and everything is clear (except your minusing TCP header, I don't understand why should we minus TCP  header when calculating MTU?).

The main thing: your settings seem quite reasonable. Thank you.

Do you think activation (in addition to your above mentioned 2 settings ) of PMTUD on both tunnel and LAN interfaces can make things worse?

IMHO, it should not. It will just allow in the future to even decrease the MTU if needed and if PMTUD works correctly, but will not make any problems for current settings even if PMTUD doesn't work correctly. Am I right?

The question is, if I have already activated both: globally configured PMTUD and tunnel PMTUD, should I disable them?

msobier123 wrote:

Coming to your 1st question:

1- The packet will first be encrypted and then Encapsulated with a GRE and then fragmented if it need fragmentation, so the order is  Encryption > Encapsulation  > Fragmentation.


Clear, thank you.

(But in Cisco's Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and  IPSEC.pdf it was an example (scenario 10) where the order was fragmentation / encapsulation /  encryption!? Why?)

msobier123 wrote:

2- You dont need this command, if the ICMP packet is too big, icmp packet will be fragmented if the Dont fragment bit is not set on the router. if the router for any reason has DF bit set, an icmp (destination host unreachable) message will be sent to the original host dictating packet needs to be fragmented but DF is set.

Well, this is clear, but first an intermediary router should send an ICMP packet to my router (as it is the starting point of tunnel and new, incapsulated packet source is his external interface ip). And my router will not "receive" it as it will be blocked by the ACL on ext. interface: block all, except ISAKMP, ESP and ICMP with source ip of another tunnel end router's ext. interface ip.

So I need to add the entry, if I want unreachables about "fragmentation needed" from intermediary routers to be passed and processed. Don't I?

msobier123 wrote:

3- Yes, it will do the same function and will set the DF bit on the icmp.

Thank you. This was very important question, as I was not sure I can get real MTU when testing through the IPSec encrypted GRE tunnel.

No, I know I can find max path MTU just by pinging with -f option from one LAN host - another LAN host!

msobier123 wrote:

4- you can either have IPsec tunnel profile or IPsec Crypto map based on your IPsec configuration.

I see this on my spokes:

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman";}

crypto isakmp policy 1

encr aes 256

authentication pre-share

group 2

crypto isakmp key 6 xxx address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 10 periodic

!

!

crypto ipsec transform-set dmvpnset esp-aes 256 esp-sha-hmac

!

crypto ipsec profile dmvpnprof

set transform-set dmvpnset

And I don't see anything that contains "ipsec tunnel" phrase.

=> As I understand I have IPSec in transport mode, which is "better" in case we are making encapsulation using GRE. Am I correct?

And MTU should be: 1500 - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430?

And this is in case line providers are not using any other encapsulations?

P.S. I'll rate your post later, I am not still satisfied and don't want the question to be marked as answered.

Mohamed Sobair Mon, 07/19/2010 - 03:21

Hi Alen,

Please se my answers in blue:

Thank you, but I read the theory many times and everything is clear (except your minusing TCP header, I don't understand why should we minus TCP  header when calculating MTU?).


TCP based applications will add 20 byte of header to the original Packet size, thats why its subtracted from the total MTU.

The main thing: your settings seem quite reasonable. Thank you.

Do you think activation (in addition to your above mentioned 2 settings ) of PMTUD on both tunnel and LAN interfaces can make things worse?

I personally suggest seeting the MTU manually, I have experienced before for one client using PMTUD wich only decreases the IP header + GRE header on the tunnel. Here you need to decrease TCP header and IPsec header values.

IMHO, it should not. It will just allow in the future to even decrease the MTU if needed and if PMTUD works correctly, but will not make any problems for current settings even if PMTUD doesn't work correctly. Am I right?

Yes , since you have manually adjusted the MTU, you could have PMTUD enabled and it would not have affect unless the adjusted MTU removed.

The question is, if I have already activated both: globally configured PMTUD and tunnel PMTUD, should I disable them?

See above

(But in Cisco's Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and  IPSEC.pdf it was an example (scenario 10) where the order was fragmentation / encapsulation /  encryption!? Why?)

Alen, I was refering to GRE in tunnel Mode, GRE Over IPsec !! Fragmentation will not happen if the packet is not encapsulated and the Encapsulation takes place after Encryption in GRE Over IPsec (Tunnel Mode). I am not sure about your document, try to resend it so we can have alook at it.

Well, this is clear, but first an intermediary router should send an ICMP packet to my router (as it is the starting point of tunnel and new, incapsulated packet source is his external interface ip). And my router will not "receive" it as it will be blocked by the ACL on ext. interface: block all, except ISAKMP, ESP and ICMP with source ip of another tunnel end router's ext. interface ip.

So I need to add the entry, if I want unreachables about "fragmentation needed" from intermediary routers to be passed and processed. Don't I?


Yes, if you have an ACL on ext, you need to permit it. and you need also to make sure (IP unreachable) at the interface level is enabled.

I see this on my spokes:

crypto isakmp policy 1

encr aes 256

authentication pre-share

group 2

crypto isakmp key 6 xxx address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 10 periodic

!

!

crypto ipsec transform-set dmvpnset esp-aes 256 esp-sha-hmac

!

crypto ipsec profile dmvpnprof

set transform-set dmvpnset

And I don't see anything that contains "ipsec tunnel" phrase.

=> As I understand I have IPSec in transport mode, which is "better" in case we are making encapsulation using GRE. Am I correct?

And MTU should be: 1500 - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430?

And this is in case line providers are not using any other encapsulations?

You have two IPsec GRE modes:


1- Tunnel Mode: GRE over IPsec header ( GRE is not encrypted)

2- Tranport Mode: GRE inside IPsec header (GRE is encrypted).


From Security prespective, its better to have GRE inside IPsec which will also encrypt the GRE unlike in Tunnel Mode.


MTU should be as follows:


1- Dilaer Interface : 1492 ( subtracting 8 byte ppp header from the original 1500)

2- GRE Tunnel : 1392 (Subtracting 20 byte Ip header and 20 byte TCP header and 56 byte IPsec header , total is 100 leading 1492 - 100 = 1392)

3- TCP MSS: Maximum segment size should be set to 1352, because of the 40 byte IP and TCP headers. subtracting 1392 - 40 = 1352)

HTH

Mohamed

Alen Danielyan Mon, 07/19/2010 - 05:20

- TCP based applications will  add 20 byte of header to the original Packet size, thats why its  subtracted from the total MTU.

Wait a moment, we are talking about MTU, not MSS. MTU is either IP level - ip MTU, or L2\L1. When we calculate any of the mentioned MTUs we should not minus TCP header!

- Yes , since you have  manually adjusted the MTU, you could have PMTUD enabled and it would not  have affect unless the adjusted MTU removed.

Very well, clear.

- Alen, I was refering to GRE  in tunnel Mode, GRE Over IPsec !! Fragmentation will not happen if the  packet is not encapsulated and the Encapsulation takes place after  Encryption in GRE Over IPsec (Tunnel Mode). I am not sure about your  document, try to resend it so we can have alook at it.

I am also talking about GRE + IPSec. The source is this one (the most famous for MTU issue):

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

See scenario 10.

- Yes, if you have an ACL on  ext, you need to permit it. and you need also to make sure (IP  unreachable) at the interface level is enabled.

I know I'need to add an entry. My question was: "Do you think it is reasonable to add  “permit  icmp any any packet-too-big” in my ACLs for incoming traffic on  the  routers WAN interfaces"?

The matter is, by disabling ip unreachables (together with the above mentioned ACL) on ext. interfaces I make my routers invisible in public networks. As I understand “permit icmp any any packet-too-big” could make them visible. My question is: is it reasonable?

Would you do that?

- You  have two IPsec GRE modes:

1- Tunnel  Mode: GRE over IPsec header ( GRE is not encrypted)

2-  Tranport Mode: GRE inside IPsec header (GRE is encrypted).

From Security prespective,  its better to have GRE inside IPsec which will also encrypt the GRE  unlike in Tunnel Mode.

Sorry, but AFAIK, the main difference is in tunnel mode also initial source and destination addresses are replaced by new ones and are encaplsulated, not only IP data is.

So you can not say Transport mode is more secure. It could be said Tunnel mode is more secure, but when using GRE this is not the case, because GRE is making addresses removal itself and there is no need in double procedure.

Hope I could explain my point.

Addition:

"IPsec can be implemented in a host-to-host transport mode, as well as in a network tunnel mode.


Transport mode
In transport mode, only the payload (the data you transfer) of the IP packet is encrypted and/or authenticated. The routing is intact, since the IP header is neither modified nor encrypted; however, when the authentication header is used, the IP addresses cannot be translated, as this will invalidate the hash value. The transport and application layers are always secured by hash, so they cannot be modified in any way (for example by translating the port numbers). Transport mode is used for host-to-host communications.

A means to encapsulate IPsec messages for NAT traversal has been defined by RFC documents describing the NAT-T mechanism.

Tunnel mode

In tunnel mode, the entire packet (which need not be IP) is encrypted and/or authenticated. It is then encapsulated into a new IP packet with a new IP header. Tunnel mode is used to create Virtual Private Networks for network-to-network communications (e.g. between routers to link sites), host-to-network communications (e.g. remote user access), and host-to-host communications (e.g. private chat).

Tunnel mode supports NAT traversal."

MTU  should be as follows:

1- Dilaer  Interface : 1492 ( subtracting 8 byte ppp header from the original 1500)

2- GRE  Tunnel : 1392 (Subtracting 20 byte Ip header and 20 byte TCP header and  56 byte IPsec header , total is 100 leading 1492 - 100 = 1392)

3- TCP  MSS: Maximum segment size should be set to 1352, because of the 40 byte  IP and TCP headers. subtracting 1392 - 40 = 1352)

Again, I can not agree.

For path ip MTU it has to be 1500 (default ip MTU for Ethernet technology networks) - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430.

And MSS in that case = 1430 - 20 (TCP header) - 20 (IP header).

Thank you for the time you spent on this topic.!

Alen Danielyan Wed, 07/21/2010 - 22:12

Mohamed, please don't leave the topic.

P.S. A new issue: after I enabled PMTUD on my routers NTP failed to work! (My spoke routers synch from regional center routers via tunnels)

Yesterday I spent half of the day to solve the problem, until finally supposed it could be connected with PMTUD. Today I turned off PMTUD (globally configured one) and NTP is working again now.

So I'll keep it disabled.

rodmoore Mon, 07/19/2010 - 21:10

What if someone were to take an opposite approach?  Lower the MTU on the crypto map interface (1300) and raise the mtu on the tunnel interface (1500). We will all agree that fragmentation will occur after GRE encapsulation and reassembly will occur on the other side the IPSEC tunnel.  So, some delay is introduced and more router CPU is used.  Is this a bad thing?  It can be if: you don't have CPU to spare or if the frag/reassembly delay affects the application (e.g. voice or video - but you can't adjust the MSS on these anyway as they use UDP).

What are the benefits?  This works for all traffic, not just TCP.  It's simple to configure, understand and maintain. The network becomes transparent to the end systems - no more wondering is this the application or the network when an app breaks.  It's the app.

I've read the same paper and why it doesn't explore this option, I have no idea.  Worse case, it sells bigger routers with more memory...

Mohamed Sobair Thu, 07/22/2010 - 04:15

Alen,

The Global Path MTUD adjust tcp based and NTP uses UDP port 123, However, as I noted earlier, I strongly suggest you to manually set the IP MTU to 1400 on the tunnel interface. there are lots of application wont work when a packet gets fragmented or it will cause high delay as well.

HTH

Mohamed

Alen Danielyan Thu, 07/22/2010 - 05:14

While I still hope you will look at and comment my comments, I have new question:

As I remember even if I only set ip MTU on tunnel interfaces everything works fine. (I am talking about branch 2 head VPN connections)

Are there any reason to add ip MTU or MSS also on the routers LAN interfaces? What can it give else?

The negative impact would be all traffic (not only tunneled) going through the routers would have packets with small MTU. What else positive or negative we get?

P.S. About my NTP issue, here is the situation: I have branch routers LAN interfaces as ntp sources, LAN interface of the head office as ntp server and I had ip MTU 1400 on LAN interfaces on all of my routers. Then after I activated global PMTUD, I removed ip MTU 1400 from the head's router LAN interface (as Internet is also connected to the router and I did not want to reduce "Internet" packets).

And after some time (that is the main sad thing) ntp synch stopped working. After I finally understand what could be the reason and disabled PMTUD on the head router everything start working again.

But later I also disabled PMTUD on all branch routers either, just in case...

Alen Danielyan Sun, 07/25/2010 - 05:41

A new question: is sending "ip unreachable" (fragmentation needed but DF bit set) from the intermediary router whose MTU is less than the one of the recieved packet, a part of PMTUD mechanism or it is a standard behaviour regardless of PMTUD activation status?

Correct Answer
Richard Burts Sun, 07/25/2010 - 14:21

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

Alen Danielyan Tue, 07/27/2010 - 06:27

rburts wrote:

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

Thank you, Richard.

Can you please also answer my last but one question?

As I remember even when I only set ip MTU on tunnel interfaces everything works fine. (I am talking about branch to head - hub to spoke VPN connections). Are there any reason to add ip MTU or MSS also on the routers LAN interfaces?

As I understand, the negative impact would be all traffic (not only tunneled) going through the routers would have packets with small MTU. What else positive or negative we get?

Thank you.

Actions

This Discussion