Solved: Question regarding IPSec VTI

XIE YAO · ‎08-01-2013

Hi guys,

I've seen strange behavior in IPSec vti, I want to seek your advice on this.

It seems that even without tunnel mode ipsec ipv4, the tunnel can still working and traffic seems to be encrypting as well, see below setup.

R1 connects to R2, and R2 connects to R3, there is a tunnel interface between R1 and R3.

R1 (F0:8.8.12.1)----(F0:8.8.12.2) R2 (F0:8.8.23.2)----(F0:8.8.23.3) R3

(T0:192.168.13.1) --------------------------------------(T0:192.168.13.3)

Scenario one, non-standard vti configuration, note that I didn't configure "tunnel mode ipsec ipv4"

R1

interface Tunnel0

ip address 192.168.13.1 255.255.255.0

tunnel source 8.8.12.1

tunnel destination 8.8.23.3

tunnel protection ipsec profile IP

R2

interface Tunnel0

ip address 192.168.13.3 255.255.255.0

tunnel source 8.8.23.3

tunnel destination 8.8.12.1

tunnel protection ipsec profile IP

ipsec configuration is pretty simple and same on both side:

crypto isakmp policy 1

authentication pre-share

crypto isakmp key cisco address 0.0.0.0 0.0.0.0

no crypto isakmp ccm

crypto ipsec transform-set IP esp-3des esp-md5-hmac

crypto ipsec profile IP

set transform-set IP

R1 can ping to R2 tunnel ip with maxium 1476 size, and packet seems to be encrypted as well.

At first glance, I thought the tunnel is actually a GRE tunnel, but I can see the IPsec SA is up, and traffic is being encrypting and decrypting

R1#ping 192.168.13.3 size 1476 df

Type escape sequence to abort.

Sending 5, 1476-byte ICMP Echos to 192.168.13.3, timeout is 2 seconds:

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 76/156/260 ms

R1#ping 192.168.13.3 size 1477 df

Type escape sequence to abort.

Sending 5, 1477-byte ICMP Echos to 192.168.13.3, timeout is 2 seconds:

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

R1#sh cry eng conn ac

ID Interface IP-Address State Algorithm Encrypt Decrypt

1 FastEthernet0/0 8.8.12.1 set HMAC_SHA+DES_56_CB 0 0

2001 FastEthernet0/0 8.8.12.1 set 3DES+MD5 10 0

2002 FastEthernet0/0 8.8.12.1 set 3DES+MD5 0 10

R1#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 192.168.13.1/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1476 bytes

R1#sh int tun0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 192.168.13.1/24

MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 8.8.12.1, destination 8.8.23.3

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Tunnel protection via IPSec (profile "IP")

Scenario two, standard vti configuration, the result is as below:

R1

interface Tunnel0

ip address 192.168.13.1 255.255.255.0

tunnel source 8.8.12.1

tunnel destination 8.8.23.3

tunnel mode ipsec ipv4

tunnel protection ipsec profile IP

R2:

interface Tunnel0

ip address 192.168.13.3 255.255.255.0

tunnel source 8.8.23.3

tunnel destination 8.8.12.1

tunnel mode ipsec ipv4

tunnel protection ipsec profile IP

R1(config)#do ping 192.168.13.3 size 1443 df

Type escape sequence to abort.

Sending 5, 1443-byte ICMP Echos to 192.168.13.3, timeout is 2 seconds:

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 84/156/296 ms

R1(config)#do ping 192.168.13.3 size 1444 df

Type escape sequence to abort.

Sending 5, 1444-byte ICMP Echos to 192.168.13.3, timeout is 2 seconds:

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

R1#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 192.168.13.1/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1443 bytes

R1#sh int t0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 192.168.13.1/24

MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 8.8.12.1, destination 8.8.23.3

Tunnel protocol/transport IPSEC/IP

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Tunnel protection via IPSec (profile "IP")

Last input 01:16:12, output 01:36:24, output hang never

Last clearing of "show interface" counters never

So, standard VTI gives the tunnel 1443 bytes MTU, then, my question is:

1. In the first scenario, why tunnel can encrypt traffic even it's showing "Tunnel protocol/transport GRE/IP"

2. Why tunnel provide higher throughput in first scenario?

Any ideas is appreciated.

Thanks

XIE

Peter Paluch · ‎08-01-2013

Hi Xie,

1. In the first scenario, why tunnel can encrypt traffic even it's showing "Tunnel protocol/transport GRE/IP"

Your first scenario is truly GRE over IPsec, or sometimes called a protected GRE tunnel. The fact you have not modified the tunnel mode simply means that the tunnels are continuing to use GRE. In your first scenario, if you went to the configuration and configured tunnel mode gre ip this command would not show in the Tunnel interface configuration because it is the implicit setting.

The key thing to keep in mind here is that the DF bit of the innermost encapsulated packets is not copied to the outer IP headers as they are added during tunneling encapsulation. That means that if the encapsulated packet can be created, it can also be fragmented even if the innermost packet could not be fragmented itself.

In your first scenario, pinging with 1477 bytes and the DF bit set resulted into failure because the IP MTU setting on GRE tunnels is set to 1476 (1500 - 4 (GRE) - 20 (new IP) = 1476). Forcing the router to send a 1477 bytes long IP packet through a Tunnel interface that allows for IP packets only up to 1476 bytes long will cause these packets to be dropped. This means that your 1477 bytes long ping packet was dropped before even being encapsulated into GRE.

However, using 1476 bytes as the ping size in your first scenario caused the GRE tunnel to accept your original packet and create a new GRE-encapsulated packet with the total size of 1500 bytes. This new GRE-encapsulated packet with its new IP header was not carrying the DF flag anymore, and was then handed over to IPsec for encryption. IPsec added yet another IP header plus ESP and the required fields and certainly ended up with a packet that exceeds 1500 bytes. However, because this new IPsec-protected IP packet did not inherit the DF flag of the innermost ping packet, it was allowed to be fragmented. That is why it worked, although the overhead in this case is quite huge - GRE+IPsec.

2. Why tunnel provide higher throughput in first scenario?

Because of the order of operations and the maximum size of packet to do the first encapsulation operation. In the first scenario, the overhead of GRE is 24 bytes, so the tunnel accepts IP packets of sizes up to 1476 bytes. After they are accepted, they get GRE-encapsulated, and the resulting encapulated packets do not inherit the DF flag, meaning that they can be fragmented. The fact that the additional IPsec protection adds another 57 bytes onto the GRE-encapsulated packet is not visible here because the GRE-encapsulated packet is fragmentable again, and so is the resulting IPsec-protected GRE-encapsulated packet.

In the second scenario, you were using a plain IPsec tunnel without GRE. The overhead of IPsec appears to be 57 bytes. IP packets that can be carried through this tunnel can be at most 1443 bytes long. If they are longer, the tunnel refuses to accept them as they exceed its IP MTU setting value. That is why the second scenario appears to accept shorter IP packets.

Remember: an IP packet entering a tunnel interface must be at most as big as the IP MTU setting on that tunnel interface. If it is bigger, the interface will summarily drop it even if the resulting encapsulated packet could be fragmented. By the way, the overhead in the second scenario is lower by 24 bytes, as there is no additional IP+GRE header involved. Just because of the MTU issues on the tunnel, it appears reversed.

This is quite a convoluted concept so please feel welcome to ask further!

Best regards,

Peter

View solution in original post

Peter Paluch · ‎08-01-2013

Hi Xie,

1. In the first scenario, why tunnel can encrypt traffic even it's showing "Tunnel protocol/transport GRE/IP"

Your first scenario is truly GRE over IPsec, or sometimes called a protected GRE tunnel. The fact you have not modified the tunnel mode simply means that the tunnels are continuing to use GRE. In your first scenario, if you went to the configuration and configured tunnel mode gre ip this command would not show in the Tunnel interface configuration because it is the implicit setting.

The key thing to keep in mind here is that the DF bit of the innermost encapsulated packets is not copied to the outer IP headers as they are added during tunneling encapsulation. That means that if the encapsulated packet can be created, it can also be fragmented even if the innermost packet could not be fragmented itself.

In your first scenario, pinging with 1477 bytes and the DF bit set resulted into failure because the IP MTU setting on GRE tunnels is set to 1476 (1500 - 4 (GRE) - 20 (new IP) = 1476). Forcing the router to send a 1477 bytes long IP packet through a Tunnel interface that allows for IP packets only up to 1476 bytes long will cause these packets to be dropped. This means that your 1477 bytes long ping packet was dropped before even being encapsulated into GRE.

However, using 1476 bytes as the ping size in your first scenario caused the GRE tunnel to accept your original packet and create a new GRE-encapsulated packet with the total size of 1500 bytes. This new GRE-encapsulated packet with its new IP header was not carrying the DF flag anymore, and was then handed over to IPsec for encryption. IPsec added yet another IP header plus ESP and the required fields and certainly ended up with a packet that exceeds 1500 bytes. However, because this new IPsec-protected IP packet did not inherit the DF flag of the innermost ping packet, it was allowed to be fragmented. That is why it worked, although the overhead in this case is quite huge - GRE+IPsec.

2. Why tunnel provide higher throughput in first scenario?

Because of the order of operations and the maximum size of packet to do the first encapsulation operation. In the first scenario, the overhead of GRE is 24 bytes, so the tunnel accepts IP packets of sizes up to 1476 bytes. After they are accepted, they get GRE-encapsulated, and the resulting encapulated packets do not inherit the DF flag, meaning that they can be fragmented. The fact that the additional IPsec protection adds another 57 bytes onto the GRE-encapsulated packet is not visible here because the GRE-encapsulated packet is fragmentable again, and so is the resulting IPsec-protected GRE-encapsulated packet.

In the second scenario, you were using a plain IPsec tunnel without GRE. The overhead of IPsec appears to be 57 bytes. IP packets that can be carried through this tunnel can be at most 1443 bytes long. If they are longer, the tunnel refuses to accept them as they exceed its IP MTU setting value. That is why the second scenario appears to accept shorter IP packets.

Remember: an IP packet entering a tunnel interface must be at most as big as the IP MTU setting on that tunnel interface. If it is bigger, the interface will summarily drop it even if the resulting encapsulated packet could be fragmented. By the way, the overhead in the second scenario is lower by 24 bytes, as there is no additional IP+GRE header involved. Just because of the MTU issues on the tunnel, it appears reversed.

This is quite a convoluted concept so please feel welcome to ask further!

Best regards,

Peter

Joseph W. Doherty · ‎08-01-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The key thing to keep in mind here is that the DF bit of the innermost encapsulated packets is not copied to the outer IP headers as they are added during tunneling encapsulation.

BTW, I thought original packet DF is copied if you enable PMTUD on the tunnel interface. I also thought I recall DF isn't added by tunnel unless tunnel PMTUD is enabled and original packet has DF set.

In any case, the OP might find these of interest:

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

http://www.cisco.com/en/US/docs/interfaces_modules/shared_port_adapters/configuration/7600series/76cfvpnb.pdf

Peter Paluch · ‎08-01-2013

Hi Joseph,

BTW, I thought original packet DF is copied if you enable PMTUD on the  tunnel interface.

Why, you are absolutely correct here. It's just that I have commented the tunnel operation as configured in Xie's examples. If the tunnel was configured with tunnel path-mtu-discovery command, the value of the DF bit of the inner packet would be copied to the outer header.

I also thought I recall DF isn't added by tunnel  unless tunnel PMTUD is enabled and original packet has DF set.

Hmmm... Isn't this the same statement, just formulated in different words?

Best regards,

Peter

Joseph W. Doherty · ‎08-01-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, I thought original packet DF is copied if you enable PMTUD on the tunnel interface.

Why, you are absolutely correct here. It's just that I have commented the tunnel operation as configured in Xie's examples. If the tunnel was configured with tunnel path-mtu-discovery command, the value of the DF bit of the inner packet would be copied to the outer header.

Well I'm glad to know, this time, my recollection was correct - I wasn't 100% certain (it's not a QoS question ). As you didn't mention tunnel PMTUD, I presumed I would be corrected if mistaken, and if not mistaken, I thought tunnel PMTUD should be mentioned so that the OP, and perhaps other readers, didn't assume tunnel's never set the DF bit.

(Oh, and rereading what I wrote, I didn't mean to imply any inaccuracy in what you posted. I apologize if it seemed I did.)

I also thought I recall DF isn't added by tunnel unless tunnel PMTUD is enabled and original packet has DF set.

Hmmm... Isn't this the same statement, just formulated in different words?

Peter, almost, and what I wrote may not make what I have in mind clear. What I had in mind was, copying a set DF bit vs. copying the DF bit, including when unset. In other words, the tunnel "mirrors" (when tunnel's PMTUD active) the original packet's DF setting, it doesn't also (I believe) ever set the DF bit when the original packet's DF bit is unset or if tunnel PMTUD is inactive (which it technically could).

XIE YAO · ‎08-01-2013

Hi Peter, Hi Joseph,

Thanks a lot for helping me understand this topic.

So what happened in the first scenario is:

1. the 1476 byte packet was firstly adding 24 header. (IP+GRE)

2. another 57 byte IPsec header was added before the new packet.

3. 1557 byte packet was fragmented again since the df set is not inherited when PMTUD is not enable, is this the so called "fragmentation after encryption", which will heavily impact the throughput and should be avioded?

Later I'm going to test a few more with PMTUD and "crypto ipsec df" command to make this topic complete.

Thanks again!

Regards

Xie

Joseph W. Doherty · ‎08-03-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Yes, that would be fragmentation after encryption, which you want to avoid.

Even worse would be if you send a 1500 byte packet. GRE adds 24 so now the GRE packet must be fragmented. First fragment would be 1500 and second fragment 44 (IP header 20 and the last 24).

The each fragement is encrypted adding another 57. The second fragment grows to 101, but the first fragment at 1557 needs to be fragmented again. Fragment 1A is 1500, fragment 1B is 77 and fragment 2 is 101.

If you don't send a packet larger than 1419, both the GRE and IPSec header can be added without the need to fragment at all.

I think the above is correct - Peter please jump in again, if not.