Andrew,

This is the maximum size i can ping with the DF bit set, it fails with any size larger than 1415 bytes. The test was done from a PC in site A with IP 10.0.0.100 to site B PC with IP 10.4.0.9

>ping 10.4.0.9 -f -l 1415

Pinging 10.4.0.9 with 1415 bytes of data:

Reply from 10.4.0.9: bytes=1415 time=273ms TTL=126

Reply from 10.4.0.9: bytes=1415 time=176ms TTL=126

Reply from 10.4.0.9: bytes=1415 time=316ms TTL=126

Reply from 10.4.0.9: bytes=1415 time=296ms TTL=126

Ping statistics for 10.4.0.9:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 176ms, Maximum = 316ms, Average = 265ms

Below is the test from the PC in site B to a PC in site A, from this site, anything larger than 1442 bytes will start failing when the DF bit is set.

#ping 10.0.0.100 source 10.4.0.1 size 1442 df-bit

Type escape sequence to abort.

Sending 5, 1442-byte ICMP Echos to 10.0.0.100, timeout is 2 seconds:

Packet sent with a source address of 10.4.0.1

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 152/267/392 ms

Can you help me as to where i should change/set the mtu/mss size? Should be on site A's ASA firewall? on any interface? or tunnel interface?

Cheers,

Esteban

Esteban,

Personally I would change the MSS & MTU in site A in the GRE tunnel. The general rule of thumb for IPSEC & GRE is Cisco recommend to change the MTU of GRE tunnels to 1400 Bytes:-

Gre Encapsulation:- 28 Bytes

IP Header:- 20 Bytes

TCP Header:- 20 Bytes

Thats a total of 68 Bytes of overhead, So if you change the MTU of the GRE tunnel to 1400, and adjust the MSS 1360 you should be OK. The command for the GRE tunnel are:-

mtu 1400

ip tcp adjust-mss 1360

HTH.

Andrew,

I haven't set up GRE tunnels. I have set up ipsec tunnels. By creating crypto maps containing the ACLs to match, ipsec settings matching at each end and applying them to the outside interfaces at each end. I believe then the changes you suggest be done on the outside interface in the ASA firewall (site A), however I m a bit afraid that by changing so it may affect other traffic as all other traffic that is directed to the internet crosses this interface. Do you know if this might be the case?

Esteban

Esteban,

You can do this one of two ways:-

1) Change the interface MTU size

mtu <> <>

2) Change the sysopt connection MSS

sysopt connection tcp-mss <>

I would test with option 2 to being with.

HTH.

Andrew,

I tried changing the mtu and mss as per your suggestion. Below are the changes I did,

1. mtu outside 1400

2. sysopt connection tcp-mss 1360

However i was still not able to browse to a pc in site B from a PC in site A. With the new values i tried few ping tests with the DF bit set and this time nothing over 1320 was allowed as soon as the sizes were increased the ping tests started failing. Also I noticed that RDPing to a PC in site B was slower and it would fail the connection at times.

Should I try any other values, perhaps trying to increase them?

Esteban

Esteban,

Firstly - change the MTU back to 1500, this is the default and from your testing it's sucessfull, and not required to stay at 1400.

You need to keep the MTU 1360 - as this is working. If the maximum ping size that works is 1320 - a data payload of 1320 + 20 bytes for the TCP header = 1340 add the IP header of 20 bytes on top - that is a total of 1360.

What should happen is a TCP connection from site A that will traverse the firewall to site B, the following will now happen.

Client A will send a TCP SYN to client B

In the TCP SYN, the options field there is the MSS value - the client will send something like 1460.

The firewall will see this SYN and the MSS value - it will change the MSS option to 1360.

Client B will recevie the SYN with the option value of 1360.

Client B will send a SYN ACK - in the SYN ACK Client B will send the option MSS as say 1460.

The firewall will see this SYN ACK - and change the MSS option value to 1360

Client A will recevie the SYN ACK with the option MSS value of 1360.

The MSS for that TCP session is now set at 1360.

From the client A machine, can you install a packet capture, then capture the packets when you initiate a session to client b??

Andrew,

Below is the output of the packet capture at the firewall on site A, at the time while trying to connect to the remote PC on site B. As far as I can understand, it seems that either site B is not responding with the ACK to change the MSS to 1360 or the firewall in site A is not changing the mss to 1360 as you mentioned it should happen...

Any ideas on how to work around it? I tried increasing mss to 1460, leaving MTU as 1500 in the firewall at site A but still not luck?

25: 11:10:07.838305 0016.4788.6bff 001c.5826.09ed 0x0800 66: 10.0.6.119.50418 > 10.4.0.13.445: S [tcp sum ok] 4292798776:4292798776(0) win 8192 (DF) (ttl 127, id 6371)

26: 11:10:08.338071 0016.4788.6bff 001c.5826.09ed 0x0800 66: 10.0.6.119.50418 > 10.4.0.13.445: S [tcp sum ok] 4292798776:4292798776(0) win 8192 (DF) (ttl 127, id 6382)

27: 11:10:08.838060 0016.4788.6bff 001c.5826.09ed 0x0800 62: 10.0.6.119.50418 > 10.4.0.13.445: S [tcp sum ok] 4292798776:4292798776(0) win 8192 (DF) (ttl 127, id 6384)

36: 11:10:10.100489 0016.4788.6bff 001c.5826.09ed 0x0800 74: 10.0.6.119 > 10.4.0.13: icmp: echo request (ttl 31, id 6404)

39: 11:10:11.625333 0016.4788.6bff 001c.5826.09ed 0x0800 74: 10.0.6.119 > 10.4.0.13: icmp: echo request (ttl 31, id 6424)

72: 11:10:13.625562 0016.4788.6bff 001c.5826.09ed 0x0800 92: 10.0.6.119.137 > 10.4.0.13.137: [udp sum ok] udp 50 (ttl 127, id 6438)

73: 11:10:15.125115 0016.4788.6bff 001c.5826.09ed 0x0800 92: 10.0.6.119.137 > 10.4.0.13.137: [udp sum ok] udp 50 (ttl 127, id 6465)

74: 11:10:16.625150 0016.4788.6bff 001c.5826.09ed 0x0800 92: 10.0.6.119.137 > 10.4.0.13.137: [udp sum ok] udp 50 (ttl 127, id 6475)

79: 11:10:18.131569 0016.4788.6bff 001c.5826.09ed 0x0800 66: 10.0.6.119.50424 > 10.4.0.13.80: S [tcp sum ok] 3577883248:3577883248(0) win 8192 (DF) (ttl 127, id 6500)

87: 11:10:21.138313 0016.4788.6bff 001c.5826.09ed 0x0800 66: 10.0.6.119.50424 > 10.4.0.13.80: S [tcp sum ok] 3577883248:3577883248(0) win 8192 (DF) (ttl 127, id 6527)

90: 11:10:27.128945 0016.4788.6bff 001c.5826.09ed 0x0800 62: 10.0.6.119.50424 > 10.4.0.13.80: S [tcp sum ok] 3577883248:3577883248(0) win 8192 (DF) (ttl 127, id 6566)

Esteban

Esteban,

From this capture - the issue is no MSS - you are not actually getting a response from the remote end. The MSS is being change by the ASA - so that's good.

Can you capture the remote end when you initate a connection?

Andrew,

These are some stats from the remote router, in site B, when i tried pinging the inside interface of this router from a host in site A, with packets larger than 1415 bytes,

001373: May 23 14:15:57.797 PCTime: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1444, mtu=1443

001374: May 23 14:16:02.746 PCTime: ICMP: echo reply sent, src 10.4.0.1, dst 10.0.6.119

001375: May 23 14:16:02.750 PCTime: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1444, mtu=1443

001376: May 23 14:16:07.756 PCTime: ICMP: echo reply sent, src 10.4.0.1, dst 10.0.6.119

001377: May 23 14:16:07.756 PCTime: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1444, mtu=1443

001378: May 23 14:16:12.766 PCTime: ICMP: echo reply sent, src 10.4.0.1, dst 10.0.6.119

001379: May 23 14:16:12.766 PCTime: CRYPTO_ENGINE: locally-sourced pkt w/DF bit set is too big,ip->tl=1444, mtu=1443

I then tried to change the mss on the inside interface of the router in site B, to 1442. hoping that the firewall on site A would have minimised the packets to sizes lower than 1442 but it did not work. For some reason the firewall on site A always tries to send packets based on a mss of 1460, which I don't know where it is getting it from....

Esteban

Esteban,

OK this is the thing - if you are sending ICMP packets with the DF bit set, with a payload size of 14xx - then the firewall as Site A CANNOT change the MSS.

As previously stated - sysopt connection tcp-mss is for TCP traffic it's looking for the SYN and SYN ACK. ICMP does NOT use TCP it does not use SYN or SYN ACK.

You need to try a connection between site A and site B, try a browse a share or someting. Do something that you expect to fail before yyou change the sysopt connection tcp-mss in the firewall.

Then debug the output of that TCP connection. Have you also changed the MTU of the outside interface back to 1500 on the firewall?

Hi Andrew,

I did the test as per your suggestion. The MTU size in the outside and inside interfaces in the firewall is set to 1500 bytes. MSS is set to 1360 bytes.

I tried to RDP from a client in site A to a client in site B and that worked as suspected. Then I tried mapping a network drive from the same client in site A to a client in site B and it failed as suspected too.

See below for the TCP SYN logs captured in the client at site A.

RDP: CLient in site A, IP: 10.0.6.119 to client in site B, IP: 10.4.0.13

Notice the MSS is set to 1460 bytes. This is the TCP Syn from client in site A to client in site B,

- Tcp: Flags=.S......, SrcPort=64932, DstPort=MS WBT Server(3389), Len=0, Seq=1975208786, Ack=0, Win=8192 (scale factor not found)

SrcPort: 64932

DstPort: MS WBT Server(3389)

SequenceNumber: 1975208786 (0x75BB4B52)

AcknowledgementNumber: 0 (0x0)

+ DataOffset: 128 (0x80)

+ Flags: .S......

Window: 8192 (scale factor not found)

Checksum: 6830 (0x1AAE)

UrgentPointer: 0 (0x0)

- TCPOptions:

- MaxSegmentSize:

type: Maximum Segment Size. 2(0x2)

OptionLength: 4 (0x4)

MaxSegmentSize: 1460 (0x5B4)

+ NoOption:

+ WindowsScaleFactor:

+ NoOption:

+ NoOption:

- SACKPermitted:

type: SACK permitted. 4(0x4)

OptionLength: 2 (0x2)

Notice the MSS is set to 1360 bytes. This is the TCP SYN ACK from client in site B to client in site A.

- Tcp: Flags=.S..A..., SrcPort=MS WBT Server(3389), DstPort=64932, Len=0, Seq=1467915900, Ack=1975208787, Win=17680 (scale factor not found)

SrcPort: MS WBT Server(3389)

DstPort: 64932

SequenceNumber: 1467915900 (0x577E9E7C)

AcknowledgementNumber: 1975208787 (0x75BB4B53)

+ DataOffset: 128 (0x80)

+ Flags: .S..A...

Window: 17680 (scale factor not found)

Checksum: 19945 (0x4DE9)

UrgentPointer: 0 (0x0)

- TCPOptions:

- MaxSegmentSize:

type: Maximum Segment Size. 2(0x2)

OptionLength: 4 (0x4)

MaxSegmentSize: 1360 (0x550)

+ NoOption:

+ WindowsScaleFactor:

+ NoOption:

+ NoOption:

- SACKPermitted:

type: SACK permitted. 4(0x4)

OptionLength: 2 (0x2)

Netbios: CLient in site A, IP: 10.0.6.119 to client in site B, IP: 10.4.0.13

Notice the MSS is set to 1460 bytes. This is the TCP Syn from client in site A

- Tcp: Flags=.S......, SrcPort=64920, DstPort=Microsoft-DS(445), Len=0, Seq=847410545, Ack=0, Win=8192 (scale factor not found)

SrcPort: 64920

DstPort: Microsoft-DS(445)

SequenceNumber: 847410545 (0x32827571)

AcknowledgementNumber: 0 (0x0)

+ DataOffset: 128 (0x80)

+ Flags: .S......

Window: 8192 (scale factor not found)

Checksum: 6830 (0x1AAE)

UrgentPointer: 0 (0x0)

- TCPOptions:

- MaxSegmentSize:

type: Maximum Segment Size. 2(0x2)

OptionLength: 4 (0x4)

MaxSegmentSize: 1460 (0x5B4)

+ NoOption:

+ WindowsScaleFactor:

+ NoOption:

+ NoOption:

- SACKPermitted:

type: SACK permitted. 4(0x4)

OptionLength: 2 (0x2)

Notice that the TCP SYN ACK below does not contain the option field as the other does, hence no MSS is specified and the path can not be found,

- Tcp: Flags=..R.A..., SrcPort=Microsoft-DS(445), DstPort=64920, Len=0, Seq=0, Ack=847410546, Win=8192 (scale factor not found)

SrcPort: Microsoft-DS(445)

DstPort: 64920

SequenceNumber: 0 (0x0)

AcknowledgementNumber: 847410546 (0x32827572)

+ DataOffset: 80 (0x50)

+ Flags: ..R.A...

Window: 8192 (scale factor not found)

Checksum: 52734 (0xCDFE)

UrgentPointer: 0 (0x0)

Do you still think that it is an MTU issue? I don't understand why the firewall or the client in site B don't set the MSS for packets for traffic other than RDP, HTTP or SSH, which always works. But whenever trying SMTP, netbios then MSS is not set. DO you think this can be related to an inspection issue in the firewall?

Changing the MSS has worked, as the response from the remote end is 1360, from the remote end.

Can you install wireshark and perform the same test again - and capture both ends.

To be honest, now this does not look like an MTU/MSS issue - that has been sovled. Not it's something else.

Do no applications work from site A to B? Is these the only sites you have problems?

The applications that work from site A and B are RDP, SSH and HTTP, I have tested. Applications like netbios and smtp don't work. I have tested all of the above applications from site A to other remote sites and I found the same results. So i am now inclined to think that it may be caused by an inspection issue in the firewall. I tried removing the inspect netbios entry from the default inspection policy under the global policy but it still did not work...

Any further ideas?

Esteban