I have a VPN tunnel between an 871 and 877, the tunnel seems to be fine, but checking the tunnel using SDM shows an error.
Checking the tunnel status... Up
Send Error :7939
Received Error :0
A ping with data size of this VPN interface MTU size and 'Do not Fragment' bit set to the other end VPN device is failing. This may happen if there is a lesser MTU network which drops the 'Do not Fragmet' packets.
1)Contact your ISP/Administrator to resolve this issue. 2)Issue the command 'crypto ipsec df-bit clear' under the VPN interface to avoid packets drop due to fragmentation.
Are the send errors anything to worry about?
Do I need to issue the 'crypto ipsec df-bit clear' on the routers?
Any info would be much appreciated.
The crypto IPSec-df bit clear will clear the df-bit in the IPSec header being formed during encryption. this may jus help in your case. You can also change the TCP MSS on the egress interface so this will rule out MTU related issues.
In general the issue has to do with what happens to a packet if it has the Do Not Fragment bit set and it gets to a network device and exceeds the maximum frame size. The normal action would be to fragment the packet and send over the segment with a smaller max frame size. But the DF bit prevents fragmentation and so the packet will be dropped.
Issues with max frame size and DF are common when doing IPSec (and when doing GRE, and especially common if doing both IPSec and GRE). This is caused by the extra headers that IPSec and GRE add to the original packet. That is what SDM is reporting, is that the IPSec tunnel can not pass packets that exceed the maximum frame size.
There are several solutions to this problem. One solution is the Path MTU Discovery. This checks over the path to discover what is the largest size packet that will pass without fragmentation. When PMTUD works correctly the end stations negotiate a frame size that will not exceed the capacity of the link in the middle. Unfortunately in many networks (especially ones which cross other public networks) PMTUD does not function correctly (most often because the ICMP error message it depends on may be filtered out). Another of the solutions is the ipsec-df bit clear which will clear the DF bit and allow fragmentation. If you do not have concerns about why the DF bit was set and are comfortable that clearing it will not impact anything (and usually clearing the DF bit has little or no impact) then you can do this. Another solution is to use the ip cp adjust-mss command which will look at TCP connection setup and will force the MSS negotiation to use a smaller MTU and will avoid the fragmentation issue.
I believe that both clear df bit and adjust-mss will work. And it may be that in your network that PMTUD is working. If your users are not experiencing problems I am not sure that I would do anything - even if SDM thinks there is an error.
Thanks for responding.
I'm not using GRE, just IPSEC. The user are complaining about the speed of the VPN, but it's very difficult for me to work out whether there is a problem, or if this is just general whinging :-) I decided to do some troubleshooting with SDM, and then found this problem.
I don't mind trying the "ipsec-df bit clear" command, and can also try "ip tcp adjust-mss"... what value should I choose for this, and does these changes need to be applied on both sides of the VPN (ie, the 877 and 871). My VPN tunnel is not a virtual interface, so should I apply this to the vlan or the outside interface?
MSS is one time value apply anywhere along the path prefarably outgoing interface and you will set the maximum segment segment. I would recommend some where around 1300 (it works for me ). Suggest giving the same MSS on both routers
I set MSS to 1300 on the outside interface of both routers, but SDM still reports a problem. I also tried setting 'crypto ipsec df-bit clear' on both routers, again SDM still reports a problem.
I'm very concerned that the Send Errors SDM shows for this tunnel could be causing performance problems.
I've just been playing with the config, and removed my ACL from the outside interface. The same SDM test completed without warnings!
Could the "deny icmp any any redirect" in my ACL have caused this problem? How should I remove this without opening up my router to icmp attacks?
ip access-list extended OUTSIDE
remark --- Drop redirects
deny icmp any any redirect
remark --- Drop loopback traffic
deny ip 127.0.0.0 0.255.255.255 any
remark --- Stupid IOS, this is traffic from inside an ipsec tunnel!
permit ip 192.168.x.x 0.0.0.255 192.168.x.x 0.0.0.255
permit ip 10.52.x.x 0.0.0.255 172.17.x.x 0.0.0.15
remark --- Drop RFC1918
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any
deny ip 192.168.0.0 0.0.255.255 any
remark --- Drop mcast
deny ip 188.8.131.52 184.108.40.206 any
remark --- Drop MS APIPA
deny ip 169.254.0.0 0.0.255.255 any
remark --- Allow incoming SSH
permit tcp any host 84.x.x.x eq 22
remark --- Allow ESP and ISAKMP
permit esp host 84.x.x.x host 84.x.x.x
permit udp host 84.x.x.x host 84.x.x.x eq isakmp
remark --- Drop everything else
deny ip any any log
First a comment about an earlier post: when I use the tcp adjust-mss I apply it to the LAN interface where users connect. In my terminology that would be the inside interface. If you used it on the outside interface, I would suggest that you move it to the inside interface and see if it makes any difference.
In the current post, I have a hard time believing that denying icmp any any redirect is causing any problem. And I doubt that removing that line would increase your exposure any significant amount. If you are concerned about icmp attacks then you need to deny a lot more icmp than just redirects.
I am curious about the fact that removing the access list allows the SDM test to complete with no errors. Perhaps you can remove the one line about icmp redirect, apply the rest of the access list, and let us know what happens.
The deny any any at the bottom includes the log parameter. Are there messages in the log about what is being denied. This might provide a clue about what SDM is seeing as a problem.
Thanks for the reply.
I correct my mistake, and re adjusted MSS onto my vlan1 interface, where the users connect. This didn't make any difference.
However, I removed the ' deny icmp any any redirect', but left the rest of the ACL in place, and the SDM test failed! I guess this is good news in a way :-)
I found this in my log entry...
018845: Nov 11 08:31:08.956 GMT: %SEC-6-IPACCESSLOGDP: list OUTSIDE denied icmp 84.x.x.x -> 84.y.y.y (0/0), 1 packet
... I guess this is caught by the last entry in my ACL of 'deny ip any any log'.
So, I guess I need to permit some types of icmp traffic, could you give me some pointers?
Moving the adjust-mss may not have corrected the error reported by SDM but you may want to observe user traffic and see if it helps to improve their response time.
You have made good progress in solving the problem when you found the entry in the log for the denied packet. I believe that you do need to allow some icmp throught your access list. I would start with permit icmp any any. This will allow all icmp. When you do this then check and verify that SDM no longer reports a problem. After you have cleared the error reported by SDM you may want to consider which types of icmp you want to allow and which types you want to deny. Most of the icmp messages are helpful. I find frequently that people are worried about possible negative aspects of icmp and deny all icmp. This frequently is counter productive and you may wind up missing information that could have been helpful. So if you want to get into this I would suggest that you start with a list of the icmp message types (online help in IOS would be one way to get that list: permit icmp any any ?). Then you can go through the icmp message types and identify those specific messages that you do not want to allow. You can create access list entries to deny those specific messages.
I've got a list of icmp types from typing 'permit icmp any any ?' in IOS... theres quite a list, 57!!
How should I decide which ones to allow and which ones to block, I don't even know what they mean :-) Do Cisco publish any recommendations?
bim7dsl(config-ext-nacl)#permit icmp any any ?
<0-255> ICMP message type
administratively-prohibited Administratively prohibited
alternate-address Alternate address
conversion-error Datagram conversion
dod-host-prohibited Host prohibited
dod-net-prohibited Net prohibited
echo Echo (ping)
echo-reply Echo reply
fragments Check non-initial fragments
general-parameter-problem Parameter problem
host-isolated Host isolated
host-precedence-unreachable Host unreachable for precedence
host-redirect Host redirect
host-tos-redirect Host redirect for TOS
host-tos-unreachable Host unreachable for TOS
host-unknown Host unknown
host-unreachable Host unreachable
information-reply Information replies
information-request Information requests
log Log matches against this entry
log-input Log matches against this entry, including input
mask-reply Mask replies
mask-request Mask requests
mobile-redirect Mobile host redirect
net-redirect Network redirect
net-tos-redirect Net redirect for TOS
net-tos-unreachable Network unreachable for TOS
net-unreachable Net unreachable
network-unknown Network unknown
no-room-for-option Parameter required but no room
option Match packets with given IP Options value
option-missing Parameter required but not present
packet-too-big Fragmentation needed and DF set
parameter-problem All parameter problems
port-unreachable Port unreachable
precedence Match packets with given precedence value
precedence-unreachable Precedence cutoff
protocol-unreachable Protocol unreachable
reassembly-timeout Reassembly timeout
redirect All redirects
reflect Create reflexive access list entry
router-advertisement Router discovery advertisements
router-solicitation Router discovery solicitations
source-quench Source quenches
source-route-failed Source route failed
time-exceeded All time exceededs
time-range Specify a time-range
timestamp-reply Timestamp replies
timestamp-request Timestamp requests
tos Match packets with given TOS value
ttl-exceeded TTL exceeded
unreachable All unreachables
Would it be better to permit all icmp where the source is the other end of my VPN, a known fixed IP? And then deny icmp from elsewhere?
Thanks for all your help on this.
Trying to go through the list and explain each one is beyond the scope of this forum. My recommendation is to go through the list, identify and deny the ones that you worry about, and permit the rest. I know that a lot of people worry about echo Echo (ping). Do you want any one to be able to ping you? Do you want anyone in your network to be able to ping outside of your network?
I believe that there are some very obvious choices. For example if you deny:
packet-too-big Fragmentation needed and DF set
then Path MTU Discovery is broken. PMTUD is especially important when you deal with IPSec tunnels because the IPSec header which is added to the packet is very likely to create the need for fragtmentation.
My perspective is that icmp is an easy target when people worry about security of the network at a high level. But when you look at the details of what is involved there are a lot of messages in icmp that are really helpful. I believe that people usually do not understand what they are giving up when they just deny icmp from remote addresses.
Rick - thanks for all the help. I've still not solved the problem, but I'm playing around with permit and deny on different icmp traffic.
TCP Adjust MSS does not work with UDP and ICMP. I have also been cautioned about performance with remove df-bit enabled, expecially globally and it will also cause issues with TCP. I use the tcp adjust-mss 1310 on my tunnel interfaces and apply a route map to remove df flag from UDP. This should take care of tcp and udp in ipsec tunnels.
ip tcp adjust-mss 1360
ip access-list extended udp-df1
permit udp any any
route-map udp-df0 permit 10
match ip address udp-df1
set ip df 0
route-map udp-df0 deny 99
ip policy route-map udp-df0
sh route-map udp-df0
route-map udp-df0, permit, sequence 10
ip address (access-lists): udp-df1
ip df 0
Policy routing matches: 124071 packets, 35760711 bytes
route-map udp-df0, deny, sequence 99
Policy routing matches: 0 packets, 0 bytes
We use dmvpn to an enormous scale over wired/broadband/wireless internet access for various business lines/uses. SSL/TLS cannot be fragmented so any application with large payload segments using SSL/TLS will have issues over GRE IPSEC tunnels. Outlined below are the necessary steps for large packets to traverse the VPN without issues.
1) TCP - On the tunnel interface, reduce the MTU and MSS to the Cisco best practice configuration for GRE/IPSEC
ip mtu 1400
ip tcp adjust-mss 1360
We have found that some wireless networks use tunneled backhauls which further reduce MTU, so we have reduced both values by -40 bytes to accommodate as follows.
ip mtu 1360
ip tcp adjust-mss 1320
2) The above configuration does nothing for UDP, because it has no way to adjust mss. Some tftp clients will use very large payload segments. We use a route map on the Ingress interface to clear the DF bit on UDP packets so fragmentation will occur.
ip policy route-map FRAG-UDP-rmap
ip access-list extended FRAG-UDP
permit udp any any
route-map FRAG-UDP-rmap permit 10
match ip address FRAG-UDP
set ip df 0
route-map FRAG-UDP-rmap deny 99
There is no implicit deny at the end of a Cisco route-map, so the explicit deny is necessary to prevent matching all traffic, not completely necessary here, but a best practice.
3) ICMP - The only reason ICMP packets should be large is for testing. You can accomplish this with the following lines.
ip access-list extended FRAG-UDP
permit icmp any any
4) You must configure these elements on both ends of the tunnels.
We have used these configurations globally without issues.
Ive had a look at your config and we are suffering the same sorts of problems with large UDP packets over DMVPN.
What I did was increase the tunnel MTU to 1500 and this allowed an end to end ping to run at 1500 packet size, whereas when the MTU was set to 1400 on the tunnel, this could not be achieved.
Unfortunately we are still seeing issues so would your config allow large UDP packets and how would I identify if the the df bit on the UDP packet was set to 1 in the first place ?
I also have the following command....is that ok ?
crypto ipsec fragmentation after-encryption
Your posts have been very helpful; I have tried to implement those suggestions, but I am still having a problem when using a GRE tunnel.
Since you seem to have experience with tunnels, could you please check this thread:
and suggest a reason why Netflow export is not working correctly?
I assume the reason is this (fragmentation causing checksum errors):
In that case, turning on pre-fragmentation for IPSec VPNs worked. But what could we do with simple GRE tunnels?