DMVPN and "ICMP Administratively Prohibited"

mcrockett · ‎07-09-2009

A full-mesh DMVPN architecture is used to implement our corporate WAN. There are security devices at each site in the WAN that establish a TCP connection to a central server at fixed intervals to report status. An alarm is generated whenever a security device fails to report.

Recently there has been a problem with the security devices alarming at several sites. Using tcpdump we have uncovered one reason for the problem.

The remote security device sends a SYN packet that results in a DMVPN connection being established to the site hosting the central server. The packet is detected at the site.

The server immediately responds with a SYN ACK packet that is sent back to the security device.

Instead of routing the response back to the security device, the DMVPN router responds with an ICMP administratively prohibited packet that is sent to the server.

The DMVPN connection has been established between the site. Why is the DMVPN router returning an ICMP Administratively Prohibited response?

This only appears to occur when a TCP connection is being established and the SYN ACK packet is received by the DMVPN router at the central site.

Giuseppe Larosa · ‎07-10-2009

Hello Merton,

just to summarize:

you see incoming TCP sessions started from remote sites blocked at central site router: the syn ack packet is blocked.

You should review firewalling configuration that in a router means CBAC inspect rules.

Are you using CBAC or zone based firewalling on central site router?

if so if the device thinks the incoming TCP session attempt is coming from an insecure interface (less secure , less trusted) it stops it.

Hope to help

Giuseppe

Richard Burts · ‎07-10-2009

Merton

Is it clear that the ICMP administratively prohibited packet is generated when the router drops a packet because it has failed an access list or perhaps CBAC filtering operation? I believe that Giuseppe is correct that you need to check the router and see what filtering of traffic is configured on it and what is causing it to drop the SYN ACK.

HTH

Rick

HTH

Rick

mcrockett · ‎07-10-2009

The SYN packet sent from the security device to the central server is forwarded by the DMVPN routers at the site hosting the security device and the site hosting the central server.

The SYN ACK packet sent from the central server triggers the ICMP Destination Unreachable, Administratively Prohibited packet. Most often this will be generated by the DMVPN router at the site hosting the central server; however, they can be generated by the DMVPN router at the site hosting the security device.

This is not a case where all connection attempts from a single security device fail. The security device is able to establish connections for hours at a time without problems.

These events tend to occur more frequently as the volume of network traffic increases; however, they do occur when the network is "idle".

For a number of years, my exposure to Cisco devices has been limited. My gut instinct is that this is a resource problem but I can't rule out the inspect rules.

Are the inspect rules applied to packets moving between the "internal" and IPSec tunnel interfaces?

Giuseppe Larosa · ‎07-10-2009

Hello Merton,

>> Are the inspect rules applied to packets moving between the "internal" and IPSec tunnel interfaces?

prepare a filtered version of the configuration of central site router and remote site router.

Remove public ip addresses (use placeholders) user/pwd pairs.

You can use the attach option to send the file on this forum.

What you have added in this post makes to think of a lack of resources as you say.

Hope to help

Giuseppe

mcrockett · ‎07-10-2009

While most of the ICMP Destination Unreachable, Administratively Prohibited packets are generated by the DMVPN router at the site hosting the central server; they are also being generated by the DMVPN routers at the sites hosting the security device.

In all cases where this occurs, the packet is generated when the SYN ACK packet is processed by the DMVPN router.

mcrockett · ‎07-11-2009

I'd like to thank those that responded for your insights. They helped, significantly, in finding and correcting the problem.

The problem stemmed from the ephemeral port range used by the security devices and an old fashioned ACL that was defined on the interior interface of the DMVPN router.

The response from the server was rejected because the connection was not yet in the established state.

Again, thank you for your insights.

Giuseppe Larosa · ‎07-11-2009

Hello Merton,

nice to hear that our comments helped.

so the problem arose when the security devices tried to use ports outside a specified range in the ACL.

Do you mean the central site DMVPN router?

And you have been kind to report the solution of this case this helps to make these forums useful

Hope to help

Giuseppe