ASA 5505 outside -> DMZ traffic problem (Deny TCP No connection and RESET-O)
Continuing a discussion started by user Joseph Lundback in this post: https://supportforums.cisco.com/message/3534223#3534223 which then diverged into a seperate topic around my questions. This post re-iterates my issue as a seperate discussion so as not to confuse the issue of the original post.
My original issue is that my 5505 is seemingly randomly permitting or denying traffic from the outside interface to the DMZ for http and https requests. Some hosts will get and in some won't; the same host will get in and then be denied on the next request. The same or differnet hosts will get in on http and fail on https, etc. All of the hosts in this case are controlled by me on rock-solid internet connections. They are Windows workstation or server boxes running IE 8.
I took the following excerpts from the syslog to demonstrate this:
Here's an example of a succeeded connection, which I just took off the syslog. This morning it is working from one particular host (my laptop):
6 Jan 13 2012 08:10:05 18.104.22.168 4116 HelpDesk 443 Built inbound TCP connection 116046 for outside:22.214.171.124/4116 (126.96.36.199/4116) to dmz:HelpDesk/443 (fios_static_ip/443)
6 Jan 13 2012 08:10:05 188.8.131.52 4116 HelpDesk 443 Teardown TCP connection 116046 for outside:184.108.40.206/4116 to dmz:HelpDesk/443 duration 0:00:00 bytes 20789 TCP FINs
and here's from another host's attempt that does not work at the same time:
6 Jan 13 2012 08:17:57 220.127.116.11 4103 HelpDesk 443 Built inbound TCP connection 116093 for outside:18.104.22.168/4103 (22.214.171.124/4103) to dmz:HelpDesk/443 (fios_static_ip/443)
6 Jan 13 2012 08:17:57 126.96.36.199 4103 HelpDesk 443 Teardown TCP connection 116093 for outside:188.8.131.52/4103 to dmz:HelpDesk/443 duration 0:00:00 bytes 20736 TCP Reset-O
Interesting: now I'm getting a different type of failure. Yesterday I had a DENY like Joseph, this morning I have a RESET-O (what the heck is that?) on the teardown message...
and here's the same failing host getting in on port 80 instead of 443 at the same time:
6 Jan 13 2012 08:23:34 184.108.40.206 4111 HelpDesk 80 Built inbound TCP connection 116106 for outside:220.127.116.11/4111 (18.104.22.168/4111) to dmz:HelpDesk/80 (fios_static_ip/80)
6 Jan 13 2012 08:23:40 22.214.171.124 4111 HelpDesk 80 Teardown TCP connection 116106 for outside:126.96.36.199/4111 to dmz:HelpDesk/80 duration 0:00:06 bytes 950 TCP FINs
At the time I originally joined the other discussion I was getting DENY messages nearly identical to those of the other user from the other post.Here;s an example:
6 Jan 10 2012 23:11:37 106015 HelpDesk 80 188.8.131.52 16582 Deny TCP (no connection) from HelpDesk/80 to 184.108.40.206/16582 flags SYN ACK on interface inside
6 Jan 10 2012 23:11:37 302013 220.127.116.11 16582 HelpDesk 80 Built inbound TCP connection 105773 for outside:18.104.22.168/16582 (22.214.171.124/16582) to dmz:HelpDesk/80 (fios_static_ip/80)
Notice that the DENY message is for the INSIDE interface, while the original connection was set up coming in from the OUTSIDE interface. When I realized that I looked at the web server responding to the request and theorized that it was randomly sending the response out the wrong nic. I don't know why exactly it would do that, but when I shut down that nic and gave the web server only one interface to listen to, the DENY behavior ceased and was replaced with the RESET behavior.
Again, just to re-iterate, the results are random. Some connections complete, others are terminated, with no apparent pattern.
Cisco responder Julio Carvaja mentioned TCP State Bypass as a possible solution, which I will ask him to repeat here as a response to this post...
ASA 5505 outside -> DMZ traffic problem (Deny TCP No connection
Ok, lots of progress with this over the weekend. There were a lot of moving parts and several things were contributing to my problems.
First thing I did, like many before me, was to blow away the entire ASA config and start from a clean slate. My advice to anyone with potential config problems: get yourself a good file-compare tool like Beyond Compare, save your current config, factory-reset the device, and start putting the config back together one step at a time. Save your config off to a text file at each step.Use the file compare tool to see differences between two configs if something stops working.
If you're using ASDM there's an option in there to show you the CLI commands before applying them as you make changes through the UI. Use this option! it's incredibly useful to see what the UI wants to do to the device and to aid in documenting what you have done with your configuration. In many cases this feature provides a better explanation of a particular UI element than does the ASDM documentation.
As I said, my problems came from several sources. The ASA had a few too many unnecessary/redundant configurations in it left over from several days of debugging. Any of these could have cause me a problem, and their presence certainly clouded the issues. Solution: blew away the entire config and methodically rebuilt it until I could see that the ASA was passing traffic as I expected it to.
Final config:an ACL rule to permit specific incoming traffic (http/s) on the outside and a static NAT rule to provide translation and forwarding between outside and DMZ. Ultimately it was not necessary to include the state inspection bypass, because there were two other problems outside of the ASA contributing to the failure:
1. The server on my DMZ network, running Apache under Ubuntu 11.10, was not responding appropriately to all incoming requests. That's related to having multiple nics on the server and it's a topic for another forum! The server seemed to be receiving requests on its DMZ nic via the firewall, but sending responses out the other nic which is on a completely different and unrelated network. WTF? Anyone who might know something about that please post here and lets talk.
2. One of my ouside test machines, running IE 7, is apparently unable to cope with the specific https connection to my system, either at the ASA end point or more likely with the actual DMZ server end point, and it was that server that was producing the RESET-O messages referenced above. My thanks to Julio for explaining this message and getting me thinking about the other peices of the system. That was the only machine that exhibited this behavior.
ASA 5505 outside -> DMZ traffic problem (Deny TCP No connection
Please note that I marked Julio's answer above as correct based on the discussion around the RESET-O message. While the state bypass method was a potential solution at the time, it turned out for various reasons not to be related to the problem. I specifically asked for an example of how to set this up in case I needed it but my current config is running correctly without any need for state bypass.
DocumentationCode download linksGoalRequirementLimitationsSupported ISR
and UCS-E ModelSupported ISRG2 and UCS-E Blades:Supported ISR4K and
UCS-E Blades:Step by Step ConfigurationConfigure one of the connectivity
options to access the Cisco IMC from the n...
Firepower Threat Defense (NGFWv) on UCS E-series - Transparent Mode in
HA DocumentationCode download linksGoalRequirementLimitationsSupported
ISR and UCS-E ModelSupported ISRG2 and UCS-E Blades:Supported ISR4K and
UCS-E Blades:Step by Step ConfigurationCo...
Question I am currently unable to specify "crypto keyring" command when
configuring VPN connection on my cisco 2901 router. The following
licenses have been activated on my router :