Upload to DMZ fails from inside interface

Unanswered Question
Sep 7th, 2010

I have an ASA 5510 running 8.2(2) with all an outside, inside and dmz port.  Have a server in the dmz running an uploading application.  I have no problem uploading files from outside the network to the dmz server, but when uploading from the inside is when I have issues.  I am able to upload a 250MB PSD file but the application fails when trying to upload a 100MB FLV file.  Again, both these files will upload just fine from an outside connection and when sending from within the DMZ.  We have tried other types of files and a smaller FLV with some success, but it seems that when I try to upload a compressed video file that is over say 30MB, the upload fails...again from the outside it works just fine with any file that I throw at it.

I have tried natting the traffic from inside to dmz and tried without nat.  I have been on the phone with Cisco TAC and they have gone over my config and tell me that this should work.  There are no errors on the ASA.  We have have looked at the application and there are no errors either.  This seems like the TCP connection gets interrupted.

Has anyone had any issues uploading files to a server in their DMZ from the Inside interface, but from any other connection it works fine?  I know this is probably something simple and I am just over analyzing it, so any help will be greatly appreciated.

Thanks in advance.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
sachinga.hcl Tue, 09/07/2010 - 21:38

Hi Clark,

My Clients also face such a problem when uploading bigger files get dropped or same.

Had you added the following commands for scaling the windows size:

tcp-options window-scale allow

and but not necessary:

exceed-mss allow

queue-limit pkt_num [timeout seconds]

Sets the maximum number of out-of-order packets that can be buffered and put in order for a TCP connection, between 1 and 250 packets. The default is 0, which means this setting is disabled and the default system queue limit is used depending on the type of traffic:

• Connections for application inspection (the inspect command), IPS (the ips command), and TCP check-retransmission (the TCP map check-retransmission command) have a queue limit of 3 packets. If the adaptive security appliance receives a TCP packet with a different window size, then the queue limit is dynamically changed to match the advertised setting.

• For other TCP connections, out-of-order packets are passed through untouched.

If you set the queue-limit command to be 1 or above, then the number of out-of-order packets allowed for all TCP traffic matches this setting. For application inspection, IPS, and TCP check-retransmission traffic, any advertised settings are ignored. For other TCP traffic, out-of-order packets are now buffered and put in order instead of passed through untouched.

The timeout seconds argument sets the maximum amount of time that out-of-order packets can remain in the buffer, between 1 and 20 seconds; if they are not put in order and passed on within the timeout period, then they are dropped. The default is 4 seconds. You cannot change the timeout for any traffic if the pkt_num argument is set to 0; you need to set the limit to be 1 or above for the timeout keyword to take effect.

Also check more tcp-map Commands :

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpnorm.html

HTH

Sachin Garg

mirober2 Wed, 09/08/2010 - 06:14

Hi Joe,

What protocol is being used to upload the files (i.e. HTTP, FTP, SMB, etc.)?

Have you had a chance to gather syslogs and packet captures on either side of the ASA during a time when the transfer fails? These would likely help narrow down the issue, or at least help you focus on the problem device.

-Mike

joeclarktx Wed, 09/08/2010 - 07:15

Hi Mike,

This is using HTTP to transfer.  I have done all the packet captures on all sides of the ASA (inside, outside, dmz) and I looked at these with the Cisco tech.  We saw a few (TCP Dup Ack) lines but the tech said that is fine.  At the end there were some lines of (TCP Retransmission)(TCP segment of a reasembled PDU).

I am going to try the fix posted above and will post back when I am able to try it.

mirober2 Wed, 09/08/2010 - 07:19

Hi Joe,

What do the captures actually show when the upload fails? Is the TCP connection reset? If so, who is sending the reset packet? Do packets just stop flowing and eventually the connection times out?

If the captures were taken simultaneously on either side of the ASA, it should be easy to see if the firewall is causing the problem (i.e. packets hit one side of the ASA but don't come out the other side, or the ASA generates the RST packet), or if the problem is somewhere else upstream (i.e. the captures look the same on both sides of the firewall). If the firewall is causing the problem, you can look at the syslogs to determine why the connection is failing. If the problem is upstream, you can start stepping through your network to find the problem.

-Mike

joeclarktx Wed, 09/08/2010 - 07:44

Hi Mike,

Unfortunately the captures were not taken at the same time.  I will try to reconfigure to allow the captures at the same time and let you know.  I will also get the syslogs for the same time.  I am using wireshark (free) for the captures, is there any other program that you would recomend?

Joe

sachinga.hcl Wed, 09/08/2010 - 08:57

Hi Clark,


Collect the syslog output and the output from these commands after the connection fails.

•show capture capture-inside

•show capture capture-outside

•show capture mss-capture

•show asp drop

Define a pair of access-lists which identify the packets as they ingress and egress the outside and inside interfaces

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a00804c8b9f.shtml

Packet Capturing using ASDM (best for beginners)

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080a9edd6.shtml

Configure a packet capture in the Cisco ASA

http://analysisandreview.com/cisco/how-to-configure-a-packet-capture-in-the-cisco-asa/

Easy packet captures straight from the Cisco ASA firewall

http://blogs.techrepublic.com.com/networking/?p=1317

You can see the video at youtube for capturing packets in ASA at follow links:

http://www.youtube.com/watch?v=FBJ1aoHfw-c

http://www.youtube.com/watch?v=7yah11PVB5U

I believe your packet capturing tutorial would be best if you go through these links I have provide above collectively as well for further analysis of captured packets.

Please rate if you find the information provided here useful to you.

HTH

Sachin Garg

Message was edited by: sachinga.hcl

joeclarktx Wed, 09/08/2010 - 10:33

Hey guys,

OK, here are the captures from both sides.  I captured with a failed file (100MB FLV) that stopped during the transger and a successful file (250MB PSD).  I say successful because it reached 100%, but it through an undefined error this time and I have the web guys looking into that.

Thanks

Joe

mirober2 Wed, 09/08/2010 - 10:49

Hi Joe,

There are a couple of problems with the captures:

1. The in-cap-failed capture is unidirectional (17.0.0.136 -> 199.117.181.101 only)

2. The dmz-cap-failed capture doesn't contain the file upload traffic

Can you try to repeat the test but make sure that the captures are bidirectional and are taken simultaneously so they both show the upload?

-Mike

joeclarktx Wed, 09/08/2010 - 13:12

Ok, I think I got this...I had to increase the buffer to get more info and there may be more but the files are at 10MB already.

mirober2 Wed, 09/08/2010 - 13:37

Hi Joe,

The captures don't contain the FLV transfer, however they do show a significant amount of out-of-order packets and indicate some packet loss happening in the network.This indicates a network problem somewhere in the path.

I would suggest checking each of the various hops in the network for any problems such as interface errors. Since the transfers seem to work okay between the outside and DMZ, I would focus my efforts on the network infrastructure behind the inside interface. If you have a spare interface on the ASA, you could connect a host directly to the ASA and try the transfer again to see if the issue remains.

Hope that helps.

-Mike

joeclarktx Wed, 09/08/2010 - 13:48

Hey Mike,

I could increase the buffers up to 50 MB and see if I get the FLV transfer if that would help.

As far as the hops go, I am attached to a switch that is connected to the router that goes straight to the ASA, and then to the DMZ.  There really isn't a very large infrastructure on the inside.

If there are a lot of out-of-order packets, shouldn't there be some type of error message, and I still can't explain why a larger PSD or EPS file will go through.

Thanks for all your help

Joe

joeclarktx Wed, 09/08/2010 - 14:03

Mike,

Here is the capture from a successful transfer of a 100MB EPS file.

joeclarktx Wed, 09/08/2010 - 14:10

And this is a failed 96MB FLV file.  the captures are a lot smaller and I don't see the drops like before.

joeclarktx Thu, 09/09/2010 - 13:35

Hey Mike,

I have tried all the solutions that were suggested and what I could find on the Internet.  I have added a second NIC to the server and gave it an internal address and the uploads work like they are supposed to.  I really don't know what else to try.

Joe

mirober2 Thu, 09/09/2010 - 13:41

Hi Joe,

Were you able to setup the ASP drop captures that Sachin suggested? If so, did you see any packets from the transfer being dropped?

If not, try to setup this capture:

capture drop type asp drop all

Then, do 'clear asp drop' to clear the drop counters, start the transfer, and then do 'show asp drop' and 'show cap drop' to see if any of the packets from the upload are being dropped by the ASA.

-Mike

mirober2 Thu, 09/09/2010 - 13:45

Also, it's odd that only FLV files are failing. Do you have any IDS/IPS devices in the network that would be inspecting these packets? Any HTTP proxies that would be inline with these transfers?

-Mike

mirober2 Thu, 09/09/2010 - 13:51

Hi Joe,

Have you tried disabling the IPS temporarily and see if the issue continues? If it's an AIP module, you can use the 'no ips inline' or 'no ips promiscuous' command inside the active policy-map. Try that and let us know if the issue remains without the IPS being active.

-Mike

joeclarktx Thu, 09/09/2010 - 14:11

I've taken the IPS out of the mix and still have the same result.

Actions

This Discussion