Troubleshooting traffic through the firewall

branfarm1 · ‎11-05-2009

Hi there,

I'm trying to troubleshoot some traffic traversing one of my firewalls and I was hoping someone could help me. I have a TCP datastream that runs around 3Mbps and when I compare packet traces from before the firewall to after the firewall, I'm seeing a lot of instances of dropped/lost packets (indicated by 'TCP Previous Segment lost' on the post-fw side). I'm trying to figure out why this would happen.

First thing I checked was the interface statistics. The physical interface is a Gigabit fiber connection to a 4500 series switch, specifically a X4306-GB line card. The switch side shows no anomalies -- no CRC errors, overruns, runts, etc. The firewall side shows some overruns and no buffers, but not a significant percentage (<.0001% of received packets were overruns, and <.00005% of received packets were 'no buffer'.) The link between the two devices is a trunk that carries 5 different VLAN's, and the average utilization on the link (from the Switch perspecive) is 97Mbps transmitted and 27Mbps received.

Nothing seems out of the ordinary, but due to the nature of the TCP stream I'm observing, the lost packets are sometimes causing issues for the receivers.

Can anyone think of somethign else I should be checking to help pinpoint this? Or, is this within standard behavior for traffic traversing a firewall?

The hardware involved is a PIX 535 running v7.2(2) and a 4506 running IOS v12.2(52)SG.

Thanks in advance,

Brandon

mkharban · ‎11-05-2009

Hi Brandon,

Please try the following setup to increase the MSS limit on the firewall:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a00804c8b9f.shtml#wa

This might be the cause of the issue. In case this does not resolve it please upload the captures and syslog at debug level for the connection in question for further analysis.

branfarm1 · ‎11-06-2009

Thanks for the response. I've attached the pre/post-fw traffic captures that I'm comparing. I'm not sure it's an MSS problem because the frames that aren't making it through the firewall are random sizes, and some that do make it through are larger than some that don't. I've also attached two screenshots where I've highlighted frames on the pre-fw side that didn't make it to the post-fw side.

One thing I noticed on the traffic capture -- The postfw captures always seem to have missing frames after frames that were 1380 bytes.

branfarm1 · ‎11-06-2009

Postfw capture

branfarm1 · ‎11-06-2009

I've attached the outputs from 'show asp drop', 'show blocks', and 'show traffic', as well as two more detailed traffic captures.