Large file copy fails through 4240 sensor

[paultribe]

Customer attempts to copy a large file from a server in an IPS protected vlan to a host in an IPS un-protected vlan and the copy fails if file is greater than about 2Gbytes in size. If the server is moved to the un-protected vlan the copy succeeds. There are no events on the IPS suggesting any blocking or other actions.

[rhermes]

When operating in-line the sensors will aotumaticly drop a packet flow that exceeds a minimum threat rating (I think it was 75% or 80%), you can look for them in the past hour with this CLI command:

sh ev al min-threat-rating 75 past 01:00

[paultribe]

Can you point me to any documentation on this. I was checking the sensor's real time event log and did not see any events at the time of the file copy attempt.

[rhermes]

I got my heads up on this topic from Marcabal on this forum. Correction, any risk rating over 90 will get dropped unless you option things differently.

Here's his post:

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Security&topic=Intrusion%20Prevention%20Systems/IDS&topicID=.ee6e1fc&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cc2a1d8

[paultribe]

I'm not sure if I am missing something here but if there are no events with a RR in the high-risk category firing during the failed copy then what can I change. Apologies if I am msunderstanding something here.

[paultribe]

I have had a look at the links on the other forum and understand the concepts of the default event action override. I need to do some more testing to see if any signatures are firing when the file copy takes place, last time I'm sure I didn't see any.

[rhermes]

If you are not seeing any signatures firing then you might rule out any intertional packet dropping by the sensor. An overly busy inline IPS sensor can drop packets too.

What is your CPU utilization?

How much traffic are you trying to pass thru your sensor?

[paultribe]

The CPU does occasionly peak at 100% when transferring a large file but the copy often fails when the CPU is significantly lower. I know a 4240 has 300Mbit/s throughput but as I understood it traffic would still be serviced but would bypass the inspection process if exceeded, maybe a transition from inspection to non inspection causes the copy to fail like a tcp reset, I may try a sniffer.

I do have TAC involved but like to try and utilise the knowledge of other expert users like yourself to try and rectify issues. Thanks for your help. If you have any other comments please let me know, I will certainly post my findings if you are interested.

[rhermes]

While the 4240 is rated for 300 Mb/s we see packet loss starting at around 100 Mb/s in promiscious mode (and that's NOT full duplex, I'm adding both directions of transmission together). In my experience the "failover" only takes place once the sensor knows that the senor app has crashed, not necessarily that a sensor has been overloaded.

[antonyabraham · Accepted solution]

There could be some normalizer engine events which can drop/modify traffic without firing an alert. Some of them seem to be on by default. Could you try enabling "produce alerts" on the normalizer signatures with deny or modify actions?

Another way would be to put an event action filter for the source or target (or both) and filter out all deny actions. In that way, you are telling the sensor do not block any traffic from or to certain IP address (based on how the filter is formed). I would use this filter to cover all signatures and sub signatures for the source/target in question.

[paultribe]

Thanks - I found that a TCP normalizer sig was causing the issue, this is now resolved.

[pauloroque]

Hi paultribe.

We have the same problem here. I found out that the IPS (in my case asa-ssm-10) is sometime reducing packet's TTL to as low as 5, which causes the packet being droppep by routers along the path.

Could you please inform me what sig you have changed?

Paulo Roque

[scothrel]

Hi guys,

I'd like the second the request for which normalizer sig is causing issues. We have seen sporadic occurances of this situation and we are trying to get a handle on the issue.

Thanks,

Scott Cothrell

IPS Engineering

[bnidacoc]

Make that a third request for the sig.

Thanks.

[nowcommsupport]

To all:

There are many TCP normalizer sigs that do not alert. I found the one that was affecting my customer by turning on alerting on all TCP normalizer sigs and emulating the issue (by attempting the file copy). I then checked the sensor logs and found the offending signature. It may not necessarily be the same signature for your issues so I would suggest you turn on the alerts and emulate the issues you have.

Paul