Processing Load Percentage = 100

[Diego Armando Cambronero Arias]

Hello,

Guys I have been working with 3 IPS 2 4260 and a 4270 since yesterday I have noticed that the Inspection LOAD is RED. On the 4260 the inspection load is.

SJDetec1# sh statistics virtual-sensor | inc Load

         Processing Load Percentage = 99

On the 4270!!!!!!!

DCDetect1# sh statistics virtual-sensor | inc Load

         Processing Load Percentage = 100

DCDetect1# sh statistics analysis-engine

Analysis Engine Statistics

   Number of seconds since service started = 174759

   The rate of TCP connections tracked per second = 0

   The rate of packets per second = 4711

   The rate of bytes per second = 8402

   Receiver Statistics

      Total number of packets processed since reset = 823334516

      Total number of IP packets processed since reset = 822979042

   Transmitter Statistics

      Total number of packets transmitted = 823478816

      Total number of packets denied = 0

      Total number of packets reset = 0

   Fragment Reassembly Unit Statistics

      Number of fragments currently in FRU = 0

      Number of datagrams currently in FRU = 0

   TCP Stream Reassembly Unit Statistics

      TCP streams currently in the embryonic state = 0

      TCP streams currently in the established state = 0

      TCP streams currently in the closing state = 0

      TCP streams currently in the system = 0

      TCP Packets currently queued for reassembly = 0

   The Signature Database Statistics.

      Total nodes active = 16115

      TCP nodes keyed on both IP addresses and both ports = 3438

      UDP nodes keyed on both IP addresses and both ports = 29

      IP nodes keyed on both IP addresses = 1715

   Statistics for Signature Events

      Number of SigEvents since reset = 153308490

For example in the 4270 we are passing almost nothing through the sensor... ANd its working in promiscuos mode. Why is the Inspection Load that High? In the 4260 is the same.. It is working in promiscuos mode..  There are alarms for missed packets as well.

I have been seeing several Discutions for the same reason but none has a fix. The Issue with the inspection load is random. DUring the day sometimes it high and sometimes is low.

Cisco Intrusion Prevention System, Version 7.0(2)E4

Signature Update    S499.0

Any advice will be really appreciated.

Diego.

[Justin Teixeira]

Hi Diego,

    The first thing to try would be to restore all your signatures to default (provided they're tuned), and disable all custom signatures and see if you still experience the problem.  Overtuning of signatures (for example, enabling every signature on the device) is the most likely reason for the processing load being that high.

Best Regards,

JT

[Diego Armando Cambronero Arias]

Hi I have been tunning the signatures since the las t months. I have tunned arround 900 signatures. Or at least the action is what I have tunned. If I set them all to their default values I will loose all the work done right?

In the 4270 the traffic passed is very very little. For example. In the last 10 hours an Only have 96 events. So it's nothing for a 4270.

Something else to try?

[Christopher Dreier]

Diego,

What is your traffic profile? Can you provide packet captures during a time of high processing load along with a show tech? What changed in your network when the issue started?

You are seeing quite a few signature events:

Analysis Engine Statistics

   Number of seconds since  service started = 174759

   ...

   Statistics for  Signature Events

      Number of SigEvents since reset =  153308490

From the limited amount of information we have, it appears that you may be experiencing traffic spikes throughout the day that lead to many events firing which cause the IPS to become oversubscribed. There is quite a bit of data that we will need to examine to validate this. Please open a TAC case and we can begin to investigate.

Thank you,
Blayne Dreier
Cisco TAC IDS Team**Please check out our Podcast**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

[Justin Teixeira]

Hi Diego,

     You can backup your configuration with the copy "current-config" before restoring the signatures to default.  You can then restore them from the backup later.

Best Regards,

JT

[Diego Armando Cambronero Arias]

Hi,

Just did it. Let's see if the INpection load goes high Again. The Missed packet % is still in a 5 % and it is a Yellow Alarm in the health status.

Thank you very much. I will keep u updated.

[Diego Armando Cambronero Arias]

I's Seeing the following in the Error Events.

Several messages of:

Inline data bypass has started

Inline data bypass has started

Hundreds of:

transmitPacket: Error TX Queue full, no lost buf yet  if=7

And some of:

A global correlation update failed: Failed to open a TLS connection to HTTP proxy server at 10.1.4.5:8002 : TLS connection failed

Messages, like this one, in the category - Reputation update failure - were logged 24 times in the last 7200 seconds.  name=errUnclassified 

So the IPS is actually doing the Bypass. The Traffic being sent to this IPS 4270 is    MINIMAL.. So i do not understand the reason why it has this problem.. And It is a 4270!!! even if the traffic is huge it must handle it without any problem...that is just imposible with the traffic that Im sending to this IPS it is no tpossible.

Thanks

[Diego Armando Cambronero Arias]

Gest what.. ANOTHER BUG

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtc18038

Cisco cisco cisco

[Diego Armando Cambronero Arias]

However it is fixed in the version 7.0(2)E4

The same that I have.

[Diego Armando Cambronero Arias]

Inspection Load is at 100% Again.

If someone can help me, I would really appreciate it. Maybe someone from the TAC checkout the TOPIC maybe there is something there. Thank you Guys

[Justin Teixeira · Accepted solution]

Hi Diego,

    I would go ahead and open a TAC case at this point for us to take a look at it.

Best Regards,

JT

[Diego Armando Cambronero Arias]

Im going to open de case. I hope that someone from Costa Rica can take a look to this. Thanks