Processing Load Percentage = 100

Answered Question
Jul 14th, 2010
User Badges:
  • Bronze, 100 points or more

Hello,


Guys I have been working with 3 IPS 2 4260 and a 4270 since yesterday I have noticed that the Inspection LOAD is RED. On the 4260 the inspection load is.



SJDetec1# sh statistics virtual-sensor | inc Load

         Processing Load Percentage = 99

On the 4270!!!!!!!
DCDetect1# sh statistics virtual-sensor | inc Load
         Processing Load Percentage = 100
DCDetect1# sh statistics analysis-engine
Analysis Engine Statistics
   Number of seconds since service started = 174759
   The rate of TCP connections tracked per second = 0
   The rate of packets per second = 4711
   The rate of bytes per second = 8402
   Receiver Statistics
      Total number of packets processed since reset = 823334516
      Total number of IP packets processed since reset = 822979042
   Transmitter Statistics
      Total number of packets transmitted = 823478816
      Total number of packets denied = 0
      Total number of packets reset = 0
   Fragment Reassembly Unit Statistics
      Number of fragments currently in FRU = 0
      Number of datagrams currently in FRU = 0
   TCP Stream Reassembly Unit Statistics
      TCP streams currently in the embryonic state = 0
      TCP streams currently in the established state = 0
      TCP streams currently in the closing state = 0
      TCP streams currently in the system = 0
      TCP Packets currently queued for reassembly = 0
   The Signature Database Statistics.
      Total nodes active = 16115
      TCP nodes keyed on both IP addresses and both ports = 3438
      UDP nodes keyed on both IP addresses and both ports = 29
      IP nodes keyed on both IP addresses = 1715
   Statistics for Signature Events
      Number of SigEvents since reset = 153308490
For example in the 4270 we are passing almost nothing through the sensor... ANd its working in promiscuos mode. Why is the Inspection Load that High? In the 4260 is the same.. It is working in promiscuos mode..  There are alarms for missed packets as well.
I have been seeing several Discutions for the same reason but none has a fix. The Issue with the inspection load is random. DUring the day sometimes it high and sometimes is low.
Cisco Intrusion Prevention System, Version 7.0(2)E4
Signature Update    S499.0
Any advice will be really appreciated.
Diego.
Correct Answer by Justin Teixeira about 6 years 8 months ago

Hi Diego,

    I would go ahead and open a TAC case at this point for us to take a look at it.


Best Regards,

JT

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Justin Teixeira Wed, 07/14/2010 - 07:03
User Badges:
  • Bronze, 100 points or more

Hi Diego,

    The first thing to try would be to restore all your signatures to default (provided they're tuned), and disable all custom signatures and see if you still experience the problem.  Overtuning of signatures (for example, enabling every signature on the device) is the most likely reason for the processing load being that high.


Best Regards,

JT

Diego Armando C... Wed, 07/14/2010 - 07:10
User Badges:
  • Bronze, 100 points or more

Hi I have been tunning the signatures since the las t months. I have tunned arround 900 signatures. Or at least the action is what I have tunned. If I set them all to their default values I will loose all the work done right?


In the 4270 the traffic passed is very very little. For example. In the last 10 hours an Only have 96 events. So it's nothing for a 4270.


Something else to try?

Christopher Dreier Wed, 07/14/2010 - 08:56
User Badges:
  • Silver, 250 points or more

Diego,


What is your traffic profile? Can you provide packet captures during a time of high processing load along with a show tech? What changed in your network when the issue started?


You are seeing quite a few signature events:


Analysis Engine Statistics
   Number of seconds since  service started = 174759

   ...

   Statistics for  Signature Events

      Number of SigEvents since reset =  153308490

From the limited amount of information we have, it appears that you may be experiencing traffic spikes throughout the day that lead to many events firing which cause the IPS to become oversubscribed. There is quite a bit of data that we will need to examine to validate this. Please open a TAC case and we can begin to investigate.

Thank you,
Blayne Dreier
Cisco TAC IDS Team


**Please check out our Podcast**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
Justin Teixeira Wed, 07/14/2010 - 09:31
User Badges:
  • Bronze, 100 points or more

Hi Diego,

     You can backup your configuration with the copy "current-config" before restoring the signatures to default.  You can then restore them from the backup later.


Best Regards,

JT

Diego Armando C... Wed, 07/14/2010 - 09:39
User Badges:
  • Bronze, 100 points or more

Hi,


Just did it. Let's see if the INpection load goes high Again. The Missed packet % is still in a 5 % and it is a Yellow Alarm in the health status.


Thank you very much. I will keep u updated.

Diego Armando C... Wed, 07/14/2010 - 10:03
User Badges:
  • Bronze, 100 points or more

I's Seeing the following in the Error Events.



Several messages of:



Inline data bypass has started

Inline data bypass has started


Hundreds of:


transmitPacket: Error TX Queue full, no lost buf yet  if=7




And some of:


A global correlation update failed: Failed to open a TLS connection to HTTP proxy server at 10.1.4.5:8002 : TLS connection failed

Messages, like this one, in the category - Reputation update failure - were logged 24 times in the last 7200 seconds.  name=errUnclassified 

So the IPS is actually doing the Bypass. The Traffic being sent to this IPS 4270 is    MINIMAL.. So i do not understand the reason why it has this problem.. And It is a 4270!!! even if the traffic is huge it must handle it without any problem...that is just imposible with the traffic that Im sending to this IPS it is no tpossible.
Thanks
Diego Armando C... Wed, 07/14/2010 - 10:18
User Badges:
  • Bronze, 100 points or more

Inspection Load is at 100% Again.


If someone can help me, I would really appreciate it. Maybe someone from the TAC checkout the TOPIC maybe there is something there. Thank you Guys

Attachment: 
Correct Answer
Justin Teixeira Wed, 07/14/2010 - 10:24
User Badges:
  • Bronze, 100 points or more

Hi Diego,

    I would go ahead and open a TAC case at this point for us to take a look at it.


Best Regards,

JT

Diego Armando C... Wed, 07/14/2010 - 10:52
User Badges:
  • Bronze, 100 points or more

Im going to open de case. I hope that someone from Costa Rica can take a look to this. Thanks

Actions

This Discussion