100% Sensor load during some period

Muhammad Anser Khan · ‎10-24-2009

Dear ALL,

Issue: IDSM is showing 100% Inspection(Sensor) Load during certain periods daily which causes more than 2000 ms delay in the network.This issue occurs after upgrading the signature engine to E3

We have also monitored the traffic during 100% Load on PRTG but traffic has still in normal utilization 40 to 50Mb.

I have found the same 100% Load issue on the Cisco TAC case.

Has any one faced this issue before in their workaround. Actually we are not sure about our issue. Is our issue also related to SMTP traffic or not as in the TAC case.

How can we identify our issue?

Please see the following TAC case:

Symptom:

IDSM is showing high CPU and a "processing load percentage" of 100 during certain periods daily. Traffic is affected at those times.

Conditions:

Issue has been identified to be linked to smtp traffic.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsz81580&from=summary

How can we solve/identify this issue?

Please provide us the appropriate solution for this issue.

Here are the details of IDSM-2:

Our IDSM is in Inline VLAN-Pair Mode:

Inline TCP Tracking Mode: Interface and VLAN

Core Switch IPS Etherchannel Setup:

-----------------------------------

Group 5: IDSM(A) and IDSM(B) Port x/7

Group 6: IDSM(A) and IDSM(B) Port x/8

Some VLAN Pair(s) are on interface x/7 and others are on x/8

There is an FWSM module also, which acts as the default gateway for all internal VLANs.

All signatures are in default state.

IPS1 version Detail:

Cisco Intrusion Prevention System, Version 6.1(2)E3

Signature Update S440.0

OS Version:2.4.30-IDS-smp-bigphys

Platform:WS-SVC-IDSM-2

Cisco6513 IOS: Version 12.2(18)SXF17

Thanks in advance.

Regards,

Anser

rhermes · ‎10-26-2009

You don't say how much traffic you are trying to push through that IDSM, but if you overload it, you will get delay and lost packets.

You can try putting the sensor into promisicous mode and doing shunning on the FWSM. That way, no matter how badly the sensor performs, you will not distrupt traffic.

Muhammad Anser Khan · ‎10-26-2009

This is not a traffic issue. Traffic shows very low during 100% sensor load.

Regards,

Anser

bnidacoc · ‎11-03-2009

I get "Red" status events on "inspectionLoad" very frequently. I am not quite sure if it has always been this frequent. I wonder if the "healthAndSecurity" status alerts can be sent off to a syslog server for long term storage, review and analysis.

Are you getting "Red" status events on "inspectionLoad"? Just recently or all the time.

There is another thread in this forum about something like highcpu or something like that. I am not sure if that one was very specific.

I wonder if this is related to Cisco updating signature files with new sigs are enabled by default related to 5 year old vulnerabilities.

evStatus: eventId=REMOVED

vendor=Cisco

originator:

hostId: REMOVED

appName: monitor

appInstanceId: 345

time: Nov 03, 2009 19:08:51 UTC offset=-300 timeZone=EST

healthAndSecurity:

description: Health and security status

healthStatus: red

securityStatus:

virtualSensor: vs0

status: green

changed:

metricValue: name=inspectionLoad

current:

value: 94

status: red

status: green

thresholds:

type: upper

yellow: 80

red: 91

warning:

metricStatus: name=inspectionLoad

status: red

andrey.dugin · ‎11-06-2009

May be there are lot of small packets going through the sensor.

Activate flood-signatures and analyze alerts.