Missed Packet Percentage

Solobone22 · ‎10-22-2008

Hello All,

We run several IDS/IPS in our clients network and for the most part they run very well. The IDSM-2 modules however experience a 10-30% missed packet percentage during heavy loads and it seems as though it never recovers. My workaround is to disable a signature then immediatly re-enable that same signature. You lose the sensor for a minute but after the engine reconfigures itself the mpp issue is resolved. Until another period of heavy traffic. I have left it run missing 23% for 24 hours and the sensor did not recover. But after the above workaround it ran fine until the next day when it got slammed again. I have tried reducing the traffic that the sensor sees and it didn't seem to help. Anybody else had this or a similar problem?

Thx,

Dave

attmidsteam · ‎10-23-2008

We've seen poor performance across the board on many of the sensor models but I don't think that is the issue here. Our issues were purely based upon the traffic mix but your issue goes away if you restart the analysis engine which leads me to believe the sensor is struggling with too many open or half-open sessions (is traffic asynchronous in your network)? What do the stats look like for the normalizer engine? 'sh stat virtual-sensor' Are you running 5.1, 6.0, or 6.1?

Solobone22 · ‎10-23-2008

Attached is the sh st virt. We are running 6.1(1)E2. Checking on the async/sync. Thx for the reply.

attmidsteam · ‎10-23-2008

Doesn't look like a normalizer issue to me and the traffic rate doesn't seem very high. I would open a TAC, what version of code are you running on it?

Solobone22 · ‎10-23-2008

Hmm.....we are running 2.4.30-IDS-smp-bigphys. I had opened a TAC and they said it was spanning too many vlans. We reduced the number of vlans but still the problem is there. Seeing if anyone else has had similar problems and what they did to remedy the problem.

attmidsteam · ‎10-23-2008

what does 'sh ver' say for the major/minor version? spanning too many vlans? the traffic volume doesn't seem too high, they may have been grasping for straws

Solobone22 · ‎10-24-2008

XXXXXXXXXXXXXX# sh ver

Application Partition: Cisco Intrusion Prevention System, Version 6.1(1)E2 Host: Realm Keys key1.0 Signature Definition: Signature Update S363.0 2008-10-23 Virus Update V1.4 2007-03-02 OS Version: 2.4.30-IDS-smp-bigphys Platform: WS-SVC-IDSM-2 Serial Number: SAD104602Y0 Licensed, expires: 01-Oct-2009 UTC Sensor up-time is 9 days. Using 1403949056 out of 1983504384 bytes of available memory (70% usage) system is using 17.7M out of 29.0M bytes of available disk space (61% usage)application-data is using 36.1M out of 166.8M bytes of available disk space (23% usage)boot is using 40.5M out of 68.6M bytes of available disk space (62% usage)application-log is using 530.1M out of 2.8G bytes of available disk space (20% usage) MainApp M-2008_APR_24_19_16 (Release) 2008-04-24T19:49:05-0500 Running AnalysisEngine ME-2008_JUN_05_18_26 (Release) 2008-06-05T18:55:02-0500 Running CLI M-2008_APR_24_19_16 (Release) 2008-04-24T19:49:05-0500 Upgrade History: * IPS-sig-S361-req-E2 19:00:06 UTC Tue Oct 14 2008 IPS-sig-S363-req-E2.pkg 19:00:22 UTC Thu Oct 23 2008 Maintenance Partition Version 2.1(2) Recovery Partition Version 1.1 - 6.1(1)E1 Host Certificate Valid from: 18-Jul-2007 to 18-Jul-2009

Thx for the help on this.

Dave

attmidsteam · ‎10-27-2008

I guess I forgot the obvious question but how much traffic is this sensor seeing? It could simply be a performance issue (we have a long history of seeing sensors not performing to spec).

Solobone22 · ‎10-29-2008

The sensor does see quite a bit of traffic and during peak periods I can see it getting a bit stressed out. I still would like to know why it never seems to recover after the peak......