CS-MARS how to verify the actual load

r.spiandorello · ‎02-28-2007

Hi, Cisco has a target performance for every CS-MARS model, but how to verify the CS-MARS load (events/sec, flow/sec) in a production phase ? is there a CLI command or a report ?

thank you in advance

rs

pmccubbin · ‎02-28-2007

I'm not aware of a CLI command or a report to check the load on the MARS box so I always do the math in my head based upon figures provided on the dashboard.

I'm currently working on a MARS 100 that is doing close to a 1 billion netflow events every day. It sounded high until we did the math:

60 seconds X 60 minutes = 3600 per hour

3600 X 24 hours = 86400 per day

1,000,000,000 / 86400 = 11,574

The spec reads 150,000 per day. You also want to have the box do no more than 60% of the spec as this extra allows for continued operation in the event of an target that creates a ton of events.

Hope this helps.

mhellman · ‎02-28-2007

Getting EPS and FPS averages is easy enough, but pretty useless. It's the peaks that matter. I'm not aware of any way to get out of CSMARS good numbers on EPS or FPS. When we were having performance problems, Cisco provided us with custom binaries that added extra performance-related logging details. You might ask them to do the same.

pmccubbin · ‎02-28-2007

Matthew,

Thanks for your spot-on response. You bring up an excellent point that there is no way either from the CLI or the GUI to tell when MARS itself is having performance issues. It basically has to come ot the point where you are simply scratching your head at the information it is producing and say to yourself that something is wrong.

We have opened a TAC case regarding performance related issues we have begun to see after the introduction of 7 IPS devices into our network. Our MARS 100 is currently in spec with regard to Netflow and Event per second but has been producing corrupted data and we are wondering if it is not being overwhelmed by the traffic being sent from the IPS, in particular, regarding Bit Torrant.

The problem has been that there have been no signals from MARS that it is dropping anything, only that it is starting to report things that can't be happening.

I'll post more details when we get it all sorted out. I'll also be asking them about giving us the custom binaries that you wrote about that add extra performance related logging details.

All and all it sounds like we are building a case for a product enhancement to MARS.

mhellman · ‎02-28-2007

A couple thoughts, sorry for the length.

1) CSMARS does try to notify you when it drops events. Unfortunately, the rule is incomplete.

Modify the "System Rule: Resource Issue: CS-MARS" and make sure it has all the following event types:

MARS Dropped Event Since Capacity is Reached - First Event in the Hour,

MARS Dropped Netflow Record Since Capacity is Reached - First Netflow Record in the Hour,

MARS Dropped Event Since Capacity is Reached - Drop Count in the Hour,

Session utilization has reached certain percent of the system capacity,

MARS Dropped Netflow Record Since Capacity is Reached - First Netflow Record in the Hour,

MARS Dropped Netflow Record Since Capacity is Reached - Drop Count in the Hour

2) you're definitely right about IPS being one of the report device types that could throw everything out of whack from a performance expectation standpoint. How do you compare a simple event like a syslog message to an IPS alarm that includes a trigger packet? I certainly would expect that processing the alarms takes more resources.

3) this may not be your issie, but there was a major bug in the way IPS events were sessionized in some versions of csmars. the session tcpip 5-tuple was incorrect and the raw events tied to it were totally unrelated. This was fixed in 4.2.3. see:

http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCsg49227

4) the new cips process appears to still have some issues. I see ours puking in the logs all the time.