MARS, outbound firewall rulesets, and Deny Packet Due to Security Policy

[clausonna]

Hi folks.

I have one ASA firewall that services 2000+ users spread out over dozens of small/remote offices. The inside-to-outside ruleset permits certain traffic (http/s, DNS, other 'normal' ports), and but has a "deny any any" at the end. I think this is fairly normal in a corporate environment, otherwise you'd have all sorts of nastiness (p2p for example) transiting your firewall and internet circuits.

The problem is that the 'deny any any' syslogs that get forwarded to MARS generate a TON of events, all GREEN level (since the packets were denied), all based off of 'Deny packet due to security Policy'.

The rules that fire include:

System Rule: Network Errors - Likely Routing Related

System Rule: Configuration Issue - Firewall

System Rule: Network Activity - Excessive Denies - Host Compromise Likely

System Rule: Network Activity - P2P

Keeping up with the volume of MARS events was proving to difficult, so I created a report that runs every hour, aggregating the above events. I used Custom Columns, with Source IP/port, Destination IP/port/protocol, Reporting Device, and Time Range. This does a great job of showing me which devices are generating the most denies on the firewall, and therefore the most events/incidents in MARS.

I've been able to isolate some of the traffic to peer-to-peer apps running on corporate laptops (tcp port 6346 is a dead giveaway), which I solved with corporate policy and not technology per-se.

However, I'm still left with a slew of events that clog up MARS. Sometimes its a laptop that must have just come from a client/project site, and they were configured to use/print to one of their servers/printers. Other times I can research the destination ports and/or IP addresses and determine what the app was that created the traffic (e.g. although IM ports are not enabled by default, the IM apps (AOL, Yahoo, etc) always do their standard checks first before defaulting to 80 or 443 (sic).)

My question is: how does everyone else deal with this? I could literally spend all day researching why IP address ABC sent traffic XYZ to some random destination. There's definite value to this (again, p2p, plus I've caught a number of systems infected with viruses, and even some of my own servers that were doing funky (but not malicious) things, etc.)

Am I doomed to forever play MARS 'whack-a-mole', or is there a better way to reduce the volume of incidents that appear?

I'd be really interested to hear how others are keeping ahead of MARS when it floods them with events.

I should also note that, as a test, I disabled the aforementioned Rules that were creating all of the incidents. This just caused other (Red level) MARS rules to fire (Excessive denies, DoS attacks, etc.)

[michaelwoolfe]

clausonna,

I am currently using a MARS 200 that is receiving thousands of even notifications per second. I am experiencing the same situation that you are with the 'whack-a-mole' battle.

But there is another way to look at this situation. You can obviously take the approach of 'stop the flood of information', but truthfully that is why you have MARS - to gather all the information together and then present it to you. Now the goal is to have some computing and intelligence applied to the information to save you time and resources.

So here is what I do. I take two main pieces of information from MARS - trends and top X violators. I use the trends to tweak anti-virus, IPS systems, Firewalls, routing architectures, denied programs via Group Policy, etc. to try to mitigate the source of the traffic. I also have reports that order the top violators, so I can spend my time on the best bang. When I get the network working harmoniously together, I am going to start creating a “self-defending” or “self-healing” network.

Another methodology to use is to relay your notifications through a syslog relay. This can let you filter out certain notifications and then send the rest to MARS. But if you have MARS pulling the

[mhellman]

I would recommend really looking at the individual inspection rules and deciding whether they should be enabled/tuned. The real challenge is that event normalization and aggregation can be a double-edged sword.

For example:

System Rule: Network Errors - Likely Routing Related

By default, this rule fires upon seeing 10 events in the following event type categories:

FirewallPolicyViolation/ACL

FirewallPolicyViolation/NAT

ConfigError/Network

This rule is probably firing mostly because of ACL denies. In my environement, 10 ACL denies is not sufficient for creating an incident. I could just increase the count, but if you look at that event category in management->event management it has all sorts of other events lumped in ("CheckPoint device detected IKE Enforcement Violation", etc). So maybe just increasing the count isn't the right answer. In this example, you might consider adding a "!= Deny packet due to security policy" so that it doesn't fire on ACL denies but does fire on everything else. We use a lot of "!=" in the inspection rules to remove the "noise" without having to increase the event count for the whole event type category. If necessary (probably not in this case because "Deny packet due to security policy" occurs in all sorts of other inspection rules, you can then create a separate inspection rule for "Deny packet due to security policy" with a higher count.

FWIW, it's a constant struggle for us as well. It's really an art to get the inspection rules tuned for your environment.