E-Mail into the Void - Looking for alert logs

crossmanj · ‎07-31-2001

Twice now, I have had event severity 11 (custom) detects occur just after 1:00pm and the e-mails have never been received. The system seems to work happily all of the rest of the time with no manual correction necessary. Any help on where to start the search for the logs is appreciated. I checked mail as the netranger ID on the director (eventd is 2.2.1) with no mention of bounced mail. Thanks in advance.

marcabal · ‎08-06-2001

Sorry for the late reply.

I responded last week, but it appears that my response never made it to the server.

Does the email of events work normally most of the time?

Are you sure that the director is configured to send you those emails?

Eventd is configured by default to only send one email every hour, so if these weren't the first alarms then they wouldn't have generated an email.

The tokens to check would be EventAlarmInterval (1 hour by default) and EventAlarmThreshold (1 by default?) in the /usr/nr/etc/eventd.conf.

One other thing you could do to help diagnose the problem is add the following lines to the top of the /usr/nr/bin/eventd/event script:

date >> /usr/nr/var/event.messages

echo "$@" >> /usr/nr/var/event.messages

Every time the script runs the current date and time as well as the parameters passed to the script will be printed in the /usr/nr/var/event.messages file.

Then when you would have expected an email you can check the file and see if the script was executed. If there is an entry in the event.messages file then you know to check the mailing system for problems. But if there is not an entry then you know to check the ids config files.

Another place you may want to check is the email for root on that box. Occasionally the emails will be sent to root on that box if there is a misconfiguration somewhere.

crossmanj · ‎08-21-2001

OK, back to this.... (exciting times for us in security these days, eh?)

E-Mail works normally from the director to my mail server all of the time. The director is configured to send me the e-mails - it just only sends SOME of the mail! I checked the eventd settings, and I have it set to send the first 5 events (EventAlarmThreshold 1,2,3,4,5) within any given 2-minute period (EventAlarmInterval 120) according to the docs.

I added the lines to the event script (nice thinking, btw!) and did a tail -f to the file. I have two protected environments, A and B. I have a Senso on the outside of each environment, and another on the inside. On the outside sensors, I have UniCode attacks and CMD.EXE access set to alarm level 11. I send a usual Unicode attack from my location on the outside of A to B, so it passes three sensors. All three sensors see the attack, all three sensors flag four alarms on the OVW console (Unicode, CMD.EXE, dot-dot-denial and dot-dot-execute), this will all go to my database ok. So the sensors see it.

For each attempt at this test, the event.messages file shows between one and three events being handled by the /usr/nr/bin/eventd/event script. Yes, I am waiting plenty of time (~15 minutes) between tests. I'd say it looks like the EventAlarmThreshold token, but the number of events is variable. And if the 1-3 events processed are event level 4 from the A sensor (I get OUT-to-IN event level 4 and 5 events to my desk, the NOC gets 9,10, and 11 events) then nothing is e-mailed to anyone (since A sees OUT-to-OUT traffic).

(Oh yeah, I made the event level 11 mail alerts triggered in EITHER direction for now, still no go.)

Whew? Does that make any sense? It seemed to work perfectly when it was first set up, but has become more and more unreliable over the last month or so. Services have been bounced all over with no effect.

Any suggestions are appreciated.....

crossmanj · ‎08-21-2001

OOps! I forgot to mention that if the event script gets the alarm 11 detects, the mailed alert is fine. It's just that in this configuration, I am getting a successful alert only 1-2 times out of ten tries.

Also, my comment about growing unreliability, is in itself an unreliable statement. I have no data, just an impression that when it was first set up, my database of alarm 11 events seemed to math the e-mailed alarms almost perfectly.

marcabal · ‎08-21-2001

Below is a common explanation for many of the problems we have had with eventd. It may help explain the issues that you are seeing or you may have found a bug that we haven't seen. If this explanation doesn't apply to your situation then we will need additional information form you to help diagnose the problem.

We would need a copy of the following files:

1) /usr/nr/etc/destinations

2) /usr/nr/etc/smid.conf

3) /usr/nr/etc/eventd.conf

4) /usr/nr/bin/eventd/event.conf

5) The output from the debug commands you added to the event script.

6) The log file from the director containing all alarms which the director received during the 2 minute interval when the emails should have been generated.

You can email them directly to my cisco email account rather than posting to the Forum.

We can then try to replicate it in our test lab.

Common explanation for issues with eventd:

----------------------------------------------------------------------

There has been some confusion as to how eventd will handle the thresholds. Some users have incorrectly assumed that the eventd thresholds are per alarm and per sensor. When in fact the thresholds are independent of the individual alarms and sensors. The thresholds are, if I remember correctly, global for any message (COMMANDS and ERRORS can also cause scripts to be triggered, so they are included with the Alarm EVENTS) which causes a script execution.

So in a simple scenario where the user has setup the eventd program to execute the event script for alarms level 4 or higher and the EventAlarmThreshold is set to 1,2,3,4,5 and the EventAlarmInterval is set to 120. If the user receives 3 level 4 alarms from 1 sensor followed by 5 level 11 alarms from a second sensor in a 2 minute interval, then the event script will only be executed for the 3 level 4 alarms and the first 2 level 11 alarms instead of all 5 level 11 alarms.

Another issue that can result is that just because the event script get's executed it does not mean that an email will be generated. The event script has to be configured using the event.conf file to send the email for specific alarms and traffic directions. So if I have the event.conf file setup so that only level 11 alarms generate emails then with the above example I will only get the 2 emails from the first 2 level 11 alarms and no emails for the level 4 alarms. This has lead some users to believe that there were errors in the eventd program.

crossmanj · ‎08-23-2001

Many thanks for the eventd data. Since I had to modify the event script, I had learned much of what you had written by watching the activities of the script. While I had understood the event thresholds were for all alerts across all probes, I had not realized that what is passed to the event script is such a superset of what I was looking for. For instance, all OUT-to-IN alarm level 4 and 5's are sent to my e-mail only (since I want the console running in the NOC). What I hadn't realized is that all event 4's and 5's are sent to the script and therefore have the thresholds apply, even though only one in several may be a OUT-to-IN (due to unfiltered false positives caused by testing of new Network Management utilities).

I will test again, and if I have further problems, then I will take this off-line with you. Thank you so very much for helping me out with this problem.