DFM upgrade to 2.0.10 - email notification stopped working

Answered Question
Mar 11th, 2008
User Badges:

After we completed the DFM upgrade from 2.0.6 to 2.0.10 on our LMS 2.6 bundle, the email notifications just stopped working. Fault History still shows the events or alerts are coming in but no emails are sent out.


Appreciate any help.

Correct Answer by Joe Clarke about 9 years 1 month ago

Nevermind again. I found a bug. I can create a test patch for you to try out. You will need to open a TAC service request to get it, though. Have your engineer contact me directly for the patch.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Joe Clarke Tue, 03/11/2008 - 09:03
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Please post a screenshot of your Notification Group configuration, the details of a new event which has been generated after the upgrade, and the output of the pdshow command.

ibc-tcs Tue, 03/11/2008 - 09:28
User Badges:

I have attached the Notification Group config and one sample event (OperationallyDown port) on my test switch. I am not too sure about where to get the output of the pdshow, I just attached the running services. I apologize if that's not the info you needed. Please let me know how to get it.

Thanks again.



Attachment: 
Joe Clarke Tue, 03/11/2008 - 11:17
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I don't see any errors per se, but I cannot confirm that Event Set A has the OperationallyDown even selected nor that the device 10.109.128.254 is selected for notifications. Verify that those things are true. If so, post the NOSServer.log and NMSROOT\log\dfmLogs\NOS\nos.log.

ibc-tcs Tue, 03/11/2008 - 11:49
User Badges:

I verified and both are true. Hoping to find out more info, I enabled "debug" in logging and then simulated the OperationallyDown event. Maybe you can find other details there. Also, I manually tested sending email by selecting the "Notify" in the Alert and Activities window. I did get the email so it appears only the auto email is not working.

Attached are the logs. Thanks again.



Attachment: 
Joe Clarke Tue, 03/11/2008 - 11:56
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I saw this once before where the EPM database becomes out of sync with itself. New events never trigger notifications because the event IDs are not what is expected. Try reinitializing the EPM database alone. You will lose all your current events, but notification and fault history will remain:


NMSROOT\bin\perl NMSROOT\bin\dbRestoreOrig.pl dsn=dfmEpm dmprefix=EPM

ibc-tcs Tue, 03/11/2008 - 12:38
User Badges:

Here's what I did as advised:


Stopped the Ciscowork Daemon Manager

C:\Program Files\CSCOpx\bin>net stop crmdmgtd

The CiscoWorks Daemon Manager service is stopping...............................

.........................................................

The CiscoWorks Daemon Manager service was stopped successfully.


Re-initialize EPM

C:\Program Files\CSCOpx\bin>perl dbRestoreOrig.pl dsn=dfmEpm dmprefix=EPM


WARNING: Existing contents of dfmEpm database will be lost.

Do you want to continue [y/n]?y

dfmEpm database initialization is completed.



C:\Program Files\CSCOpx\bin>net start crmdmgtd

The CiscoWorks Daemon Manager service is starting.

The CiscoWorks Daemon Manager service was started successfully.


Re-booted the server...


Simulated the event. The Alert and Activities panel logged the alert/event but same problem - no auto email..

Joe Clarke Tue, 03/11/2008 - 12:55
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

I presume you left debugging on? If so, please post the new nos.log.

ibc-tcs Tue, 03/11/2008 - 13:06
User Badges:

No not really. Anyway I re-enabled debug and simulated the event again. You can look at the attached NOS log beginng at 11-March-2008-15:58:05.

Thanks.



Attachment: 
Joe Clarke Tue, 03/11/2008 - 13:13
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

This doesn't help. I need to see the log from when NOSServer is first started. Keep debugging enabled, then stop dmgtd. Clear out nos.log, then restart. Then simulate the event, and post the new nos.log.

ibc-tcs Tue, 03/11/2008 - 14:00
User Badges:

After the log was cleared and the server restarted, a strange thing happended - I suddenly got 3 email notifications(?).

To verify, I simulated the event again but no email came.... I thought for a momemnt it's working again.


Anyway, here's the log.



Attachment: 
Joe Clarke Tue, 03/11/2008 - 14:28
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

According to this, email should now be working:


11-Mar-2008|16:41:16.921|DEBUG|NOS|Email Notify Thread:Pooled Thread:1|EmailUtility|createAndSendMail()|.|

Mail was sent successfully to [email protected]


I suggest you check your SMTP server (10.108.51.42) to make sure it is relaying this mail. It may also be getting filtered in SPAM. The contents should be:


EVENT ID = 00000RW

ALERT ID = 00000RT

TIME = Tue 11-Mar-2008 15:59:12 EDT

STATUS = Active

SEVERITY = Critical

MANAGED OBJECT = 10.109.128.254

MANAGED OBJECT TYPE = Switches and Hubs

EVENT DESCRIPTION = OperationallyDown::Component=PORT-10.109.128.254/0.2 [Fa0/2];Type=ETHERNETCSMACD;OperStatus=DOWN;InterfaceCode=CODEUNKNOWN;DuplexMode=FULLDUPLEX;AdminStatus=UP;LastChangedAt=11-Mar-2008 3:59:12 PM;IsFlapping=false;MaxSpeed=100000000;Mode=NORMAL;

CUSTOMER IDENTIFICATION = IBC

CUSTOMER REVISION = 3.0

Joe Clarke Tue, 03/11/2008 - 14:29
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Nevermind. I just reread what you typed. When did you simulate this latest event?

Correct Answer
Joe Clarke Tue, 03/11/2008 - 14:40
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Nevermind again. I found a bug. I can create a test patch for you to try out. You will need to open a TAC service request to get it, though. Have your engineer contact me directly for the patch.

ibc-tcs Wed, 03/12/2008 - 06:14
User Badges:

Will do. Many thanks again for all the help.


Best regards,

Gerry Romero

Insurance Bureau of Canada

Actions

This Discussion