cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1562
Views
0
Helpful
20
Replies

LMS2.6 - DFM alarm notification get's delayed

ptrigueira
Level 1
Level 1

Hi to all,

We have DFM configured to send trap notifications towards an external machine, and we see that sometimes the alarms that arrives to the external NMS are delayed. some o them are sent only days after it were received in CW .

Any clue ??... correlation issues maybe ?

Cheers,

Pedro

1 Accepted Solution

Accepted Solutions

The attachment didn't come through, but I'm not sure how useful it would have been. It sounds like he sent a screenshot of a sniffer trace which isn't very helpful. We would need to see the binary sniffer trace file. In any event, I think I found an example in the log, and it looks like address resolution might be the culprit.

Basically, DFM takes the IP address that is configured and performs a lookup on the address. So, DFM tries to use the system resolver to resolve 192.168.168.134. This takes about three seconds to complete. The way Windows address resolution works can cause excessive delays when performing these kind of operations. As a workaround, try adding this address to DNS or your server's hosts file, and see if this problem improves.

View solution in original post

20 Replies 20

Joe Clarke
Cisco Employee
Cisco Employee

I have not heard of any delay problems with sending trap notifications. However, there are a few problems where notifications are lost. Are you sure the notifications showing up on the external NMS are actually delayed, or is it a new notification that is showing up? How have you configured your trap notification subscription, and what do one of these delayed notifications look like?

I'm sure it's the same notification cause the id is the same ...i compare them, i mean, the alarm that appears on DFM and the alarm that appears later on external nms...

P

What about the answers to my other two questions?

Something you should try to debug this is to place a sniffer on the LMS server, and filter on UDP 162 traffic to your external NMS. See if the traps are leaving the LMS server in a timely fashion. If not, you can enable Notification Services debugging under DFM > Configuration > Other Configurations > Logging, reproduce this problem, then check the NMSROOT/log/dfmLogs/NOS/nos.log for errors.

My DFM (lms2.6; dmf 2.0.8) email notification doesn't work automatically. If i click on notify and fill in the details , the email comes through. But if i manually generate an alarm , the NOS.log file says that it sees it, and its preparing to send but then it doesn't.

28-Feb-2007|09:39:52.453|DEBUG|NOS|Email Notify Thread:Pooled Thread:1|EmailUtility|createAndSendMail()|.|Ready to send

28-Feb-2007|09:39:52.453|ERROR|NOS|Email Notify Thread:Pooled Thread:1|EmailUtility|createAndSendMail()|.|

Mail was not sent to ; pierre.vanvuuren@za.didata.com

Hi Pierre,

Is the NOS Server running? Check this by searching for "Common Services -> Server -> Admin -> Processes" the NOSServer process. Is it running?

With DFM 2.0.4 i had an issue with the NOSServer registration.

Navigate to "Common Services -> Server -> Admin -> Processes"

Click on the entry "NOSServer"

The following INformation should appear for 2.0.8:

Process: NOSServer

Path: /opt/CSCOpx/bin/cwjava

Flags: -server -cp:p MDC\tomcat\webapps\triveni\WEB-INF\lib\log4j.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\nos.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\ctm.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-client.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\cogs.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-kc1.0.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-kilnervirtualasa1.0.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\activation.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\mail.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\tis-client.jar,

MDC\tomcat\webapps\triveni\WEB-INF\lib\pm.jar,

lib\classpath\jconn2.jar,lib\classpath\epm-v0-5-0.jar,

lib\classpath\cesar-v0-5-0b.jar,

MDC\tomcat\shared\lib\NATIVE.jar,

MDC\tomcat\shared\lib\MICE.jar,

MDC\bin,objects\nos\config,MDC\tomcat\webapps\triveni\WEB-INF\classes,

-cw:jre D:\PROGRA~1\CSCOpx\MDC\jre -cw:xrs -Djava.compiler=NONE com.cisco.nm.trx.nos.server.NOSServer

Startup: Started automatically at boot.

Dependencies: EPMDbEngine EPMServer INVDbEngine PMServer DFMOGSServer

If there is a difference you might try tu unregister and register NOSServer.

Windows

@ECHO OFF

SET NMSROOT=c:\CSCOpx

CALL %NMSROOT%\bin\pdreg.cmd -u NOSServer

CALL %NMSROOT%\bin\pdreg.cmd -r NOSServer -d "EPMDbEngine,EPMServer,INVDbEngine,PMServer,DFMOGSServer" -e %NMSROOT%\bin\cwjava.exe -f "-server^-cp:p^MDC\tomcat\webapps\triveni\WEB-INF\lib\log4j.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\nos.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\ctm.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-client.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\cogs.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-kc1.0.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\ogs-kilnervirtualasa1.0.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\activation.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\mail.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\tis-client.jar,MDC\tomcat\webapps\triveni\WEB-INF\lib\pm.jar,lib\classpath\jconn2.jar,lib\classpath\epm-v0-5-0.jar,lib\classpath\cesar-v0-5-0b.jar,MDC\tomcat\shared\lib\NATIVE.jar,MDC\tomcat\shared\lib\MICE.jar,MDC\bin,objects\nos\config,MDC\tomcat\webapps\triveni\WEB-INF\classes,^-cw:jre D:\PROGRA~1\CSCOpx\MDC\jre^-cw:xrs^-Djava.compiler=NONE com.cisco.nm.trx.nos.server.NOSServer"

Hope that helps a bit. Ohterwise help us an provide a bit more details about your setup (Notification Groups and settings).

Best regards,

Frank

Stick a sniffer on the LMS server, and filter for TCP port 25 to your DFM SMTP server. That trace may provide a reason why the mail is not being sent automatically.

Hi JC,

Can you hepl me out here ... What "function module" should y raise to debug level for the delay problem we are having with notification been delayed.

partial nos.log attached...

I don't see any problems with delay on the DFM side. I see an alert come in at 10:08:34, and goes out as a trap at 10:08:50. As I said previously, you should start deploying sniffers to watch the trap move from the DFM server to the external NMS.

Hi JC,

you're right, and we aredoing that. The problem only occurs when we he have instability in the network where a bunch of alarms are sent to the CW.

The attached NOS.log is not in debug but you can see the ammount of messages in the same second ...

There is bug retrieving event details that will be worked out in LMS 3.0, but it shouldn't delay the sending of a notification trap. Without a debug showing the problem, or a sniffer trace showing where in the network the delay is occurring, I cannot offer any other suggestions.

Hi JC,

About the delay, every time the system is NOT loaded we see a regular flow of traps being sent from CW to the notification server ( i.e. several packets on the same second), BUT when high load is seen, the flow of traps towards NServer lowers to one trap every 4 seconds...

How loaded is loaded? Four seconds could be expected under high load situations. What are the specs on this server?

Hi JC,

Loaded means thousands of traps recevived on a certain period of time. The Server is a DL360 and the proccessor load doesn't goes more than 40%. I do believe that it is queueing delay, as that if i receive let's say 10 traps one the same second and i'm sending notifications for those traps one every 4 second's ...with time i will have delay.

I can say that at the moment, when we see load on the DFM, we experience more than 40 minutes delay from a alarm apearing on DFM and being sent to notification server...

P

Hi JC,

Loaded means thousands of traps recevived on a certain period of time. The Server is a DL360 and the proccessor load doesn't goes more than 40%. I do believe that it is queueing delay, as that if i receive let's say 10 traps one the same second and i'm sending notifications for those traps one every 4 second's ...with time i will have delay.

I can say that at the moment, when we see load on the DFM, we experience more than 40 minutes delay from a alarm apearing on DFM and being sent to notification server...

P

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: