SyslogCollector Status... missing Collector!

Unanswered Question
Nov 26th, 2007

Hi folks!

I've RME 4.0.5 on LMS suite...

After done a stop/start of SyslogCollector and SyslogAnalyzer processes, the collector is vanished from the Collector Status window.

The processes are up and from the logs seems that they runs correctly but...

In the SyslogAnalyzerUI.log file I found the following error:

[ Mon Nov 26 16:53:45 MET 2007 ],ERROR,[Ajp13Processor[9009][2]],SA0015: Cannot contact SyslogAnalyzer service for up

dating Collector status information

com.cisco.nm.xms.ctm.common.CTMException: URN_NOT_FOUND : urn "SyslogAnalyzerService" : Not found !!

at com.cisco.nm.xms.ctm.client.CTMCall.establishIPC(CTMCall.java:201)

at com.cisco.nm.xms.ctm.client.CTMCall.<init>(CTMCall.java:181)

at com.cisco.nm.xms.ctm.client.CTMClient.invoke(CTMClient.java:253)

at com.cisco.nm.xms.ctm.client.CTMClient.invoke(CTMClient.java:123)

at com.cisco.nm.rmeng.sa.ui.helper.RmeSaCollectorStatus.getCollectorStatus(RmeSaCollectorStatus.java:35)

..............

etc..

So I cannot ADD a new Collector, and no errors give me during the ADD acticvity

Is not a peer server certificate problem, because it's setted correctly.

Another things:

The local DB was restored into another twin server. This twin server was affected with same problem.

Any helps would be appreciated!

Thanks

Leonardo

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3.4 (8 ratings)
Loading.
Joe Clarke Mon, 11/26/2007 - 09:52

There is a bug where the SyslogAnalyzer daemon takes a long time to initialize on Windows if there are a lot of automated actions configured, and lot of devices in those automated actions. So, if the SyslogAnalyzer daemon is running according to pdshow, you may be seeing this problem. The workaround is to wait (or add the IPs of all of your devices to DNS or the local hosts file).

Since you just restarted SyslogAnalyzer, I'm betting this is what you're seeing. The AnalyzerDebug.log with SyslogAnalyzer debugging enabled will confirm it.

Leonardo Roberto Mon, 11/26/2007 - 10:34

Hi joe,

thnaks for your answer!

All devices are into DNS, and I have 33 AA.

I tryed to restart the SyslogAnalyzer and Subscribe the collector again but doesn't appear.No collector can be Subscribed.

Where I have to wait? During Subscribing?

This error appeared into rme_ctm.log:

[ Mon Nov 26 18:07:34 MET 2007 ],INFO ,[Ajp13Processor[9009][1]],com.cisco.nm.xms.ctm.registry.CTMRegistryServer,49, Waiting to read the data ...

[ Mon Nov 26 18:07:34 MET 2007 ],INFO ,[Thread-0],com.cisco.nm.xms.ctm.registry.CTMRegistryHandler,454,Registry = {47006 [tmplmgr-log-RMELogLevelChange] [TemplateCacheMgr]} {47003 [dmserver-RMELogLevelChange] [EssentialsDM]} {47011 [SyslogAnalyzer-RMELogLevelChange]} {47008 [ChangeAudit-RMELogLevelChange] [ChangeAuditService]} {47002 [inventory.ics.server-RMELogLevelChange] [ICS_Inv_Change_Filter] [RMEICServer]} {47005 [NetShowMgr] [Netshow.server-RMELogLevelChange]} {47001 [RMECSTMServer-RMELogLevelChange]} {47004 [archive.service-RMELogLevelChange] [Archive Mgmt]} {47007 [jrmw-log-RMELogLevelChange] [JRM]} {47010 [ogs_server_urn] [rme_ogs_cache_update] [remote_ogs_urn]}

[ Mon Nov 26 18:07:34 MET 2007 ],ERROR,[Ajp13Processor[9009][1]],com.cisco.nm.xms.ctm.client.CTMCall,200,URN_NOT_FOUND : urn "SyslogAnalyzerService" : Not found !!

[ Mon Nov 26 18:07:34 MET 2007 ],ERROR,[Ajp13Processor[9009][1]],com.cisco.nm.xms.ctm.client.CTMCall,209,URN_NOT_FOUND : urn "SyslogAnalyzerService" : Not found !!

Thanks

Leonardo

Joe Clarke Mon, 11/26/2007 - 11:00

This tells me nothing other than the SyslogAnalyzer daemon has not fully initialized. I already mentioned a way to debug SyslogAnalyzer to see why it is not fully initializing. You say that all your devices are in DNS. Have you created PTR records for all of their IP addresses? That is, can you nslookup a device's IP, and get its hostname?

Leonardo Roberto Mon, 11/26/2007 - 11:57

I enabled debug into SyslogAnalyzer and into AnalyzerDebug.

Into AnalyzerDebug file I found a lot of IP address... I have to find also the Hostname?

In the syslog_info file I found the syslog messages with the hostname resolved.

This mean that all devices are reachble from DNS.

Joe Clarke Mon, 11/26/2007 - 12:04

If you are seeing lines such as:

Expanded device list...

Followed by a long list of IPs, then you are hitting this bug. As I said, the workarounds are to wait, or add all of the IP addresses you see on these lines to DNS or local hosts file (by add to DNS, I mean add proper PTR records for these addresses).

Leonardo Roberto Mon, 11/26/2007 - 12:26

in effect... there are a lot of IP Address...

so.. waiting after restart the SyslogAnalyzer doesn't take any effects....

I will upgrade the DNS.

Thank you very much!

Leonardo

Leonardo Roberto Mon, 11/26/2007 - 12:50

Sorry joe...

a last question:

this happened only two days ago.... before it ran correctly.

Into AnalyzerDebug.log, I found a lot of public IP Address that I dind't insert into the DCR before...

Do you know from where the SyslogAnalyzer get the IP Address shown into AnalyzerDebug.log?

Thanks!

Leonardo

Joe Clarke Mon, 11/26/2007 - 12:53

From all the interfaces on all of the devices in the automated action configuration.

Leonardo Roberto Mon, 11/26/2007 - 13:01

ok.. now it's clear

(I tried to disabled all Automated Action, but the problem still occur...)

the only doubt is...

Why the SyslogAnalyzer since two days ago was run correctly, and now ... doesn't work?

Do you know which actions that can be caught this bug?

Thanks

Leonardo

Joe Clarke Mon, 11/26/2007 - 13:29

Only configuring autoamted actions can lead the delay I've described. Since I haven't seen the log, I cannot say if there are any other problems.

Leonardo Roberto Mon, 12/03/2007 - 09:29

Hi Joe,

here you are the logs!

We have waited four hours, and the collector was subscribed again without any own actions.

Why the SyslogAnalyzer need all this time?

We have purged the syslog DB... maybe this action could be the resolution?

Thanks a lot!

Leonardo

Joe Clarke Mon, 12/03/2007 - 10:08

The SyslogAnalyzer needs to reverse resolve all the IP addresses on all interfaces configured in each automated action. On Windows, this can take up to four seconds for each address. This bug is fixed in RME 4.0.6 and RME 4.1.

Leonardo Roberto Wed, 12/05/2007 - 00:36

Thanks joe.

A last question about..

we have two DNS servers named, A and B.

This problem was occur when one of two DNS servers (B) has shutted down.

On the LMS solaris system, the first DNS choice was the running DNS (A).

Can this "shut" cause the problem on SyslogAnalyzer?

I would like to avoid again this problem.

Thanks in advance

Leonardo

Joe Clarke Wed, 12/05/2007 - 00:43

A misconfigured DNS can also cause this problem. Assuming server A had all of the PTR entries, server B should never have been contacted. But, any missing entries would have caused RME to query server B which would lead to timeouts.

You can try removing server B, and see if the situation improves. Or you could upgrade to RME 4.0.6 which has the fix.

Leonardo Roberto Thu, 12/13/2007 - 07:59

Hi joe.

I see that this problem is the bug CSCsh66475.

This bug is referer to windows machine.

Thus this behaviour is the same for solaris and the bug id is valid also for solaris.

Why the bug details doesn't write solaris also?

It has to be update?

We have solaris 8.

My customer would like to know if this bug is valid also for solaris.

He want to justify the update to RME 4.0.6.

Thanks in advance!

Leonardo

Joe Clarke Thu, 12/13/2007 - 08:21

The bug can be valid for Solaris if the DNS is misconfigured. The bottom line is that RME resolves all of the interface IPs. If this takes a long time, then SyslogAnalyzer will take a long time to fully initialize. This typically only affects Windows as Windows name resolution is a bit more involved than Solaris.

In either case, this can be fixed by upgrading to Common Services 3.0.6 and RME 4.0.6.

Actions

This Discussion