Syslog Collector: Unable to resurrect connection to a subscriber

Answered Question
Jan 19th, 2009
User Badges:

Hi Experts,


I found this logs in SyslogCollector.log.


NMSROOT is C:/PROGRA~2/CSCOpx

propFileC:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\C:\PROGRA~2\CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

Unable to find the file C:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\C:\PROGRA~2\CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

NMSROOT is C:/PROGRA~2/CSCOpx

propFileC:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

SyslogCollector - [Thread: main] INFO , 19 Jan 2009 17:31:31,500, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 19 Jan 2009 17:31:31,500, System Initialized.

SyslogCollector - [Thread: main] WARN , 19 Jan 2009 17:31:37,203, Unable to get the filters for subscriber ciscoworkProd. Default value will be used.

NMSROOT is C:/PROGRA~2/CSCOpx

propFileC:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\C:\PROGRA~2\CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

Unable to find the file C:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\C:\PROGRA~2\CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

NMSROOT is C:/PROGRA~2/CSCOpx

propFileC:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data\Collector.properties

SyslogCollector - [Thread: main] INFO , 19 Jan 2009 17:41:22,093, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 19 Jan 2009 17:41:22,093, System Initialized.

SyslogCollector - [Thread: main] INFO , 19 Jan 2009 17:41:22,171, Subscriber list is empty!


Can the Experts advise me why is the collector properties file is not found? I have checked the NMS root directory, the file is there.


Secondly, how can I re-subscribe the collector? Unsubscribe the collector for troubleshooting purpose.


I really appreciate it greatly if the Experts can show me some light. Thanks a billion!!!


Regards,

Yi Shyuan

Correct Answer by Joe Clarke about 8 years 6 months ago

Your syslog problem probably has to do with your filter settings. Please post a screenshot of your syslog filter settings.


The LMS packet capture tool should capture the syslog messages provided you started it on the correct interface, and your filter was correct (udp port 514).

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Joe Clarke Mon, 01/19/2009 - 09:57
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

There are no problems here. If you need to resubscribe the Analyzer to the Collector go to RME > Tools > Syslog > Syslog Collector Status, and unsubscribe the current Collector, then click the Subscribe button to resubscribe.

jeeyishyuan Mon, 01/19/2009 - 22:55
User Badges:

Hi jclarke,


Is it normal for the SyslogCollector.log to show unable to find collector.properties file?


I have tried to unsubscribe and re-subscribe the collector but the SyslogCollector.log shows that the subscriber list is empty. Due to this error, I have to restore Ciscoworks LMS.


Isn't the path directory is strange to show "C:/PROGRA~2/CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\C:\PROGRA~2\CSCOpx\MDC\tomcat\webapps\rme\WEB-INF\classes\com\cisco\nm\rmeng\csc\data

\Collector.properties"

where the string seems to "restart" the whole string again when it shows up to "classes" folder?


Please advise.


Thanks & Regards,

Yi Shyuan

Joe Clarke Mon, 01/19/2009 - 22:59
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Yes, this error is common, and it means nothing. Don't look at the log files. What do you see in the GUI when you try to unsubscribe and resubscribe the Collector? It would be helpful to see a set of screenshots illustrating what you're doing, and what you see.

jeeyishyuan Mon, 01/19/2009 - 23:24
User Badges:

Hi jclarke,


I had a successful re-subscribe this time. However, I can only receive syslog messages from ASA devices but not from other switches that I intended to receive.


I have installed a sniff packet software on the LMS server and received syslog packets from other switches. However, packet capture in Device Centre did not capture any syslog packets from the targeted switch. I wonder why is it so?


Thanks & Regards,

YS

Correct Answer
Joe Clarke Mon, 01/19/2009 - 23:30
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Your syslog problem probably has to do with your filter settings. Please post a screenshot of your syslog filter settings.


The LMS packet capture tool should capture the syslog messages provided you started it on the correct interface, and your filter was correct (udp port 514).

Joe Clarke Tue, 01/20/2009 - 06:22
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

This is your problem. You have your filter mode set to KEEP, but you only have filters defined for firewall, debugging, and link up/down messages. Either set your mode to DROP, or add additional filters for other messages in which you are interested.

jeeyishyuan Tue, 01/20/2009 - 07:42
User Badges:

Hi jclarke,


So I have configured the filter mode drop and disabled the firewall filters.


I restarted the crmlog aft the I made the changes. But i still dun see any syslog messages from the switches and the update of the syslog.log is not up to the real time. (as compared to kiwi syslog server)


Currently the syslog.log file size is 2947KB while the recommended file size which I found from the log file status report is 1048576KB. So it's not likely that syslog.log has exceeded the file size.


Please advise as I am really running out ideas to troubleshoot.


Thanks & Regards

YS

Joe Clarke Tue, 01/20/2009 - 07:45
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

If you set the mode to drop, and disabled all the filters, then all of the messages will be dropped. I said to EITHER set the mode to drop, OR add more filters. If all your filters are disabled, set the mode to KEEP. If you have filters enabled, and those filters define messages you do NOT want to see, then set the mode to DROP. If you have filters enabled, and those filters define messages you DO want to see, set the mode to KEEP.


You can control the size of syslog.log by configuring logrot.pl. Consult the online help for Common Services on how to configure this. Search for "logrot".

jeeyishyuan Tue, 01/20/2009 - 19:29
User Badges:

Hi jclarke,


I have configured a message filter to have all * to receive any syslog messages from the switches. I tried to restart the CWCS syslog service but it the service seems to hang aft it tried to stop for 1 hour.


Can you advise from here?


Thanks & Regards

YS

Joe Clarke Tue, 01/20/2009 - 21:49
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

What exactly is hanging?

jeeyishyuan Tue, 01/20/2009 - 22:19
User Badges:

Hi jclarke,


CWCS Syslog Service is currently hanging. I can only force the service to rerun by restarting the server.


I have attached a screen shot to show the error if a second time of net stop crmlog is run while the service is hanging.


Thanks & Regards

YS



Joe Clarke Tue, 01/20/2009 - 22:28
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Check the Windows Event View and syslog_debug.log for any errors. But at this point, you will probably need to reboot to continue receiving syslog messages.

jeeyishyuan Tue, 01/20/2009 - 22:47
User Badges:

Hi jclarke,


So I have my server rebooted.


Set a new message filter (shown in the attached screen shot) with asterisk for all columns.


Possible to advise if this filter going to work in order to receive all syslog messages from any switches configured to log to the LMS server?


FYI, I only receive syslog messages from ASA devices at this point of time


Thanks & Regards

YS



Joe Clarke Tue, 01/20/2009 - 22:50
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

This filter will match any message. If you enable this filter, you will need to set your mode to KEEP to receive any messages. That said, you could also disable or remove all filters, set the mode to KEEP, and achieve the same result.

jeeyishyuan Tue, 01/20/2009 - 22:59
User Badges:

Hi jclarke,


This is my current filter setting (attached screen shot).


Based on your last reply, I should be able to receive any syslog messages from switches that configured to log to LMS server. However, I still don't receive any log for that.


Thanks & Regards

YS



Joe Clarke Tue, 01/20/2009 - 23:04
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Are new messages being written to the syslog.log file? What does the Syslog Collector Status page look like?

Joe Clarke Tue, 01/20/2009 - 23:15
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

According to this, new messages are being written to syslog.log. The Collector has forwarded 483 messages to the Analyzer for database insertion since the server was rebooted. Why exactly do you think it's not working?

jeeyishyuan Tue, 01/20/2009 - 23:21
User Badges:

As I don't see any syslog messages from other devices beside the ASA devices in syslog.log


However, I am able to view records for switches in the recent generated 24 hour report.


As I need to achieve all the received syslog messages for audit purpose, I need to make sure that all the syslog messages are received in the log file (if I'm not wrong, syslog.log should be the one).


If possible, you can advise me on the archive portion too?


Thank you very much!!!

Joe Clarke Tue, 01/20/2009 - 23:25
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

The messages must be making it to syslog.log, then. There is no other log file. Messages are first written to syslog.log by crmlog. Then, the SyslogCollector reads the messages from that file, and applies filters. If the messages pass the filters, then they are forwarded to the Analyzer which inserts the messages into the database.


Since you can run reports, and see the desired messages, they must be in syslog.log. You can use the logview command to tail syslog.log in real-time to look at incoming messages. For example:


C:\> logview C:\PROGRA~1\CSCOpx\log\syslog.log

jeeyishyuan Tue, 01/20/2009 - 23:38
User Badges:

Yes, the logview command does show me the log from syslog.log but the logs shown are not real time as well. I have refreshed the syslog collector status a few times, where the number of the received message doesn't seem to increase in short duration as well.


I find this strange as more syslog messages are received by Kiwi Syslog Server.

Joe Clarke Wed, 01/21/2009 - 08:47
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

In order to scale, the crmlog daemon doesn't immediately write the syslog messages it receives to syslog.log. It buffers then, then does periodic flushes, or writes when the buffer becomes full. It depends on the amount of syslog messages being received as to how often it writes out the messages. There are ways to increase this flush period. If you open a TAC service request, those techniques can be explained to you.

jeeyishyuan Wed, 01/21/2009 - 17:39
User Badges:

Thanks jclarke for the info!


I would like to ask for question in archiving Syslog 24 hour report. Should I start a new conversation or continue here?


My question is how can I view the report once it is archived? As I realized that the output file format is not readable using notepad.


Thanks & Regards

YS

Joe Clarke Wed, 01/21/2009 - 18:02
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Syslog messages that are purged from the database, and written to a flat file can be viewed simply by opening the archive files in a text editor/browser. The syslog messages are stored in a format similar to that of syslog.log.

Actions

This Discussion