Cisco works 2.6 syslog problem

[paul2_uk]

Hi

I have a problem with LMS 2.6 with syslog, whenever a device sends a syslog message to the LMS server it does not register with the server for about 30 mins yet if I get the switch to send to another device it is pretty much instant. Is there something I can change to get LMS to update quicker?

Cheers

[David Stanford]

How many syslog messages are you sending to the server per minute?

Also, does it take 30 minutes for it to reach the syslog.log or syslog_info or to get into the RME reports?

[paul2_uk]

Couldn't tell you how many per minute.

It seems like the syslog.log file gets updated once every 30 mins with all the information so all my traps within that 30 mins are logged as all happening at the same time.

example:

Jan 17 2007 15:50:38 PORT_SECURITY 2 SECURITYREJECT Security violation occurred on module 0 port 14 caused by MAC address xxxx.xxxx.xxxx

Jan 17 2007 15:50:38 ILPOWER 3 CONTROLLER_PORT_ERR Controller port error, Interface Fa0/5: Power given, but link is not up

These traps are from different switches, I have used RME reporting to get the info.

[nickmaiolo]

Try monitoring the syslog_info file (/var/log/syslog_info on solaris) and see if syslogs are received there in real time or not.

If they're delayed in this file, then its an Operating system issue.

[paul2_uk]

I have implemented the windows version which uses the syslog.log file. Has anyone come across any OS problems with LMS on Windows Server 2003?

[dcwalker]

I have just installed LMS 2.6 and noticed that messages don't instantly appear.

It looks like there is a process running that peridically kicks in and processes the new messages in bulk.

We get a few hundred messages an hour and I notice that the syslog won't change for at least 10 minutes and then dozens of messages appear.

I suspect they have tried to reduce the load of processing messages by batching them up a bit.

[David Stanford]

The delay is usually due to a few variables including a socket time of 30 seconds and other variables including CrmMsgCount, MaxFlushCount, and FlushIntervalInMillis.

crmlog listens for syslogs on 514/udp. It will block on the UDP socket for no more than 30 seconds PER MESSAGE. This timeout is hard-coded. It will not return the total messages read until either the 30 second timeout expires for at least one message or CrmMsgCount messages have been read.

Once it does return, it passes the messages off to be written to the syslog.log file. Those messages will not be flushed to the file, however, until either the message count reaches MaxFlushCount or the FlushIntervalInMillis timer expires.

[boldizsar.nagy]

30 sec delay is acceptable, but we get usually 5-10 minutes delay which is not.

We upgraded our system from LMS 2.1 which worked well, but now LMS 2.6 syslogs are delayed. There was no change on the transport network. I'm sure this is LMS specific.

What could casuse this behaviour?

[Joe Clarke · Accepted solution]

As Dave said, the new syslog damon on Windows was optimized to support more messages per second. The downside of this is that installations with fewer messages per second will see big delays. The solution is to re-tune the crmlog daemon. By tuning this daemon to provide more real-time syslog updates, you will reduce its scalability factor.

Try setting CrmMsgCount to 30 in the Windows Registry under HKLM\SYSTEM\CurrentControlSet\Services\crmlog\Parameters then restart the crmlog service (net stop crmlog / net start crmlog). This will reduce the number of messages needed to fill the flush buffer. The lower you make this number, the more real-time syslog becomes, but the fewer messages per-second you can handle.

[paul2_uk]

Thanks, now I have changed that registry setting everything is coming through quicker.