RME SyslogCollector: Anonymous Dropping the syslog as queue is full

[Martin Ermel]

I found the following entries in SyslogCollector.log:

SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:33,427, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:33,428, System Initialized.

SyslogCollector - [Thread: main] WARN , 27 Mar 2009 18:00:35,078, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:35,145, Service started...

SyslogCollector - [Thread: SyslogObjectForwarder] ERROR, 08 May 2009 14:48:34,775, Could not send syslogs, removing the subscriber...Connection

refused

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:04,354, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:04,356, System Initialized.

SyslogCollector - [Thread: main] WARN , 08 May 2009 14:53:05,985, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:06,049, Service started...

SyslogCollector - [Thread: SyslogObjectForwarder] ERROR, 09 Jun 2009 19:44:50,365, Could not send syslogs, removing the subscriber...Connection

refused

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:54,337, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:54,349, System Initialized.

SyslogCollector - [Thread: main] WARN , 09 Jun 2009 21:12:57,014, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:57,073, Service started...

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:28,440, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,490, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,491, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,492, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:36,520, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:40,560, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:42,579, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:46,620, Anonymous Dropping the syslog as queue is full 100000

RME Syslog Reports were not working anymore. After Restarting SyslogAnalyzer and SyslogCollector everything seems to be fine.

Is the SyslogCollector not able to recovery itself if its queue is full? Is a message storm producing this failure or what else could be the reason?

[Joe Clarke]

Something was broken with syslog. It may have been the Analyzer. The buffer will maintain itself. As messages are read from the buffer, the buffer size will shrink allowing for new messages to be processed from the syslog log file. I suppose you could have gotten into a situation where sustained logging exceeded more than 1000 messages per second, and caused a race which deadlocked things. However, if a restart corrected this, I suspect there may have been a problem with the Analyzer.

[Martin Ermel]

First I thought it was a problem with the syslog_info file size which was 4 GB (but had no obvious message drops). But backing it out with logrot did not change the situation. I also thought of a high syslog load (3500 device) but I could not believe that this situation was for 3 days ...

Perhaps restarting Analyzer alone first and if nothing changed restarting SyslogCollector in the next step could have pinpointed the problem to the problematic process...

[Joe Clarke · Accepted solution]

Maybe, but you may have had to manually resubscribe the Collector. We typically recommend the two daemons are always restarted together.

LMS 3.2 will have much better scalability when it comes to syslog. LMS 3.1 supports a sustained rate of 200 messages per second with bursts to 1000. However, the Analyzer can only process at 50 per second with bursts to 200. This is fixed in LMS 3.2.