RME SyslogCollector: Anonymous Dropping the syslog as queue is full

Answered Question
Jun 15th, 2009
User Badges:
  • Blue, 1500 points or more

I found the following entries in SyslogCollector.log:


SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:33,427, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:33,428, System Initialized.

SyslogCollector - [Thread: main] WARN , 27 Mar 2009 18:00:35,078, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 27 Mar 2009 18:00:35,145, Service started...

SyslogCollector - [Thread: SyslogObjectForwarder] ERROR, 08 May 2009 14:48:34,775, Could not send syslogs, removing the subscriber...Connection

refused

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:04,354, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:04,356, System Initialized.

SyslogCollector - [Thread: main] WARN , 08 May 2009 14:53:05,985, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 08 May 2009 14:53:06,049, Service started...

SyslogCollector - [Thread: SyslogObjectForwarder] ERROR, 09 Jun 2009 19:44:50,365, Could not send syslogs, removing the subscriber...Connection

refused

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:54,337, Logging System Initialized.

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:54,349, System Initialized.

SyslogCollector - [Thread: main] WARN , 09 Jun 2009 21:12:57,014, Unable to resurrect connection to a subscriber.

SyslogCollector - [Thread: main] INFO , 09 Jun 2009 21:12:57,073, Service started...

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:28,440, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,490, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,491, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:33,492, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:36,520, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:40,560, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:42,579, Anonymous Dropping the syslog as queue is full 100000

SyslogCollector - [Thread: EvaluatorThread-0] INFO , 12 Jun 2009 04:33:46,620, Anonymous Dropping the syslog as queue is full 100000



RME Syslog Reports were not working anymore. After Restarting SyslogAnalyzer and SyslogCollector everything seems to be fine.

Is the SyslogCollector not able to recovery itself if its queue is full? Is a message storm producing this failure or what else could be the reason?


Correct Answer by Joe Clarke about 7 years 9 months ago

Maybe, but you may have had to manually resubscribe the Collector. We typically recommend the two daemons are always restarted together.


LMS 3.2 will have much better scalability when it comes to syslog. LMS 3.1 supports a sustained rate of 200 messages per second with bursts to 1000. However, the Analyzer can only process at 50 per second with bursts to 200. This is fixed in LMS 3.2.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Joe Clarke Mon, 06/15/2009 - 10:55
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Something was broken with syslog. It may have been the Analyzer. The buffer will maintain itself. As messages are read from the buffer, the buffer size will shrink allowing for new messages to be processed from the syslog log file. I suppose you could have gotten into a situation where sustained logging exceeded more than 1000 messages per second, and caused a race which deadlocked things. However, if a restart corrected this, I suspect there may have been a problem with the Analyzer.

Martin Ermel Mon, 06/15/2009 - 12:22
User Badges:
  • Blue, 1500 points or more

First I thought it was a problem with the syslog_info file size which was 4 GB (but had no obvious message drops). But backing it out with logrot did not change the situation. I also thought of a high syslog load (3500 device) but I could not believe that this situation was for 3 days ...

Perhaps restarting Analyzer alone first and if nothing changed restarting SyslogCollector in the next step could have pinpointed the problem to the problematic process...

Correct Answer
Joe Clarke Mon, 06/15/2009 - 12:28
User Badges:
  • Cisco Employee,
  • Hall of Fame,

    Founding Member

Maybe, but you may have had to manually resubscribe the Collector. We typically recommend the two daemons are always restarted together.


LMS 3.2 will have much better scalability when it comes to syslog. LMS 3.1 supports a sustained rate of 200 messages per second with bursts to 1000. However, the Analyzer can only process at 50 per second with bursts to 200. This is fixed in LMS 3.2.

Actions

This Discussion