Syslog Push

Unanswered Question
Sep 30th, 2008

Im having an issue with broken logs being pushed off the IronPort. My setup involves a cluster of Ironports set to do syslog push to a remote host that uses rsyslog. Local logs generated by rsyslog are referenced by a splunk forward server and sent to another splunk index server where they can be easily referenced.

The problem comes in here after the log has been sent to rsyslog, I am receiving full logs on some transactions and others not, below is a broken log - the Ironport logs this verbosely and includes all the data i need to see what happened, the rsyslog server only includes 3 lines of data which is useless to me. Bear in mind this does work on the majority of the logs so between rsyslog and IronPort these log extracts are broken up. Any help will be appreciated

IronPort

Tue Sep 30 13:10:44 2008 Info: Start MID 255869040 ICID 893156518
Tue Sep 30 13:10:44 2008 Info: MID 255869040 ICID 893156518 From: <x>
Tue Sep 30 13:10:45 2008 Info: MID 255869040 ICID 893156518 LDAPACCEPT bypass applied to <x>
Tue Sep 30 13:10:45 2008 Info: MID 255869040 ICID 893156518 RID 0 To: <x>
Tue Sep 30 13:10:46 2008 Info: MID 255869040 Message-ID '<BA3FC561D75D4248804FBDB7E3C162051D2A184C>'
Tue Sep 30 13:10:46 2008 Info: MID 255869040 Subject "RE: Subject"
Tue Sep 30 13:10:46 2008 Info: MID 255869040 ready 181041 bytes from <domain1>
Tue Sep 30 13:10:46 2008 Info: MID 255869040 matched all recipients for per-recipient policy DEFAULT in the inbound table
Tue Sep 30 13:10:46 2008 Info: MID 255869040 was too big (181041/131072) for scanning by CASE
Tue Sep 30 13:10:46 2008 Info: MID 255869040 interim AV verdict using Sophos CLEAN
Tue Sep 30 13:10:46 2008 Info: MID 255869040 antivirus negative
Tue Sep 30 13:10:46 2008 Info: MID 255869040 queued for delivery
Tue Sep 30 13:10:46 2008 Info: Delivery start DCID 41961984 MID 255869040 to RID [0]
Tue Sep 30 13:10:46 2008 Info: Message done DCID 41961984 MID 255869040 to RID [0]
Tue Sep 30 13:10:46 2008 Info: MID 255869040 RID [0] Response '2.6.0 <BA3FC561D75D4248804FBDB7E3C162051D2A184C> Queued mail for delivery'
Tue Sep 30 13:10:46 2008 Info: Message finished MID 255869040 done


Rsyslog


Sep 30 13:10:45 ironp6 splunk_log Info: MID 255869040 ICID 893156518 LDAPACCEPT bypass applied to <domain2>
Sep 30 13:10:46 ironp6 splunk_log Info: MID 255869040 Message-ID '<BA3FC561D75D4248804FBDB7E3C162051D2A184C>'
Sep 30 13:10:46 ironp6 splunk_log Info: MID 255869040 Subject "RE: Subject"

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Douglas Hardison Thu, 10/02/2008 - 14:49

Obviously something is being lost in transport between the IronPort and the syslog server.

The IronPort is using a standard remote syslog push, as would any *nix box.

In this case, I would definitely recommend using traffic analysis, such as tcpdump. If I understand your configuration correctly, you have multiple hosts the data is passing to. You would want to check the traffic between each host to see where is drop is occurring.

bpoyner_ironport Thu, 10/02/2008 - 15:22

Obviously something is being lost in transport between the IronPort and the syslog server.

The IronPort is using a standard remote syslog push, as would any *nix box.

In this case, I would definitely recommend using traffic analysis, such as tcpdump. If I understand your configuration correctly, you have multiple hosts the data is passing to. You would want to check the traffic between each host to see where is drop is occurring.


In addition to traffic analysis, check your network settings to make sure you're not misconfigured. Sometimes autonegotiation with the switch will give you a bad setting and you'll end up with collisions and dropped packets. Since syslog uses UDP packets, a dropped packet will not be resent.

Here's how to tell if you're getting ethernet collisions and errors:


Ironport> netstat

Choose the information you want to display:
1. List of active sockets.
2. State of network interfaces.
3. Contents of routing tables.
4. Size of the listen queues.
5. Packet traffic information.
[1]> 2

Select the ethernet interface whose state you wish to display:
1. Data 1
2. Data 2
3. Management
4. ALL
[]> 4

Show the number of bytes in and out? [N]> y

Show the number of dropped packets? [N]> y
steven_geerts Thu, 10/02/2008 - 21:21

Hi,

It might help to use Syslog push over TCP. be sure your receiving syslog daemon is capable of receiving TCP syslog streams.
SyslogNG from Balabit (http://www.balabit.com/network-security/syslog-ng/opensource-logging-sys...) is a good candidate for that.
Another issue we had with it was our firewall that performed package inspection and detected that the TCP514 stream we pulled up was not a "remote shell" stream. (In the pre-ssh time zone, TCP514 used to be the port for remote shell). We had to disable the package inspection for TCP514 to get it functional.

Steven

Donald Nash Fri, 10/03/2008 - 17:51

Syslog uses unreliable UDP as its transport, so dropped packets mean lost log data and therefore corrupted log files. Bpoyner mentions collisions and improper switch settings as possible culprits, but there could be other factors as well. For example, UDP packets can be dropped if any host or router along the path (including the source host itself), happens to have a transient memory shortage at just the right time. It still shocks me that UC Berkeley (Eric Allman in particular), chose such an unreliable transport for something as important at log information.

Steven's suggestion to use syslog over TCP should do the trick, assuming that AsyncOS itself can do syslog over TCP (I haven't checked that).

Actions

This Discussion