Syslog Push

martinc8306 · ‎09-30-2008

Im having an issue with broken logs being pushed off the IronPort. My setup involves a cluster of Ironports set to do syslog push to a remote host that uses rsyslog. Local logs generated by rsyslog are referenced by a splunk forward server and sent to another splunk index server where they can be easily referenced.

The problem comes in here after the log has been sent to rsyslog, I am receiving full logs on some transactions and others not, below is a broken log - the Ironport logs this verbosely and includes all the data i need to see what happened, the rsyslog server only includes 3 lines of data which is useless to me. Bear in mind this does work on the majority of the logs so between rsyslog and IronPort these log extracts are broken up. Any help will be appreciated

IronPort

Tue Sep 30 13:10:44 2008 Info: Start MID 255869040 ICID 893156518
Tue Sep 30 13:10:44 2008 Info: MID 255869040 ICID 893156518 From: <x>
Tue Sep 30 13:10:45 2008 Info: MID 255869040 ICID 893156518 LDAPACCEPT bypass applied to <x>
Tue Sep 30 13:10:45 2008 Info: MID 255869040 ICID 893156518 RID 0 To: <x>
Tue Sep 30 13:10:46 2008 Info: MID 255869040 Message-ID '<BA3FC561D75D4248804FBDB7E3C162051D2A184C>'
Tue Sep 30 13:10:46 2008 Info: MID 255869040 Subject "RE: Subject"
Tue Sep 30 13:10:46 2008 Info: MID 255869040 ready 181041 bytes from <domain1>
Tue Sep 30 13:10:46 2008 Info: MID 255869040 matched all recipients for per-recipient policy DEFAULT in the inbound table
Tue Sep 30 13:10:46 2008 Info: MID 255869040 was too big (181041/131072) for scanning by CASE
Tue Sep 30 13:10:46 2008 Info: MID 255869040 interim AV verdict using Sophos CLEAN
Tue Sep 30 13:10:46 2008 Info: MID 255869040 antivirus negative
Tue Sep 30 13:10:46 2008 Info: MID 255869040 queued for delivery
Tue Sep 30 13:10:46 2008 Info: Delivery start DCID 41961984 MID 255869040 to RID [0]
Tue Sep 30 13:10:46 2008 Info: Message done DCID 41961984 MID 255869040 to RID [0]
Tue Sep 30 13:10:46 2008 Info: MID 255869040 RID [0] Response '2.6.0 <BA3FC561D75D4248804FBDB7E3C162051D2A184C> Queued mail for delivery'
Tue Sep 30 13:10:46 2008 Info: Message finished MID 255869040 done

Rsyslog

Sep 30 13:10:45 ironp6 splunk_log Info: MID 255869040 ICID 893156518 LDAPACCEPT bypass applied to <domain2>
Sep 30 13:10:46 ironp6 splunk_log Info: MID 255869040 Message-ID '<BA3FC561D75D4248804FBDB7E3C162051D2A184C>'
Sep 30 13:10:46 ironp6 splunk_log Info: MID 255869040 Subject "RE: Subject"

Douglas Hardison · ‎10-02-2008

Obviously something is being lost in transport between the IronPort and the syslog server.

The IronPort is using a standard remote syslog push, as would any *nix box.

In this case, I would definitely recommend using traffic analysis, such as tcpdump. If I understand your configuration correctly, you have multiple hosts the data is passing to. You would want to check the traffic between each host to see where is drop is occurring.

bpoyner_ironport · ‎10-02-2008

Obviously something is being lost in transport between the IronPort and the syslog server.

The IronPort is using a standard remote syslog push, as would any *nix box.

In this case, I would definitely recommend  using traffic analysis, such as tcpdump.  If I understand your configuration correctly, you have multiple hosts the data is passing to. You would want to check the traffic between each host to see where is drop is occurring.

In addition to traffic analysis, check your network settings to make sure you're not misconfigured. Sometimes autonegotiation with the switch will give you a bad setting and you'll end up with collisions and dropped packets. Since syslog uses UDP packets, a dropped packet will not be resent.

Here's how to tell if you're getting ethernet collisions and errors:


Ironport> netstat

Choose the information you want to display:
1. List of active sockets.
2. State of network interfaces.
3. Contents of routing tables.
4. Size of the listen queues.
5. Packet traffic information.
[1]> 2

Select the ethernet interface whose state you wish to display:
1. Data 1
2. Data 2
3. Management
4. ALL
[]> 4

Show the number of bytes in and out? [N]> y

Show the number of dropped packets? [N]> y

steven_geerts · ‎10-02-2008

Hi,

It might help to use Syslog push over TCP. be sure your receiving syslog daemon is capable of receiving TCP syslog streams.
SyslogNG from Balabit (http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/) is a good candidate for that.
Another issue we had with it was our firewall that performed package inspection and detected that the TCP514 stream we pulled up was not a "remote shell" stream. (In the pre-ssh time zone, TCP514 used to be the port for remote shell). We had to disable the package inspection for TCP514 to get it functional.

Steven

Donald Nash · ‎10-03-2008

Syslog uses unreliable UDP as its transport, so dropped packets mean lost log data and therefore corrupted log files. Bpoyner mentions collisions and improper switch settings as possible culprits, but there could be other factors as well. For example, UDP packets can be dropped if any host or router along the path (including the source host itself), happens to have a transient memory shortage at just the right time. It still shocks me that UC Berkeley (Eric Allman in particular), chose such an unreliable transport for something as important at log information.

Steven's suggestion to use syslog over TCP should do the trick, assuming that AsyncOS itself can do syslog over TCP (I haven't checked that).

ashishonok007 · ‎06-11-2019

Hello,

I have an exactly the same problem as was described by user martinc8306. We are using syslog-ng PE and when ironport has been configured to dump log as a file, syslog-ng was able to generate the syslog where all log entries have two timestamps - the original one, generated by ironport appliance and the timestamp, generated by syslog-ng

Sample of log, generated out of saved file (two timestamps - from ironport and from syslog-ng):

2018-12-19T14:08:21-05:00 123-esa1.xxx.yyy Wed Dec 19 13:24:28 2018 Info: New SMTP DCID 3903453 interface 15.18.13.13 address 10.11.0.75 port 25

Once we switch ironport appliances from saving log files to a streaming via syslog, the resulted syslog format looks like this:

2019-06-10T16:42:12-04:00 10.24.11.14 mail_logs_stream: Info: New SMTP DCID 5991471 interface 66.22.15.22 address 155.24.1.98 port 25

As you can see, when ironport is sending log via syslog, it didn't put a timestamp into a log. Is this correct assumption or there is a way to preserve an appliance timestamp in a streaming log?

Thanks

Paul Thomas Cyblue · ‎06-21-2019

That is correct.

Write to file - file timestamp cannot be used for all log entries, so each line receives a timestamp and msg content

Stream via Syslog - timestamp and msg added to the relevant syslog protocol properties and parsed by the receiving syslog server.

Their problem was they used UDP which simply hid the lost packet issues - likely Rsyslog could not keep up with volume and needed some queue tuning. Changing to TCP will generate alerts by the ESA if the syslog server isn't capable of receiving traffic fast enough.