Cisco ASA 5585-X SSP-20 8.4(2) - TCP Syslog problem

[Jouni Forss]

Hi,

We have a firewall service environment where logging is handled with UDP at the moment.

Recently we have noticed that some messages get lost on the way to the server (Since the server doesnt seem to be under huge stress from syslog traffic). We decided to try sending the syslog via TCP.

You can imagine my surprise when I enabled the "logging host <interface name> <server ip> tcp/1470" on an ASA Security context and find out that all the connections through that firewall are now being blocked. Granted, I could have checked the command reference for this specific command but I never even thought of the possibility of a logging command beeing able to stop all traffic on a firewall.

The TCP syslog connection failing was caused by a missmatched TCP port on the server which got corrected quickly. Even though I could now view log messages from the firewall in question in real time, the only message logged was the blocking of new connections with the following syslog message:

"%ASA-3-201008: Disallowing new connections."

Here start my questions:

- New connections are supposed to be blocked when the the TCP Syslog server aint reachable. How is it possible that I am seeing the TCP syslog sent to the server and the ASA Security Context is still blocking the traffic?

- I configured the "logging permit-hostdown" after I found the command and it supposedly should prevent the above problem/situation from happening. Yet after issuing this command on the Security Context in question, connections were still being blocked with the same syslog message. Why is this?

- Eventually I changed the logging back to UDP. This yet again caused no change to the situation. All the customer connections were still being blocked. Why is this?

- After all the above I removed all possible logging configurations from the Security Context. This had absolutely no effect on the situation either.

- As a last measure I changed to the system context of the ASA and totally removed the syslog interface from the Security Context. This also had absolutely no effect on the situation.

At the end I was forced to save the configuration on the ASAs Flash -memory, remove the Security Context, create the SC again, attach the interfaces again and load the configuration from the flash into the Security Context. This in the end corrected the problem.

Seems to me this is some sort of bug since the syslog server was receiving the syslog messages from the SC but the ASA was still blocking all new connections. Even the command "logging permit-hostdown" command didnt help or changing back to UDP.

It seems the Security Context in question just simply got stuck and continued blocking all connections even though in the end it didnt have ANY logging configurations on.

Seems to me that this is quite a risky configuration if you are possibly facing cutting all traffic for hundreds of customers when the syslog connection is lost or the above situation happens and isnt corrected by any of the above measures we took (like the command "logging permit-hostdown" which is supposed to avoid this situation alltogether).

- Jouni

[Julio Carvajal]

Hello Jouni,

Totally understand what you mean!

I remember the time I when I did a lab and I could not get the internal users out to the internet (Easy task) but I did not know I need it the command logging permit-hostdown if by any chance the TCP syslog server was not able to receive the packets and send an ACK.

I remember that after I did that I clear the local-host table and everything started to work!

Did you clear it?

Regards,

Julio

CSC it's a free support community take your time to rate all the engineer's responses that help you resolving your problems.

[Jouni Forss]

Hi,

Thanks for the reply

I didnt issue a "clear local-host all" command. I was kind of expecting the fact that the syslog server was receiving the messages and the fact that the "logging permit-hostdown" commands was issued to correct the situation.

If that really does correct the problem I guess the ASA is really picky on in what order you give the logging configurations. In my situation it would mean that if I havent issued "logging permit-hostdown" command before acticating TCP syslogging (which could be considered logical order though) the ASA wont react to it when the problem scenario that I described is already happening AND then I issue the command "logging permit-hostdown"

A quick glance to the command refence and the command "clear local-host" suggests that the command is used to for example clear all connections for a specific host so that a new configuration change will start to apply to the hosts connections.

I guess I will have to test this in a test setup on the same ASA device. I have a couple of test contexts on the ASA that I can use. I will probably build a connection to our lab and test/create the situation and try the "clear local-host" command and see if it changes the situation.

If it does correct the situation it would still mean that  this sort of change (Syslog from UDP to TCP) would require clearing all  existing connections from the ASAs / SCs which means this change has to  be done at a time when it doesnt disturb customer connections.

I will let you know (and rate the post)  here if I managed to produce the same situation again and correct it with the command.

I have also asked our Cisco contacts to look into the case but havent yet received any information related to this.

- Jouni

[Julio Carvajal]

Hello Jouni,

FYI placing a capture on the interface where the syslog server is located matching UDP 514  to check if the ASA is indeed sending the right packets,

Regards,

Julio

PD: Please let us know any update on this ( really interesting topic)

[Jouni Forss]

Hi,

I FINALLY had the time to look at this issue as I was testing something else in our lab too.

In short, here is what I did:

• I configured the TCP logging in the same way as in the original post
• I configured the TCP logging giving the commands in different order
• Did some other tests related to the proble

Device used: ASA 5585-X

Software: 8.4(2)

Original Device and software : ASA 5585-X running 8.4(1)9

Heres the above scenarions and what actually happened

Original situation

Before doing any changes the test firewall context in question is working normally and the log sent by UDP/514 is arriving to the Syslog server as usual.

I now change the syslog to TCP by giving a command "logging host tcp/1471" (actual port being TCP/1470)

The firewall immediatly starts blocking all connections going through it.

I change the configuration to the correct port TCP/1470 after which log starts appearing in my realtime view on the syslog server. The firewall context in question is still sending only the message "Disallowing new connections" even though the TCP -port on the Syslog server is clearly reachable and the connection is active.

After this I try to do the suggest "clear local-host all" command. This has no effect on the firewall context. No connections are getting through. No connections/xlates are formed on the firewall. I can only see the firewall doing DNS queries with its outside interface (related to another configuration).

After this I try to start correcting the situation the same way as before. I add "logging permit-hostdown" command which has no effect on the situation. I remove all logging configurations and it doesnt have any effect on the situation.

After this I activate UDP logging and can see the logs arriving on the syslog server but again I can only see "Disallowing new connections" message.

In the end I have no other option (to my knowledge) other than to delete the Security Context and create it again with same interfaces and with the configuration saved to the Flash -memory of the ASA.

After this the connections work like usual. (UDP logging in the saved configuration)

Giving the configurations in different order

After I've created the firewall again and all is working I have another try in configuring the TCP Syslog while giving the commands in different order.

First I add the command "logging permit-hostdown" command

Then I add the command "logging host tcp/1470"

After this logs start arriving on the syslog server and connections work as usual. Seems giving the "logging permit-hostdown" first before any other configurations is the right way to go.

Removing the "logging permit-hostdown" command

After I saw that everything was working I tried to remove the "logging permit-hostdown" command and see what happens. Everything worked fine.

Configuring wrong TCP port to "logging host" command

I decide to try and change the TCP port used to a wrong one and see if anything happens. (logging permit-hostdown is active). Firewall works as usual. Naturally no logs can be viewed at the syslog server.

Configuring the TCP Syslogging without "logging permit-hostdown" but with correct port

Finally I tried to configure the TCP Syslogging on ASA with the correct TCP port without issuing the "logging permit-hostdown" command. Everything seemed to work fine after this.

So in conclusion it seems that IF you don't have the "logging permit-hostdown" command issued before you start configuring "logging host tcp/xxxx" , you might run into problems IF you don't have matching settings on the ASA sending the log and the Syslog server receiving the log.

There doesnt seem to be any easy way to correct the situation (with the connections getting blocked) after you have once messed up the configurations. Seems your only option is to reconfigure the Security Context (which is easy) or if this problem exists in the same way in a single ASA you will have to reboot the device which means longer downtime than reconfiguring a context.

There would still be a couple of things to test but at the moment I have no more time for this. I will update if there is any new information.

- Jouni

[stevechege]

Thanks Jouni. I run into this on a Cisco ASA 5510. The only way to restore service was to do a painful reboot where the whole office was down.

Cisco's documentation regarding TCP syslog:

The %ASA-3-201008: Disallowing new connections error message is seen when ASA is unable to contact syslog server and no new connections are allowed...This message appears when you have enabled TCP system log messaging and the syslog server cannot be reached.

If the syslog server goes down and the TCP logging is configured either use the logging permit-hostdown command or switch to UDP logging.

[dhawthorne]

What version are you running on your 5510?