SSH connections through asa hanging

[waqas gondal]

Hi,

I have been troubleshooting this intermittent issue with Cisco TAC for about 3 weeks now.

The issue is that when users connect to external servers via ssh whether it is over a VPN or not, the connections hang. The timing is random and is not dependant on anything. After a user connects, the connection will hang and the show local host indicates that the connection is idle.

We are using an ASA 5505 with version 8.4(5) and a 5510 with 8.4(6) at another location with a site-to-site VPN in between them. The users who initiate the ssh connection are behind the 5505 and sometimes connect to servers behind the 5510, those connections also hang randomly.

When we create ssh traffic through the VPN to servers behind the 5510:

The packet captures don't show a reason for the connection to hang, they just show that packets have stopped going through.

Syslog messages show nothing on the 5505 when the connection hangs, syslogs on the 5510 sometimes show the "deny tcp (no connection)"

Any ideas as to what might cause this issue?

[Jon Are Endrerud]

Is it possible this could have something to do with your ISP or other providers?

Sent from Cisco Technical Support iPhone App

[waqas gondal]

It is possible but I have not checked with them yet.

[Julio Carvajal]

Hello,

So basically a SSH session that already was closed is still present on the local-host table of the ASA and the connection table??

Can you check the Timeout configuration on your firewall and also the MPF setup.

What's the Idle time you have configured for a TCP session?

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[waqas gondal]

The timeout on both ASAs is 1 hour. However if an ssh connection is established from behind the 5505 to a destination behind the 5510 the hanging connection is not present in the table of the 5510, and idle on the 5505.

The problem isn't getting the connection out of the connection table. The problem is trying to figure out why the connections are hanging intermittently.

There is no MPF setup

[Julio Carvajal]

Hi,

When you say the hanging connection what do you mean?

Do you mean the connection is closed but still present on one of the FWs?

That let us know we can focus on the ASA 5505

Can you share the configuration used there

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[waqas gondal]

By hanging connections I mean there is no packet in any of the wireshark captures that indicates the connection has closed. A connection remains on the 5505 but the amount of bytes passed through do not increase. The ssh connection window to the device behind the 5510 is still open but inactive with no messages indicating close.

Unfortuantly I cannot share configuration for policy reasons. Here I just wanted some ideas for things to look for when troubleshooting.

Thank you for your time.

[Julio Carvajal]

Well,

I would create a capture on both of the interfaces (as you said you did).

I would check the MPF configuration for any specific set connection timeout

I would also check the Global timeout connection.

And of course enable logging on the FW to capture as much information as possible (between this sessions)

Is this the only traffic affected?

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[waqas gondal]

Hi,

SSH is the only affected connection type.

And I have tried all of the things which you have mentioned. It is difficult to look through the firewall logs because they are extensive and don't show what caused the connection to hang.

Thanks,

Waqas

[Julio Carvajal]

Well,

I am basically troubleshooting on blind mode so we cannot move forward bud.

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[waqas gondal]

Thank you for your input though, I really appreciate it.

[Julio Carvajal]

Sure,

If you are willing to work providing updates and config related to the problem let us know.

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[cciesec2011]

The problem can be resolved very easily without touching the Cisco device.  By enabling ssh keep-alive on either the ssh client or the ssh server.

/etc/ssh/sshd_config

Look for TCPKeepAlive and make sure it is set to yes and add the following lines after it:

ClientAliveInterval 30

ClientAliveCountMax 10000

service sshd restart

This will help the ssh connection from disconnecting.  If you still experience it, it is the cisco ASA

[Julio Carvajal]

Hello,

I seriously do not think your configuration on the client/server side will make any difference.

We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.

On the other ASA (5510) the connection is succesfully removed from all of the respective tables.

So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

[cciesec2011]

jcarvaja wrote:
Hello,
I seriously do not think your configuration on the client/server side will make any difference.
We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.
On the other ASA (5510) the connection is succesfully removed from all of the respective tables.
So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

You can not base that on what the user described.  In order to understand the issue, you need packet capture.

What I am suggesting is commonly used for connectivity traversing the firewall to prove whether the issue is on network or application itself.  By enabling keepalive on the application, you can see how it behaves.