Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Step-by-Step Configuration and Troubleshooting Best Practices for the NGFW, NGIPS and AMP Technologies A Visual Guide to the Cisco Firepower Threat Defense (FTD)
Community Member

SSH connections through asa hanging

Hi,

I have been troubleshooting this intermittent issue with Cisco TAC for about 3 weeks now.

The issue is that when users connect to external servers via ssh whether it is over a VPN or not, the connections hang. The timing is random and is not dependant on anything. After a user connects, the connection will hang and the show local host indicates that the connection is idle.

We are using an ASA 5505 with version 8.4(5) and a 5510 with 8.4(6) at another location with a site-to-site VPN in between them. The users who initiate the ssh connection are behind the 5505 and sometimes connect to servers behind the 5510, those connections also hang randomly.

When we create ssh traffic through the VPN to servers behind the 5510:

The packet captures don't show a reason for the connection to hang, they just show that packets have stopped going through.

Syslog messages show nothing on the 5505 when the connection hangs, syslogs on the 5510 sometimes show the "deny tcp (no connection)"

Any ideas as to what might cause this issue?

24 REPLIES
Community Member

Re: SSH connections through asa hanging

Is it possible this could have something to do with your ISP or other providers?

Sent from Cisco Technical Support iPhone App

Please rate as helpful, if that would be the case. Thanx
Community Member

SSH connections through asa hanging

It is possible but I have not checked with them yet.

Re: SSH connections through asa hanging

Hello,

So basically a SSH session that already was closed is still present on the local-host table of the ASA and the connection table??

Can you check the Timeout configuration on your firewall and also the MPF setup.

What's the Idle time you have configured for a TCP session?

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

The timeout on both ASAs is 1 hour. However if an ssh connection is established from behind the 5505 to a destination behind the 5510 the hanging connection is not present in the table of the 5510, and idle on the 5505.

The problem isn't getting the connection out of the connection table. The problem is trying to figure out why the connections are hanging intermittently.

There is no MPF setup

SSH connections through asa hanging

Hi,

When you say the hanging connection what do you mean?

Do you mean the connection is closed but still present on one of the FWs?

That let us know we can focus on the ASA 5505

Can you share the configuration used there

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

By hanging connections I mean there is no packet in any of the wireshark captures that indicates the connection has closed. A connection remains on the 5505 but the amount of bytes passed through do not increase. The ssh connection window to the device behind the 5510 is still open but inactive with no messages indicating close.

Unfortuantly I cannot share configuration for policy reasons. Here I just wanted some ideas for things to look for when troubleshooting.

Thank you for your time.

Re: SSH connections through asa hanging

Well,

I would create a capture on both of the interfaces (as you said you did).

I would check the MPF configuration for any specific set connection timeout

I would also check the Global timeout connection.

And of course enable logging on the FW to capture as much information as possible (between this sessions)

Is this the only traffic affected?

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

Hi,

SSH is the only affected connection type.

And I have tried all of the things which you have mentioned. It is difficult to look through the firewall logs because they are extensive and don't show what caused the connection to hang.

Thanks,

Waqas

Re: SSH connections through asa hanging

Well,

I am basically troubleshooting on blind mode so we cannot move forward bud.

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

Thank you for your input though, I really appreciate it.

Re: SSH connections through asa hanging

Sure,

If you are willing to work providing updates and config related to the problem let us know.

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

The problem can be resolved very easily without touching the Cisco device.  By enabling ssh keep-alive on either the ssh client or the ssh server.

/etc/ssh/sshd_config

Look for TCPKeepAlive and make sure it is set to yes and add the following lines after it:

ClientAliveInterval 30

ClientAliveCountMax 10000

service sshd restart

This will help the ssh connection from disconnecting.  If you still experience it, it is the cisco ASA

Re: SSH connections through asa hanging

Hello,

I seriously do not think your configuration on the client/server side will make any difference.

We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.

On the other ASA (5510) the connection is succesfully removed from all of the respective tables.

So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

jcarvaja wrote:

Hello,

I seriously do not think your configuration on the client/server side will make any difference.

We have already found where the problem is (ASA 5505) as the connection is hanging there while not traffic is being seeing.

On the other ASA (5510) the connection is succesfully removed from all of the respective tables.

So your work-around will not make any difference here as for the client/server the connection has been already closed (this after the customer description of the problem)

You can not base that on what the user described.  In order to understand the issue, you need packet capture.

What I am suggesting is commonly used for connectivity traversing the firewall to prove whether the issue is on network or application itself.  By enabling keepalive on the application, you can see how it behaves.

Re: SSH connections through asa hanging

You can not base that on what the user described.  In order to understand the issue, you need packet capture.

If customer does not provide us access to the box, inputs that we request we got to trust what he says. This is the case!

Now, I do not think you understand what I am saying..

If this were an app issue then the orphaned sessions would exist on both firewalls! Not just on one. As simple as that

Looking for some Networking Assistance? 
Contact me directly at jcarvaja@laguiadelnetworking.com

I will fix your problem ASAP.

Cheers,

Julio Carvajal Segura
http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Community Member

Re: SSH connections through asa hanging

Thanks,

This is something I will check.

Community Member

Re: SSH connections through asa hanging

I had this kind of issue long time ago.

I enabled an inside host to accept ssh connection and the ASA failed to enable it.

I remember that one of the steps that I've done, was.. reconfigure (ssh) the ASA from scratch, and test it many times to make sure, it was working properly.

Crypto, aaa, username, ip, inside/outside... All that.

Then I went to create object, nat (inside, outside) host or subnet, acl then access-group.

That way worked for me, but it tool me 3 days to figure that out.

You could try that, hope it works.

Regards,
Oscar



Sent from Cisco Technical Support iPhone App

Community Member

Re: SSH connections through asa hanging

This issue is a little different I think.

My ASA allows the connection through but after a random amount of time the connection hangs.

I do understand your logic though, I have had issues where I simply erased the config, applied it again and everything was functonal.

Thanks for your input,

Waqas

Community Member

Re: SSH connections through asa hanging

waqas gondal wrote:

My ASA allows the connection through but after a random amount of time the connection hangs.

That's reason why I suggest you turn on ssh keep-alive and see if the issue goes away.  If the issue goes away, then you know it is a firewall issue.

Community Member

Re: SSH connections through asa hanging

When does it hang? Is it possible to use it at all?

Is it after big blocks of text passing through the terminal?

What is the MTU between the ssh server and client?

Sent from Cisco Technical Support iPhone App

Please rate as helpful, if that would be the case. Thanx
Community Member

Re: SSH connections through asa hanging

When I open an ssh window to any destination through the ASA, it freezes after a random amount of time. It does not matter whether anything is being done through the window. After that it is not possible to use unless the connection is re-initiated.

I will check the MTU on the clients and servers.

Thanks,

Waqas

Community Member

Re: SSH connections through asa hanging

MTU are a know factor when it comes to SSH freeze over VPN, you should also turn on SSH keep alive. What SSH clients are used?

Sent from Cisco Technical Support iPhone App

Please rate as helpful, if that would be the case. Thanx
Community Member

Re: SSH connections through asa hanging

SSH clients are putty mostly, but it does not matter what client is used, the connections hang either way.

I am not at the office right now so I cannot check the MTU at the moment.

Here I am gathering notes on things that should be checked.

Thanks,

Waqas

Community Member

Re: SSH connections through asa hanging

Putty has a keep alive setting that you you should check in the options or connection setting box.

Sent from Cisco Technical Support iPhone App

Please rate as helpful, if that would be the case. Thanx
2543
Views
0
Helpful
24
Replies
CreatePlease to create content