cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3072
Views
5
Helpful
7
Replies

Is Prime Infrastructure killing ssh on my NX-OS?

sdavids5670
Level 2
Level 2

Recently we've been experiencing a rather irritating issue with SSH on our Nexus 7Ks.  This occurred back when we were running 6.0(2).  Now we are running this version:

Software

  BIOS:      version 3.22.0

  kickstart: version 6.1(3)

  system:    version 6.1(3)

  BIOS compile time:       02/20/10

  kickstart image file is: bootflash:///n7000-s1-kickstart.6.1.3.bin

  kickstart compile time:  12/25/2020 12:00:00 [02/22/2013 23:54:07]

  system image file is:    bootflash:///n7000-s1-dk9.6.1.3.bin

  system compile time:     2/15/2013 14:00:00 [02/23/2013 01:08:44]

and we continue to have the problem.  The issue is that ssh server stops accepting connections.  The workaround is to console into the device and "no feature ssh" followed by "feature ssh" to recycle the ssh server.  This started happening around about the same time we added the devices to Cisco Prime Infrastructure.  At about the same time we turned on a security auditing product called nCircle which does weekly scans of our management network.  In the past, nCircle has caused issues with Cisco devices (specifically ssh) but newer versions of IOS code have fixed those issues.  If I had to bet money my guess is that it's nCircle again so I've asked the nCircle admin to stop scanning the management network.  However, I figured I'd post this here just in case somebody else has had an issue with ssh on NX-OS after adding to Prime Infrastructure.

1 Accepted Solution

Accepted Solutions

edeloscobos
Level 1
Level 1

Man,

Check this info that I get from the TAC:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCue74597

Description

=========

While polling, the device will create an ssh session to perform this. Once the polling device has finished polling, the ssh session will be left on the N7K. Eventually this will lead to all usable VTY lines being held by stale ssh sessions. We are not terminating the ssh sessions locally on the N7K. This seems to independent of NX-OS code versions as we have seen similar behaviour on multiple code versions of NX-OS. The stale sessions can be removed by disabling/enabling feature ssh.

Symptom:

========

Stale ssh sessions on N7K are using all available VTY lines and SSH is no longer available to the device.

Conditions:

=========

Stale SSH sessions.

Workaround:

==========

Disable/Enable feature ssh to clear stale sessions.

To verify this, you can run:

=====================

sh processes cpu | in sshd

show system internal processes memory | in ssh

sh socket connection

You should see a number of SSH processes running. If that is the case, the work around would be to disable/ enable SSH from the console. I believe this should be fixed in 6.1(5) and 6.2 which will be released in July.

View solution in original post

7 Replies 7

edeloscobos
Level 1
Level 1

Man,

Check this info that I get from the TAC:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCue74597

Description

=========

While polling, the device will create an ssh session to perform this. Once the polling device has finished polling, the ssh session will be left on the N7K. Eventually this will lead to all usable VTY lines being held by stale ssh sessions. We are not terminating the ssh sessions locally on the N7K. This seems to independent of NX-OS code versions as we have seen similar behaviour on multiple code versions of NX-OS. The stale sessions can be removed by disabling/enabling feature ssh.

Symptom:

========

Stale ssh sessions on N7K are using all available VTY lines and SSH is no longer available to the device.

Conditions:

=========

Stale SSH sessions.

Workaround:

==========

Disable/Enable feature ssh to clear stale sessions.

To verify this, you can run:

=====================

sh processes cpu | in sshd

show system internal processes memory | in ssh

sh socket connection

You should see a number of SSH processes running. If that is the case, the work around would be to disable/ enable SSH from the console. I believe this should be fixed in 6.1(5) and 6.2 which will be released in July.

That's it!  Thanks!

David Niemann
Level 3
Level 3

I had this issue as well and upgraded to 6.2.2, but it still persists.  I've opened another TAC case.  I think the problem is caused by Prime polling the Nexus, but regardless the Nexus shouldn't be allowing devices to tie up all the ssh sessions.

We got so tired of this issue that we finally just removed all of the Nexus devices from Prime Infrastructure.  It's pretty incredible that Cisco cannot get their own products to play well together.  This should be high on their priority list.  If you get anywhere with TAC please keep us posted.

David Niemann
Level 3
Level 3

I may have to do the same.  At this point we have Prime infrastructure and still have Prime LMS 4.2 so I may just remove it from Prime Infra for now.  We actually monitor all our devices with both products and are dependant on them for generating alerts so I'm hoping they get this resolved.

brandoncwilson
Level 1
Level 1

I'm dealing with this same issue of stale SSH sessions hanging open after a network management/monitoring tool (Nagios) logs in to collect the running config and do its scans. About once a month I have to go in and execute no feature ssh and then feature ssh to clear the stale sessions out, otherwise the system will run out of VTY resources and start denying SSH attempts..

I'm running NX-OS version 7.3(0)DX(1) on a pair of 7718s and both are suffering from this bug. Interestingly enough I have a 7710 that is running version 7.2(1)D1(1) and it doesnt have this same problem.

Will be opening a ticket on this today to see what TAC says.

TAC thinks we are seeing a known bug in  NX-OS that causes the stale SSH sessions to pile up.

Here is a link to the bug report

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuu61774

Note that the bug report lists a number of NX-OS versions that are known not to suffer from this bug. We are going to look at upgrading to a version in the 7.3(1) family very soon to see if it corrects this issue.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: