Tftp session stuck in active by crypto key generate rsa command.

[Marcusbergh]

Hello everyone,

I have run into a problem with the command "crypto key generate rsa" when sending a config file via tftp from a server to a switch. This seems to have stuck the tftp session as an active session from server to multiple 2960 switch. By The session not becoming idle, the TFTP session is not automatically closed 600 seconds (default, I believe) after it has gone idle, as it should. The session have been active for more than 10 days.

-bash-4.1$ snmpwalk -v2c -c ABCDEF 172.24.9.14 .1.3.6.1.4.1.9.9.96.1.1.1.1 | grep "266 "
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.2.266 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.3.266 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.4.266 = INTEGER: 4
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.5.266 = IpAddress: 1.1.1.1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.6.266 = STRING: "tobosc/configfile_rm"
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.9.266 = INTEGER: 2
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.10.266 = INTEGER: 2
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.11.266 = Timeticks: (18598797) 2 days, 3:39:47.97
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.12.266 = Timeticks: (0) 0:00:00.00
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.14.266 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.15.266 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.16.266 = STRING: "1.1.1.1"

configfile_rm:

no ip name-server
no snmp-server enable traps

line vty 0 15
 access-class 5 in

crypto key generate rsa
2048

no access-list 10
ip access-list standard 99
no permit 172.21.252.64 0.0.0.63
end

part of the script, the snmp commands used to push config to switch:

 snmpset -v 2c -c ABCDEF 172.24.9.14 1.3.6.1.4.1.9.9.96.1.1.1.1.2.266 i 1 1.3.6.1.4.1.9.9.96.1.1.1.1.3.226 i 1 1.3.6.1.4.1.9.9.96.1.1.1.1.4.226 i 4 1.3.6.1.4.1.9.9.96.1.1.1.1.5.226 a 172.21.252.182 1.3.6.1.4.1.9.9.96.1.1.1.1.6.226 s tobosc/configfile_rm 1.3.6.1.4.1.9.9.96.1.1.1.1.14.226 i 4

Kill the snmp session 226: 

snmpset -v 2c -c ABCDEF 172.24.9.14 1.3.6.1.4.1.9.9.96.1.1.1.1.14.226 i 6

The issue lies in the configfile_rm, everything above the crypto key generate rsa command in the config has been pushed to the switches config. However, the crypto key generate rsa and everything bellow it has not been sent to the switches. The script used is confirmed to work since we have pushed other commands to switches before, without problems.

Since only one TFTP session can be active at a time, other sessions are queued behind this stuck in active TFTP session, which presents a problem since we need to be able to gather configuration from the switches on a daily basis.

From what can be seen in the firewall configuration it seems that the session has been build and closed correctly, however as seen in the above output that is not the case from the servers point of view. 

Does anyone have any other solution than to reload every single switch? is it possible to kill the active tftp sessions in some way?

One solution is to reload the switches, the session is terminated a while after we reload the Cisco 2960 switch, however since we currently got approximately 1000 stuck tftp sessions active on 1000 different  switches(one tftp session per switch), we would like to avoid reloading 1000 switches if can. All switches have the same IOS c2960-lanbasek9-mz.150-2.SE5.bin and a few worked, and didn't experience this issue where tftp session got stuck, approx. 40 out of 1100 switches were able to close their session, even though the configuration below crypto key generate rsa was not entered into the switch.

[Vinod Arya]

Ideally it is not recommmend to use this command with snmp config copy operation.

Don't perform a SNMP copy configuration with a configuration file that contains "crypto key generate rsa". If the switch has existing keys, the IOS expects either a YES or NO response to indicate if the user wants to replace the existing keys. If the switch does not have existing keys, the IOS expects the key size. The IOS never receives the YES or NO response nor the key size since the copy operation is performed from SNMP.

Instead, perform the SNMP copy configuration with the following command to generate the keys:

     crypto key generation rsa general-keys modulus <size of key modulus>

Otherwise, power cycle the switch to clear the problem or execute the command  "clear configuration lock" to allow the reload command to execute to completion. Please note that the switch will immediately reload after "clear configuration lock" is executed.  The global configuration command "service internal" is a prerequisite for "clear configuration lock".

You can also contact TAC by opening a case to know any other workarounds are available to release the TFTP session without relaod.

-Thanks

Vinod

**Encourage Contributors. RATE Them.**

[Marcusbergh]

Thank you for the reply, Vinod. We figured, like you, that we should have used the crypto key generate rsa general-keys modulus. However while sending this command our test-switch doesn't seem to update its keys anyway, but it doesn't get stuck as it does with the crypto key generate rsa command.

I'll open a TAC case for this to ask Cisco if this is a bug or not. Even though the switch can't accept the command crypto key generate rsa it shouldn't make the whole tftp service for the switch unusable. and if they can't help me I guess we'll just reload all the switches. 

Thanks again, cheers!

[Vinod Arya]

Thanks for update. I think TAC should be able to give you fix, as they can contact development engineers directly on IOS side.

Else, they can file a new defect, which can help other to get rid of such issues.

In any case, please share your findings with the CSC team.

-Thanks

Vinod