Solved: ACS and "key mismatch"

Erik Molenaar · ‎01-07-2009

Today our ACS (ver 4.1 on windows 2003 server) suddenly stopped working. We use it to authenticate and authorize key access to switches via tacacs+. I see "key mismatch" errors in ACS, but the shared secret is the same on both sides (switch vs. ACS), so that is not an issue.

A "debug tacacs" on a switch shows:

Jan 7 21:32:00: Telnet2: 1 1 251 1

Jan 7 21:32:00: TCP2: Telnet sent WILL ECHO (1)

Jan 7 21:32:00: Telnet2: 2 2 251 3

Jan 7 21:32:00: TCP2: Telnet sent WILL SUPPRESS-GA (3)

Jan 7 21:32:00: Telnet2: 80000 80000 253 24

Jan 7 21:32:00: TCP2: Telnet sent DO TTY-TYPE (24)

Jan 7 21:32:00: Telnet2: 10000000 10000000 253 31

Jan 7 21:32:00: TCP2: Telnet sent DO WINDOW-SIZE (31)

Jan 7 21:32:00: TAC+: send AUTHEN/START packet ver=192 id=3650164881

Jan 7 21:32:00: TAC+: Using default tacacs server-group "tacacs+" list.

Jan 7 21:32:00: TAC+: 10.1.2.2(3650164881) AUTHEN/START/LOGIN/ASCII queued

Jan 7 21:32:05: TAC+: 10.1.2.2 (3650164881) AUTHEN/START/LOGIN/ASCII -- TIMED OUT

Jan 7 21:32:05: TAC+: (3650164881) AUTHEN/START/LOGIN/ASCII processed

Jan 7 21:32:05: TAC+: Using default tacacs server-group "tacacs+" list.

Jan 7 21:32:05: TAC+: 10.10.10.1 (3650164881) AUTHEN/START/LOGIN/ASCII queued

Jan 7 21:32:10: TAC+: 10.10.10.1 (3650164881) AUTHEN/START/LOGIN/ASCII -- TIMED OUT

Jan 7 21:32:10: TAC+: (3650164881) AUTHEN/START/LOGIN/ASCII processed

Jan 7 21:32:10: TAC+: Using default tacacs server-group "tacacs+" list.

(etc...)

There are two ACS servers, they sync and both have the same problem.

Reachability is not the problem.

What else should i check for?

Erik

Richard Burts · ‎01-29-2009

Mohamed

Since this is a new issue you might get more and better responses if you start a new thread rather than adding what might appear to be more comments about an onging issue.

However since you did ask the question here, I have an observation and a suggestion. This line of output:

QA: TAC+: Invalid AUTHEN/START/LOGIN/ASCII packet (check keys).

suggests that something (probably the shared key for device authentication) is not properly in sync between your device and the TACACS server.

My suggestion would be to look in the Failed Attempts report on the TACACS server and see if you see the authentication attempt from this device. If it is there what does the TACACS server say about the cause of the failure?

HTH

Rick

HTH

Rick

View solution in original post

sachinraja · ‎01-07-2009

Erik

did you try restarting the service, shut and restart it ? the debug messages dont show a key mismatch ? do u have any other debugs with you which shows that ? From the debug posted by you, i see that the authentication times out, for some reason ! Did you check for the system logs in your ACS server ? Is your ACS server behind a firewall etc, and the FW had some issues ? do a "show tacacs" and see if you have failed/ successful connections.. if the failed connection increases, then you need to run some more debugs to find out ! probably debug aaa authentication etc...

Hope this helps.. let us know..

Regards

Raj

Erik Molenaar · ‎01-09-2009

Hi,

Problem solved!

We just discovered by accident what caused the problem.

Someone installed 'The Dude', a freeware tool that sketches out and tests ip's in the network.

It does this by pinging and scanning whole subnets. An option is to test connectivity by letting the application set up a telnet connection, which is dropped as soon as a reply is received.

ACS records show no trace of it but apparently it generates enough traffic to make ACS stop working!

And there are no signs of this in the event viewer!

This detail will also make you shiver: the process of setting up and breaking down Telnet connections will continue, even if 'The dude' is not running! Only uninstalling did the trick.

This is the output of debug commands on a switch:

switch#sh deb

General OS:

TACACS access control debugging is on

AAA Authentication debugging is on

Jan 9 13:31:27: AAA: parse name=tty1 idb type=-1 tty=-1

Jan 9 13:31:27: AAA: name=tty1 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=1 channel=0

Jan 9 13:31:27: AAA/MEMORY: create_user (0x80F87874) user='' ruser='' port='tty1' rem_addr='xxx.yyy.zzz.219' authen_type=ASCII service=LOGIN priv=1

Jan 9 13:31:27: AAA/AUTHEN/START (448648205): port='tty1' list='' action=LOGIN service=LOGIN

Jan 9 13:31:27: AAA/AUTHEN/START (448648205): using "default" list

Jan 9 13:31:27: AAA/AUTHEN/START (448648205): Method=tacacs+ (tacacs+)

Jan 9 13:31:27: TAC+: send AUTHEN/START packet ver=192 id=448648205

Jan 9 13:31:27: TAC+: Using default tacacs server-group "tacacs+" list.

Jan 9 13:31:27: TAC+: 10.xx.yy.zz (448648205) AUTHEN/START/LOGIN/ASCII queued

Jan 9 13:31:27: TAC+: (448648205) AUTHEN/START/LOGIN/ASCII processed

Jan 9 13:31:27: TAC+: ver=192 id=448648205 received AUTHEN status = GETUSER

Jan 9 13:31:27: AAA/AUTHEN (448648205): status = GETUSER

Jan 9 13:31:27: AAA/AUTHEN/ABORT: (448648205) because Carrier dropped.

Jan 9 13:31:27: TAC+: send abort reason=Carrier dropped

Jan 9 13:31:27: TAC+: 10.xx.yy.zz (448648205) AUTHEN/CONT queued

Jan 9 13:31:27: TAC+: (448648205) AUTHEN/CONT processed

Jan 9 13:31:27: AAA/MEMORY: free_user (0x80F87874) user='' ruser='' port='tty1' rem_addr='xxx.yyy.zzz.219' authen_type=ASCII service=LOGIN priv=1

...and so on, for every 30 seconds, for every switch in the network.

So why the heck is ACS not capable of handling those loads? And why does it not complain!!? It's unbelievable that someone using a freeware tool can bring down an expensive and crucial application like ACS!

Mohamed Shafeer Kunhu · ‎01-28-2009

This the debug o/p I'm getting when I try to login. Can anyone help me telling why is this happening?

Jan 29 10:00:47 QA: AAA: parse name= idb type=-1 tty=-1

Jan 29 10:00:47 QA: AAA/MEMORY: create_user (0x1E61F21C) user='a' ruser='NULL' ds0=0 port='' rem_addr='NULL' authen_type=ASCII service=LOGIN priv=1 initial_task_id='0', vrf= (id=0)

Jan 29 10:00:47 QA: TAC+: send AUTHEN/START packet ver=192 id=1540374853

Jan 29 10:00:47 QA: TAC+: Using default tacacs server-group "tacacs+" list.

Jan 29 10:00:47 QA: TAC+: Opening TCP/IP to 172.19.62.10/49 timeout=1

Jan 29 10:00:47 QA: TAC+: Opened TCP/IP handle 0x1341DC74 to 172.19.62.10/49 using source 172.19.0.7

Jan 29 10:00:47 QA: TAC+: 172.19.62.10 (1540374853) AUTHEN/START/LOGIN/ASCII queued

Jan 29 10:00:47 QA: TAC+: (1540374853) AUTHEN/START/LOGIN/ASCII processed

Jan 29 10:00:47 QA: TAC+: received bad AUTHEN packet: length = 6, expected 53578

Jan 29 10:00:47 QA: TAC+: Invalid AUTHEN/START/LOGIN/ASCII packet (check keys).

Jan 29 10:00:47 QA: TAC+: Closing TCP/IP 0x1341DC74 connection to 172.19.62.10/49

Jan 29 10:00:47 QA: TAC+: Using default tacacs server-group "tacacs+" list.

Jan 29 10:00:47 QA: AAA/MEMORY: free_user (0x1E61F21C) user='a' ruser='NULL' port='' rem_addr='NULL' authen_type=ASCII service=LOGIN priv=1 vrf= (id=0)

Richard Burts · ‎01-29-2009

Mohamed

Since this is a new issue you might get more and better responses if you start a new thread rather than adding what might appear to be more comments about an onging issue.

However since you did ask the question here, I have an observation and a suggestion. This line of output:

QA: TAC+: Invalid AUTHEN/START/LOGIN/ASCII packet (check keys).

suggests that something (probably the shared key for device authentication) is not properly in sync between your device and the TACACS server.

My suggestion would be to look in the Failed Attempts report on the TACACS server and see if you see the authentication attempt from this device. If it is there what does the TACACS server say about the cause of the failure?

HTH

Rick

HTH

Rick

abolampe · ‎01-20-2012

Just one thing I noticed, in ACS 4.2 you can define the shared secret in both the 'Network Device Group' and for each individual client. If you define it in the Network Device Group, ACS appears to ignore the key you define under the individual client, and inherits the one from the group. This is counter-intuitive, and I'm sure I'm not the only one daft enough to be caught out by it.

Adam

Jatin Katyal · ‎01-20-2012

Exactly, you're not the only one but quite common error we see in most of the scenarios.

This is deafult behaviour and well documented in the ACS end user guide.

The key defined on the NDG level over-rides the key at the AAA client level.

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.2/user/guide/NetCfg.html#wp342738

"Each device that is assigned to the Network Device Group will use the shared key that you enter here. The key that was assigned to the device when it was added to the system is ignored. If the key entry is null, the AAA client key is used."

Regards,

Jatin

Do rate helpful posts-

~Jatin