switch 3560 and acs 4.2: fail authentication after reboot

ngo duyen · ‎08-30-2010

we use acs to aaa with network devices.

we have 4 swich 3560 with same problem: after around 10 minutes from reboot switch, I cant login these switchs. But from these switch, i still ping ACS server. IF I reboot again, the same thing happen.

Could you help me, what s the matter with these switchs?

The config for AAA on each swich:

!
aaa new-model
!
aaa authentication login TACACS+ group tacacs+ local
!
aaa authorization commands 15 AAA group tacacs+ local
!
aaa accounting exec AAA start-stop group tacacs+
aaa accounting network AAA start-stop group tacacs+
aaa accounting connection AAA start-stop group tacacs+
aaa accounting system default start-stop group tacacs+
!
tacacs-server host x.x.x.x
tacacs-server directed-request
tacacs-server key 7 02050D480809
!
line con 0
accounting connection AAA
accounting commands 15 AAA
accounting exec AAA
line vty 0 4
authorization commands 15 AAA
accounting connection AAA
accounting commands 15 AAA
accounting exec AAA
login authentication TACACS+
transport input telnet
line vty 5 15
login authentication TACACS+
!

Javier Henderson · ‎08-31-2010

What is the reason seen on ACS' reports for failing the authentication request?

Please enable:

debug aaa authentication

debug tacacs

Then capture the output while the switch is in the failed state and post it here.

ngo duyen · ‎08-31-2010

Nothing special in ACS

my debug on sw:

S3560-04#debug aaa authentication
AAA Authentication debugging is on
S3560-04#
S3560-04#
S3560-04#
1w5d: AAA/AUTHEN (2070922190): status = ERROR
1w5d: AAA/AUTHEN/START (2070922190): Method=LOCAL
1w5d: AAA/AUTHEN (2070922190): status = GETUSER
S3560-04#
S3560-04#
S3560-04#
S3560-04#debug aaa authentication
AAA Authentication debugging is on
S3560-04#
1w5d: AAA/AUTHEN/ABORT: (2070922190) because Login timed out.
1w5d: AAA/MEMORY: free_user (0x33B00B8) user='NULL' ruser='NULL' port='tty1' rem_addr='10.0.0.63' authen_type=ASCII service=LOGIN priv=1
1w5d: AAA: parse name=tty1 idb type=-1 tty=-1
1w5d: AAA: name=tty1 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=1 channel=0
1w5d: AAA/MEMORY: create_user (0x3362B98) user='NULL' ruser='NULL' ds0=0 port='tty1' rem_addr='10.0.0.63' authen_type=ASCII service=LOGIN priv=1 initial_task_id='0', vrf= (id=0)
1w5d: AAA/AUTHEN/START (1041751841): port='tty1' list='TACAS+' action=LOGIN service=LOGIN
1w5d: AAA/AUTHEN/START (1041751841): found list TACAS+
1w5d: AAA/AUTHEN/START (1041751841): Method=tacacs+ (tacacs+)
1w5d: TAC+: send AUTHEN/START packet ver=192 id=1041751841
1w5d: AAA/AUTHEN (1041751841): status = ERROR
1w5d: AAA/AUTHEN/START (1041751841): Method=LOCAL
1w5d: AAA/AUTHEN (1041751841): status = GETUSER
1w5d: AAA/AUTHEN/CONT (1041751841): continue_login (user='(undef)')
1w5d: AAA/AUTHEN (1041751841): status = GETUSER
1w5d: AAA/AUTHEN/CONT (1041751841): Method=LOCAL
1w5d: AAA/AUTHEN (1041751841): status = GETPASS
1w5d: AAA/AUTHEN/CONT (1041751841): continue_login (user='tester')
1w5d: AAA/AUTHEN (1041751841): status = GETPASS
1w5d: AAA/AUTHEN/CONT (1041751841): Method=LOCAL
1w5d: AAA/AUTHEN (1041751841): User not found
1w5d: AAA/AUTHEN (1041751841): status = FAIL
1w5d: AAA/AUTHEN/ABORT: (1041751841) because Unknown.
1w5d: AAA/MEMORY: free_user_quiet (0x3362B98) user='tester' ruser='NULL' port='tty1' rem_addr='10.0.0.63' authen_type=1 service=1 priv=1
1w5d: AAA: parse name=tty1 idb type=-1 tty=-1
1w5d: AAA: name=tty1 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=1 channel=0
1w5d: AAA/MEMORY: create_user (0x34B7470) user='NULL' ruser='NULL' ds0=0 port='tty1' rem_addr='10.0.0.63' authen_type=ASCII service=LOGIN priv=1 initial_task_id='0', vrf= (id=0)
1w5d: AAA/AUTHEN/START (9466938): port='tty1' list='TACAS+' action=LOGIN service=LOGIN
1w5d: AAA/AUTHEN/START (9466938): found list TACAS+
1w5d: AAA/AUTHEN/START (9466938): Method=tacacs+ (tacacs+)
1w5d: TAC+: send AUTHEN/START packet ver=192 id=9466938
1w5d: AAA/AUTHEN (9466938): status = ERROR
1w5d: AAA/AUTHEN/START (9466938): Method=LOCAL
1w5d: AAA/AUTHEN (9466938): status = GETUSER
S3560-04#
1w5d: AAA/AUTHEN/ABORT: (9466938) because Login timed out.
1w5d: AAA/MEMORY: free_user_quiet (0x34B7470) user='NULL' ruser='NULL' port='tty1' rem_addr='10.0.0.63' authen_type=1 service=1 priv=1
1w5d: AAA: parse name=tty1 idb type=-1 tty=-1
1w5d: AAA: name=tty1 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=1 channel=0
1w5d: AAA/MEMORY: create_user (0x34B8E10) user='NULL' ruser='NULL' ds0=0 port='tty1' rem_addr='10.0.0.63' authen_type=ASCII service=LOGIN priv=1 initial_task_id='0', vrf= (id=0)
1w5d: AAA/AUTHEN/START (2867507997): port='tty1' list='TACAS+' action=LOGIN service=LOGIN
1w5d: AAA/AUTHEN/START (2867507997): found list TACAS+
1w5d: AAA/AUTHEN/START (2867507997): Method=tacacs+ (tacacs+)
1w5d: TAC+: send AUTHEN/START packet ver=192 id=2867507997
1w5d: AAA/AUTHEN (2867507997): status = ERROR
1w5d: AAA/AUTHEN/START (2867507997): Method=LOCAL
1w5d: AAA/AUTHEN (2867507997): status = GETUSER
S3560-04#
S3560-04#
S3560-04#
S3560-04#
S3560-04#

1w5d: AAA/AUTHEN/ABORT: (2867507997) because Login timed out.
1w5d: AAA/MEMORY: free_user (0x34B8E10) user='NULL' ruser='NULL' port='tty1' rem_addr='10.0.0.63' authen_type=ASCII service=LOGIN priv=1

Javier Henderson · ‎09-01-2010

1w5d: AAA/AUTHEN/START (1041751841): port='tty1' list='TACAS+' action=LOGIN service=LOGIN
1w5d: AAA/AUTHEN/START (1041751841): found list TACAS+
1w5d: AAA/AUTHEN/START (1041751841): Method=tacacs+ (tacacs+)
1w5d: TAC+: send AUTHEN/START packet ver=192 id=1041751841
1w5d: AAA/AUTHEN (1041751841): status = ERROR

So the switch can't talk to the ACS server anymore for some reason. Since it works at first, I assume that the TACACS+ shared secret is correct.

Can you ping the ACS server after the authentications start to fail?

ngo duyen · ‎09-07-2010

thank your help, I have topology and more detail info.

sw3560-01 havent got this error. All other sw3560 have this error.

when error happen, I still ping server 10.a.b.22 (the same subnet with ACS, ACS doesnt allow ping it)

And at that time, I can access that switch ( 3,4,5,6,7) by using a local username/password when I connect with a vlan in the system.

All thing work well, only thing that these switchs lost authentication after short time. I really want to know what happen with these switchs

Javier Henderson · ‎09-08-2010

When the problem happens, can you telnet to port 49 on the ACS server from the affected switches?

ngo duyen · ‎09-08-2010

Tacacs+ port?

I try telnet ip_of_ACS and port 2002

and it work for short time after reboot and doesnt work when it get error.

Javier Henderson · ‎09-08-2010

Yes, TCP/49 is the TACACS+ port.

I would suspect a networking issue, and concentrate troubleshooting there. Could the firewall shown in the diagram that you uploaded be a factor?

ngo duyen · ‎09-08-2010

we get that error around 2 month ago, before that time the system work well for very long time.

If that s a firewall problem, I dont know why it happen after it work well for few minutes. Is there any kind of dynamic ACL?

After reboot it work again and ...

I thinkink about ip default-gateway. all L2 switch work without default-gateway. Is int vlan1 enough for this topology?

I read somewhere and they said that, we need ip default-gateway when we want to manage from other subnet.

ngo duyen · ‎09-12-2010

pls give me some ideas to resol this problem!

I also try change IOS but nothing change

ngo duyen · ‎09-12-2010

I have checked ACS, it work well. And we have many other device using ACS.

port 49, 2002 work in short time and doesnt work after that.

Firewall I use: permit ip any for testing.

From fail switchs, I can ping 10.a.b.22 without problem.

when I change L2 sw to L3 sw using these command, AAA work well:

ip routing

interface Loopback10
ip address 10.x.x.x 255.255.255.255

ip route 0.0.0.0 0.0.0.0 10.x.y.1
ip tacacs source-interface Loopback10

10.x.y.1 is ip address of L3 switch

Pls help me. thank you

ngo duyen · ‎09-14-2010

I add

ip default-gateway 10.x.y.1

and .... it works well

But I dont underscant why it work around 3 minustes after reboot without this command