Problem in ACS v 4.1

[ranjit123]

Dear All,

We had faced a problem in our ACS Server 4.1, it refused all the user connections for 15 mins and we were not able to authenticate through our TACACS username and password during this perdiod. After 15 mins things became normal

Below were the logs generated by the server during this period.

==========================================================================================

Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig GetIfTable size = 11192
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig, adpt Idx = 16777220, en adpt Idx = 16777219
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig, adpt Idx = 16777219, en adpt Idx = 16777219
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig ip < ip address>, mask 255.255.255.240, gateway < ip address>,
(Fri Oct 15 17:09:00 2010): Trying to get current administrator name...
(Fri Oct 15 17:09:01 2010): checking Administrator: admin...
(Fri Oct 15 17:09:01 2010): Administrator admin found
(Fri Oct 15 17:09:01 2010): Trying to get current administrator name...
(Fri Oct 15 17:09:01 2010): checking Administrator: admin...
(Fri Oct 15 17:09:01 2010): Administrator admin found
(Fri Oct 15 17:19:54 2010): Trying to get current administrator name...
(Fri Oct 15 17:19:54 2010): checking Administrator: admin...
(Fri Oct 15 17:19:54 2010): Administrator admin found
(Fri Oct 15 17:19:54 2010): Trying to get current administrator name...
(Fri Oct 15 17:19:54 2010): checking Administrator: admin...
(Fri Oct 15 17:19:54 2010): Administrator admin found

===========================================================================================

Regards,

Ranjit

[Tiago Antunes]

Do you have replication configured?

If yes, can you check if this 15 mins were during the replication process? If yes, it is expected.

Can you share with us the csmon.log file from the C:\Program Files\CiscoSecure ACS v4.2\CSMon\Logs directory?

Cheers,
Tiago

--

If this answers your question please mark the question as "answered" and rate it, so other users can easily find it.

[ranjit123]

Hi,

Thanks for your reply,replications is configured but its duration is 120 mins

please find the csmon.log file attached along with this mail.

ACS was implemented in 2008 and this problem occured the first time since then.

Regards,

Ranjit

[Tiago Antunes]

Hi,

Thanks but this is not the file I was asking for.

Can you share with us the "csmon.log" text file from the C:\Program Files\CiscoSecure ACS v4.2\CSMon\Logs directory?

Are you sure no one else configure replication?

Thanks,

Tiago

[ranjit123]

Hi Tiago,

It is a Appliance and LINUX based.

below is the snap shot of the Diagnostic logs avaiable on the box.

Please update me which logs do you want for reference.

Regards,

Ranjit

[Tiago Antunes]

Hi Ranjit,

Yes, indeed it is an appliance, however please be aware that it is Windows based even though you don't have access to the OS level.

Ok, so you can collect the package.cab file that you can obtain when you go to System Configuration -> Support -> Collect log file, and collect log files from previous x days making sure you catch the time of the outage.

Thanks,

Tiago

[ranjit123]

Hi!,

Please find the package.cab  file attached.

Regards,

Ranjit

[Tiago Antunes]

Hi Ranjit,

I see that the timestamp on your initial post isa bit deslocated in relation to the time on the ACS.

On the ACS i see that the authentications stopped between 10/15/2010 16:49:46 and 17:08:09:

...

CSMon 10/15/2010 16:49:46 A 0523 15836 CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 16:49:56 I 0718 15836 Auth Failure Retry 1 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:06 I 0718 15836 Auth Failure Retry 2 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:16 I 0718 15836 Auth Failure Retry 3 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:26 I 0718 15836 Auth Failure Retry 4 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:46 I 0747 15836 Confirmed alert on CSTacacs
CSMon 10/15/2010 16:50:46 E 0748 15836 CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 16:50:46 A 0641 43980 CSTacacs: State 6 0 Event Detected Level:4 Message:CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 17:06:36 A 0152 43980 Services were all restarted. Attempt 1.

CSMon 10/15/2010 17:08:09 I 0530 15836 CSTacacs: Authenticated
CSMon 10/15/2010 17:08:09 I 0653 43980 CSTacacs: State 0 6  No Problems

...

This tell us that something happened with the tacacs+ service that made the ACS restart the services to resume normal operations.

Unfortunately the TCS logs of the package.cab you sent do not include any logs prior to Oct 16th... Have you collected the package.cab for how many previous days? Please try to collect for previous 3 days, to make sure we get the logs of the 15th Oct.

Thanks,
Tiago

--

If this helps you and/or answers your question please mark the question as "answered" and rate it, so other users can easily find it.

[ranjit123]

Hi!,

Please check the same

Regards,

Ranjit

[Tiago Antunes]

Hi Ranjit,

Unfortunately, there is nothing there again...if you open the package.cab yourself, you will see that the file TCS.log contains no logs for the 15th Oct...

Sorry but without them there is no way to know why the tacacs+ service was failing...

Cheers,

Tiago

[ranjit123]

Hi!,

If replication happening i guess it will refuse all connections.

Regards,

Ranjit

[Tiago Antunes]

Hi,

Correct, with replication all services would stop, and not only TACACS+.

This was for sure something specific with TACACS+.

I would leave the LogLevel to FULL and monitor the ACS so that if it happens again, you can collect the pacage.cab imediately after the problem occurs and the needed logs will be there.

HTH,
Tiago

--

If  this helps you and/or answers your question please mark the question as  "answered" and/or rate it, so other users can easily find it.

[ranjit123]

HI!,

What do you suggest me to do next, so that we can capture the logs if the problem reoccurs again.

Regards,

Ranjit

[Tiago Antunes · Accepted solution]

As i wrote before:

I  would leave the LogLevel to FULL and monitor the ACS so that if it  happens again, you can collect the package.cab imediately after the  problem occurs and the needed logs will be there.

HTH,
Tiago

--

If   this helps you and/or answers your question please mark the question  as  "answered" and/or rate it, so other users can easily find it.

[samir ahmed]

We are having a same issue. here it is log.

find attached log for you reference.

(Thu Oct 25 10:29:23 2012): Trying to get current administrator name...

(Thu  Oct 25 10:29:23 2012): checking Administrator: admin...

(Thu Oct 25 10:29:23  2012): Administrator admin found

(Fri Nov 02 09:08:30 2012): Trying to get  current administrator name...

(Fri Nov 02 09:08:30 2012): checking  Administrator: admin...

(Fri Nov 02 09:08:30 2012): Administrator admin  found

(Fri Nov 02 09:08:30 2012): Trying to get current administrator  name...

(Fri Nov 02 09:08:30 2012): checking Administrator: admin...

(Fri Nov 02 09:08:30 2012): Administrator admin found

(Fri Nov 02  09:10:19 2012): Trying to get current administrator name...

(Fri Nov 02  09:10:19 2012): checking Administrator: admin...

(Fri Nov 02 09:10:19 2012):  Administrator admin found

(Fri Nov 02 09:10:19 2012): Trying to get current  administrator name...

(Fri Nov 02 09:10:19 2012): checking Administrator:  admin...

(Fri Nov 02 09:10:19 2012): Administrator admin found

(Fri Nov  02 09:12:24 2012): Trying to get current administrator name...

(Fri Nov 02  09:12:24 2012): checking Administrator: admin...

(Fri Nov 02 09:12:24 2012):  Administrator admin found

(Fri Nov 02 09:12:24 2012): Trying to get current  administrator name...

(Fri Nov 02 09:12:24 2012): checking Administrator:  admin...

(Fri Nov 02 09:12:24 2012): Administrator admin found

(Fri Nov  02 09:13:07 2012): Trying to get current administrator name...

(Fri Nov 02  09:13:07 2012): checking Administrator: admin...

(Fri Nov 02 09:13:07 2012):  Administrator admin found

(Fri Nov 02 09:13:07 2012): Trying to get current  administrator name...

(Fri Nov 02 09:13:07 2012): checking Administrator:  admin...

(Fri Nov 02 09:13:07 2012): Administrator admin found

(Fri Nov  02 09:15:11 2012): Trying to get current administrator name...

(Fri Nov 02  09:15:11 2012): checking Administrator: admin...

(Fri Nov 02 09:15:11 2012):  Administrator admin found

(Fri Nov 02 09:15:11 2012): Trying to get current  administrator name...

(Fri Nov 02 09:15:11 2012): checking Administrator:  admin...

(Fri Nov 02 09:15:11 2012): Administrator admin found

(Fri Nov  02 09:28:01 2012): Trying to get current administrator name...

(Fri Nov 02  09:28:01 2012): checking Administrator: admin...

(Fri Nov 02 09:28:01 2012):  Administrator admin found

(Fri Nov 02 09:28:01 2012): Trying to get current  administrator name...

(Fri Nov 02 09:28:01 2012): checking Administrator:  admin...

(Fri Nov 02 09:28:01 2012): Administrator admin found

(Wed Nov  07 20:49:33 2012): Trying to get current administrator name...

(Wed Nov 07  20:49:33 2012): checking Administrator: admin...

(Wed Nov 07 20:49:33 2012):  Administrator admin found

(Wed Nov 07 20:49:33 2012): Trying to get current  administrator name...

(Wed Nov 07 20:49:33 2012): checking Administrator:  admin...

(Wed Nov 07 20:49:33 2012): Administrator admin found

(Wed Nov  07 20:50:21 2012): Trying to get current administrator name...

(Wed Nov 07  20:50:21 2012): checking Administrator: admin...

(Wed Nov 07 20:50:21 2012):  Administrator admin found

(Wed Nov 07 20:50:21 2012): Trying to get current  administrator name...

(Wed Nov 07 20:50:21 2012): checking Administrator:  admin...

(Wed Nov 07 20:50:21 2012): Administrator admin found

(Mon Nov  12 15:48:06 2012): Trying to get current administrator name...

(Mon Nov 12  15:48:06 2012): checking Administrator: admin...

(Mon Nov 12 15:48:06 2012):  Administrator admin found

(Mon Nov 12 15:48:06 2012): Trying to get current  administrator name...

(Mon Nov 12 15:48:06 2012): checking Administrator:  admin...

(Mon Nov 12 15:48:06 2012): Administrator admin found

(Mon Nov  12 15:51:36 2012): Trying to get current administrator name...

(Mon Nov 12  15:51:36 2012): checking Administrator: admin...

(Mon Nov 12 15:51:36 2012):  Administrator admin found

(Mon Nov 12 15:51:36 2012): Trying to get current  administrator name...

(Mon Nov 12 15:51:36 2012): checking Administrator:  admin...

(Mon Nov 12 15:51:36 2012): Administrator admin found

(Tue Nov  20 14:17:34 2012): Trying to get current administrator name...

(Tue Nov 20  14:17:34 2012): checking Administrator: admin...

(Tue Nov 20 14:17:34 2012):  Administrator admin found

(Tue Nov 20 14:17:34 2012): Trying to get current  administrator name...

(Tue Nov 20 14:17:34 2012): checking Administrator:  admin...

(Tue Nov 20 14:17:34 2012): Administrator admin found

(Wed Nov  21 15:29:00 2012): Trying to get current administrator name...

(Wed Nov 21  15:29:00 2012): checking Administrator: admin...

(Wed Nov 21 15:29:00 2012):  Administrator admin found

(Wed Nov 21 15:29:00 2012): Trying to get current  administrator name...

(Wed Nov 21 15:29:00 2012): checking Administrator:  admin...

(Wed Nov 21 15:29:00 2012): Administrator admin found

(Wed Nov  21 18:08:26 2012): Trying to get current administrator name...

(Wed Nov 21  18:08:26 2012): checking Administrator: admin...

(Wed Nov 21 18:08:26 2012):  Administrator admin found

(Wed Nov 21 18:08:26 2012): Trying to get current  administrator name...

(Wed Nov 21 18:08:26 2012): checking Administrator:  admin...

(Wed Nov 21 18:08:26 2012): Administrator admin found

(Fri Nov  23 12:48:34 2012): Trying to get current administrator name...

(Fri Nov 23  12:48:34 2012): checking Administrator: admin...

(Fri Nov 23 12:48:34 2012):  Administrator admin found

(Fri Nov 23 12:48:34 2012): Trying to get current  administrator name...

(Fri Nov 23 12:48:34 2012): checking Administrator:  admin...

(Fri Nov 23 12:48:34 2012): Administrator admin found

(Fri Nov  23 12:51:35 2012): Trying to get current administrator name...

(Fri Nov 23  12:51:35 2012): checking Administrator: admin...

(Fri Nov 23 12:51:35 2012):  Administrator admin found

(Fri Nov 23 12:51:35 2012): Trying to get current  administrator name...

(Fri Nov 23 12:51:35 2012): checking Administrator:  admin...

(Fri Nov 23 12:51:35 2012): Administrator admin found

(Fri Nov  23 12:52:01 2012): Trying to get current administrator name...

(Fri Nov 23  12:52:01 2012): checking Administrator: admin...

(Fri Nov 23 12:52:01 2012):  Administrator admin found

(Fri Nov 23 12:52:01 2012): Trying to get current  administrator name...

(Fri Nov 23 12:52:01 2012): checking Administrator:  admin...

(Fri Nov 23 12:52:01 2012): Administrator admin found

(Fri Nov  23 14:15:11 2012): Trying to get current administrator name...

(Fri Nov 23  14:15:11 2012): checking Administrator: admin...

(Fri Nov 23 14:15:11 2012):  Administrator admin found

(Fri Nov 23 14:15:11 2012): Trying to get current  administrator name...

(Fri Nov 23 14:15:11 2012): checking Administrator:  admin...

(Fri Nov 23 14:15:11 2012): Administrator admin found

(Fri Nov  23 14:15:29 2012): Info: GetApplNICConfig GetIfTable size = 11192

(Fri Nov  23 14:15:29 2012): Info: GetApplNICConfig, adpt Idx = 65540, en adpt Idx = 65539

(Fri Nov 23 14:15:29 2012): Info: GetApplNICConfig, adpt Idx = 65539, en  adpt Idx = 65539

(Fri Nov 23 14:15:29 2012): Info: GetApplNICConfig, adpt  Idx = 65540, en adpt Idx = 65540

(Fri Nov 23 14:15:29 2012): Info:  GetApplNICConfig ip 10.212.15.1, mask 255.255.240.0, gateway 10.212.0.1

(Sun  Nov 25 19:58:41 2012): Trying to get current administrator name...

(Sun Nov  25 19:58:41 2012): checking Administrator: admin...

(Sun Nov 25 19:58:41  2012): Administrator admin found

(Sun Nov 25 19:58:41 2012): Trying to get  current administrator name...

(Sun Nov 25 19:58:41 2012): checking  Administrator: admin...

(Sun Nov 25 19:58:41 2012): Administrator admin  found

(Sun Nov 25 20:01:34 2012): Trying to get current administrator  name...

(Sun Nov 25 20:01:34 2012): checking Administrator: admin...

(Sun Nov 25 20:01:34 2012): Administrator admin found

(Sun Nov 25  20:01:34 2012): Trying to get current administrator name...

(Sun Nov 25  20:01:34 2012): checking Administrator: admin...

(Sun Nov 25 20:01:34 2012):  Administrator admin found

(Sun Nov 25 20:10:31 2012): Trying to get current  administrator name...

(Sun Nov 25 20:10:31 2012): checking Administrator:  admin...

(Sun Nov 25 20:10:31 2012): Administrator admin found

(Sun Nov  25 20:10:31 2012): Trying to get current administrator name...

(Sun Nov 25  20:10:31 2012): checking Administrator: admin...

(Sun Nov 25 20:10:31 2012):  Administrator admin found

(Fri Nov 30 10:44:28 2012): Info: GetApplNICConfig  GetIfTable size = 11192

(Fri Nov 30 10:44:28 2012): Info: GetApplNICConfig,  adpt Idx = 65540, en adpt Idx = 65539

(Fri Nov 30 10:44:28 2012): Info:  GetApplNICConfig, adpt Idx = 65539, en adpt Idx = 65539

(Fri Nov 30 10:44:28  2012): Info: GetApplNICConfig, adpt Idx = 65540, en adpt Idx = 65540

(Fri  Nov 30 10:44:28 2012): Info: GetApplNICConfig ip 10.212.15.1, mask  255.255.240.0, gateway 10.212.0.1

(Fri Nov 30 10:46:16 2012):  ApplGetSnmpConfig: service SNMP is running 1