Auth Failure Traps

Unanswered Question
Feb 24th, 2010

After i changed snmp strings on our network devices , I see a list of devices with Auth Failure Traps on Syslog server.

Ive check the snmp credential strings on CW for each device and they're correct.

This is the error message on my syslog server:

mm-dd-yyyy    11:23:16    Local0.Info 4 0  Authentication failure 1

This message wasnt there before i re-new the snmp community string. After I chnage the snmp string on my routers and switches, I a lots of traps on my syslog server.

How can I stop this?

Thank you for your help


I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Joe Clarke Thu, 02/25/2010 - 17:13

If you are absolutely sure DCR is correct for these devices, then check your CS Discovery SNMP settings under Common Services > Device and Credentials > Device Discovery > Discovery Settings.  Make sure those community strings are correct as well.  If you still can't figure it out, start a sniffer trace filtering on udp/161 traffic between the LMS server and the device, and check to see what SNMP objects are being polled using the incorrect community string.  That should help narrow down the problem application.

baotran09 Thu, 02/25/2010 - 18:10

Hi Clarke,

Thanks for your help.

I used Ethereal to sniff the packets. The result shows that CW is polling using the OLD snmp string even though I have updated the DCR database with the new strings (ive verified the snmp string with device credential and doublecheck in DCR/exports to convirm the device credentialg - DCR db has the new strings). My question is why it uses the old string to poll? I've deleted the whole database and import it again but still seeing "snmp authentication failure" on syslog server.

How can I track which aplication is doing the snmp polling (I have RME, IPM, DFM and CS all on one server)

How can I stop the snmp polling completely? Ive stopped daemon and change aaa to local but no success.

Joe Clarke Thu, 02/25/2010 - 21:27

If you've stopped Daemon Manager, and the polling continues, then the problem is not LMS.  You must have some other application installed on the server which is doing the polling.

baotran09 Thu, 02/25/2010 - 22:20

There's no other applciation, only Ciscoworks on this server. Etherreal showing missing MIB (see atatachment)

All I did was changing the snmp community string on routers and switches and removed the one one.

What else can I check?

Martin Ermel Fri, 02/26/2010 - 00:26

exactly where do you see the authentication failure ? Is it on the LMS server? In which application? Is it on another server, which application do you use to get this message?
If you have stopped the LMS daemons (net stop crmdmgtd on windows or \etc\init.d\dmgtd stop on solaris), open 2 terminal sessions to one of the devices and  issue the following commands in the first sessions:
term mon
debug snmp packets

use the second session to easily disable the debugging if the message ouput in the first session is too much. Use this command:
undebug all

and you should see the source of the request. Is it still pointing to LMS?

baotran09 Fri, 02/26/2010 - 02:40

Hi Mermel,

Thank you for your rhelp.

I see the 'authentication failure message' on the syslog server, not on LMS server. Im using Kiwi syslog service manager to capture these messages.

As advised, tos atrtw ith I enable daemon and performed the debug command, I could see the source of the request (Ciscoworks server) after I entered 'debugged snmp packets'

Feb 25 19:36:45.375 AWST: SNMP: Packet received via UDP from (CWserver) on Serial0/0/1
Feb 25 19:36:45.379 AWST: SNMP: Queuing packet to (syslogserver)
Feb 25 19:36:45.379 AWST:
Outgoing SNMP packet
Feb 25 19:36:45.379 AWST: v1 packet
Feb 25 19:36:45.379 AWST: community string: readstring
Feb 25 19:36:45.379 AWST: SNMP: V1 Trap, ent snmpTraps, addr (remote router loopback interface), gentrap 4, spectrap 0
lsystem.5.0 =
ciscoMgmt.412. = 1
ciscoMgmt.412. =
Feb 25 19:36:45.631 AWST: SNMP: Packet sent via UDP to

How can I stop this?

Joe Clarke Fri, 02/26/2010 - 04:54

Based on the packets you captured previously, it could be DFM or HUM doing the polling.  Try shutting down DfmServer and DfmServer1, and see if the polling stops.  If not, shutdown UPMProcess.

But just to be clear, if you do "net stop crmdmgtd" does the polling stop?

baotran09 Fri, 02/26/2010 - 07:43

I did "net stop crmdmgtd" but the polling didnt stop. I have only 1 ciscoworks server and no other application running in the background other than Ciscoworks.

Correction, I mentioned CW used old snmp string, I was wrong. It uses a new string when polling, but I dont know why it giving me a authentication failure. I've check thwe string on my routers and switches and again on CW.

I shutdown dfmserver and dfmserver 1 but still see the polling, my server doesnt have HUM so there's no UPMprocess option to shutdown

Every minute I get about 100 traps, I have 1000+ routers and switches, so basically sonner or later it going to killing my syslog server.

Attached is the screen capture of the repated polling message on my syslog server

Thank you for your input.

Joe Clarke Fri, 02/26/2010 - 10:30

With Daemon Manager shutdown.  Post a list of all processes running on the server.

baotran09 Sat, 02/27/2010 - 09:39

Hi Clarke,

Document 1 contains a list of processes with with daemon shutdown.

Document 2 contains a list of processes with daemon turned on.Im seeing alot of cwjava.exe processes, is it normal?

Should sm_server processes be stopped when I stopped the daemons? Is this as bug (CSCsx23656-DFM3.2: sm_server does not stop when daemons are stopped. ?)

Which processes can I kill in order to stop the snmp polling?

Joe Clarke Sat, 02/27/2010 - 10:20

Yes, I mean reboot the server.  The root cause is that the DFM polling processes are not shutting down when Daemon Manager goes down.

baotran09 Mon, 03/01/2010 - 20:27

I've applied the patch and rebooted the server but the auth failure re-appeared.

C:\Program Files\CSCOpx\bin>perl

The patch is getting installed.....

The CiscoWorks Daemon Manager service is stopping..
The CiscoWorks Daemon Manager service was stopped successfully.

The CiscoWorks Daemon Manager service is starting.
The CiscoWorks Daemon Manager service was started successfully.

Generating the information file for the patch
Patch Installation was successful...

I notice that after rebooting the server, the auth failure went away for a few minutes, so its definitely the CW server that sending out uneccessary polling!

Attached is the server processes.

Is there another patch?


Joe Clarke Mon, 03/01/2010 - 22:35

If you shutdown Daemon Manager, is the server still polling these devices?

Joe Clarke Tue, 03/02/2010 - 09:30

To be clear, the polling is generating authFail traps still, or are the devices being polled with the correct SNMP credentials, and you just want polling to stop?

baotran09 Tue, 03/02/2010 - 09:51

Hi Joe,

Yes, the polling is still generating  authFail traps.

Yes, the devices being polled with the  correct SNMP credentials.

Yes I want the polling to  stop.

There are many cwjava.exe and sm_server.exe process when Daemon is enabled. Is this normal?

Can you see any issues with the server processes.

Joe Clarke Tue, 03/02/2010 - 11:28

Polling with the correct credentials will not cause authFailures.  If you want the polling to stop altogether, just remove the devices in question from DCR.  Yes, when Daemon Manager is running, there should be four sm_server processes, and quite a few cwjava processes.

baotran09 Wed, 03/03/2010 - 01:54

How do I verify whether the device has correct credentials apart from running one rme snmp credential checks?

Attached is the authentication error on kiwi syslog (1- kiwisyslog error.JPG)

When i verified the snmp credential check the read., write and ssh are all oK. 2 - snmp credential verified on RME.JPG

I want to stop all polling on ciscoworks, can you give me a list of the polling availbe on CW?

Joe Clarke Wed, 03/03/2010 - 09:10

If you remove the devices from DCR, and stop running Common Services Discovery, nothing in LMS will poll the devices.  Periodic polling is available from Common Services (with Discovery and DCR device status polling), RME, DFM, Campus, IPM, and HUM.  But all of those apps feed off of DCR to determine what devices to poll.

baotran09 Wed, 03/03/2010 - 09:27

Can you direct me to a link so I can download and upgrade the latest software for all of my CW apps?

Joe Clarke Wed, 03/03/2010 - 09:31

What versions of the LMS applications do you currently have installed?

baotran09 Wed, 03/03/2010 - 09:49

I'd like to upgrade all the apps to the latest/stable version, could you

please advise?

Name                                             Version   
CiscoWorks Common Services        3.3.0   
Campus Manager                            5.2.0   
CiscoView                                      6.1.9   
CiscoWorks Assistant                     1.2.0   
Device Fault Manager                      3.2.0   
Internetwork Performance Monitor     4.2.0   
Integration Utility                            1.9.0   
LMS Portal                                    1.2.0   
Resource Manager Essentials        4.3.1

Thanks Joe

baotran09 Wed, 03/03/2010 - 21:41

Hi Joe,

The root cause of authentication failure messages was due to dfmserver. When I stop it, the message disappeared.

Startup:Started automatically at boot.

Before applying the patch, when I shutdown dfmserver, I could still see the polling. After applying the patch, the polling stop.

There are only 2 patches for DFM. I have also applied fix CSCta56151.

Patches installed

Patch NameVersionInstalled Date
CSCtb87449-0002 Mar 2010, 11:28:07 WST
CSCta56151-0004 Mar 2010, 14:18:46 WST

Any more tips Joe?

Joe Clarke Wed, 03/03/2010 - 22:20

No, there is no known issue here.  It could be that when you changed the community strings in DCR, DFM was not updated (because of the bug you were encountering previously).  Remove the devices which are generating the authFail from DFM.  Make sure that DfmServer and DfmServer1 are running when you do this.  Then, re-add them, and that should hopefully sync everything.  DFM will still be polling the devices, but the authFail messages should stop.

baotran09 Thu, 03/04/2010 - 07:35

Thanks Joe, you're a legend.

The errors have gone. I had to reinitialize the dfm db.


This Discussion