Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

Auth Failure Traps

After i changed snmp strings on our network devices , I see a list of devices with Auth Failure Traps on Syslog server.

Ive check the snmp credential strings on CW for each device and they're correct.

This is the error message on my syslog server:

mm-dd-yyyy    11:23:16    Local0.Info    10.1.1.1    10.1.1.2.150 4 0  Authentication failure 10.1.1.254(CiscoWorks) 1 10.1.1.254(CiscoWorks)

This message wasnt there before i re-new the snmp community string. After I chnage the snmp string on my routers and switches, I a lots of traps on my syslog server.

How can I stop this?

Thank you for your help

Thanks

29 REPLIES
New Member

Re: Auth Failure Traps

Any ideas?

please advise

Cisco Employee

Re: Auth Failure Traps

If you are absolutely sure DCR is correct for these devices, then check your CS Discovery SNMP settings under Common Services > Device and Credentials > Device Discovery > Discovery Settings.  Make sure those community strings are correct as well.  If you still can't figure it out, start a sniffer trace filtering on udp/161 traffic between the LMS server and the device, and check to see what SNMP objects are being polled using the incorrect community string.  That should help narrow down the problem application.

New Member

Re: Auth Failure Traps

Hi Clarke,

Thanks for your help.

I used Ethereal to sniff the packets. The result shows that CW is polling using the OLD snmp string even though I have updated the DCR database with the new strings (ive verified the snmp string with device credential and doublecheck in DCR/exports to convirm the device credentialg - DCR db has the new strings). My question is why it uses the old string to poll? I've deleted the whole database and import it again but still seeing "snmp authentication failure" on syslog server.

How can I track which aplication is doing the snmp polling (I have RME, IPM, DFM and CS all on one server)

How can I stop the snmp polling completely? Ive stopped daemon and change aaa to local but no success.

Cisco Employee

Re: Auth Failure Traps

If you've stopped Daemon Manager, and the polling continues, then the problem is not LMS.  You must have some other application installed on the server which is doing the polling.

New Member

Re: Auth Failure Traps

There's no other applciation, only Ciscoworks on this server. Etherreal showing missing MIB (see atatachment)

All I did was changing the snmp community string on routers and switches and removed the one one.

What else can I check?

Re: Auth Failure Traps

exactly where do you see the authentication failure ? Is it on the LMS server? In which application? Is it on another server, which application do you use to get this message?
If you have stopped the LMS daemons (net stop crmdmgtd on windows or \etc\init.d\dmgtd stop on solaris), open 2 terminal sessions to one of the devices and  issue the following commands in the first sessions:
term mon
debug snmp packets

use the second session to easily disable the debugging if the message ouput in the first session is too much. Use this command:
undebug all

and you should see the source of the request. Is it still pointing to LMS?

New Member

Re: Auth Failure Traps

Hi Mermel,

Thank you for your rhelp.

I see the 'authentication failure message' on the syslog server, not on LMS server. Im using Kiwi syslog service manager to capture these messages.

As advised, tos atrtw ith I enable daemon and performed the debug command, I could see the source of the request (Ciscoworks server) after I entered 'debugged snmp packets'

Feb 25 19:36:45.375 AWST: SNMP: Packet received via UDP from 10.1.1.1 (CWserver) on Serial0/0/1
Feb 25 19:36:45.379 AWST: SNMP: Queuing packet to 10.1.1.2 (syslogserver)
Feb 25 19:36:45.379 AWST:
Outgoing SNMP packet
Feb 25 19:36:45.379 AWST: v1 packet
Feb 25 19:36:45.379 AWST: community string: readstring
Feb 25 19:36:45.379 AWST: SNMP: V1 Trap, ent snmpTraps, addr 192.168.1.1 (remote router loopback interface), gentrap 4, spectrap 0
lsystem.5.0 = 10.1.1.1
ciscoMgmt.412.1.1.1.0 = 1
ciscoMgmt.412.1.1.2.0 = 10.1.1.1
Feb 25 19:36:45.631 AWST: SNMP: Packet sent via UDP to 10.1.1.2

How can I stop this?

Cisco Employee

Re: Auth Failure Traps

Based on the packets you captured previously, it could be DFM or HUM doing the polling.  Try shutting down DfmServer and DfmServer1, and see if the polling stops.  If not, shutdown UPMProcess.

But just to be clear, if you do "net stop crmdmgtd" does the polling stop?

New Member

Re: Auth Failure Traps

I did "net stop crmdmgtd" but the polling didnt stop. I have only 1 ciscoworks server and no other application running in the background other than Ciscoworks.

Correction, I mentioned CW used old snmp string, I was wrong. It uses a new string when polling, but I dont know why it giving me a authentication failure. I've check thwe string on my routers and switches and again on CW.

I shutdown dfmserver and dfmserver 1 but still see the polling, my server doesnt have HUM so there's no UPMprocess option to shutdown

Every minute I get about 100 traps, I have 1000+ routers and switches, so basically sonner or later it going to killing my syslog server.

Attached is the screen capture of the repated polling message on my syslog server

Thank you for your input.

Cisco Employee

Re: Auth Failure Traps

With Daemon Manager shutdown.  Post a list of all processes running on the server.

New Member

Re: Auth Failure Traps

Hi Clarke,

Document 1 contains a list of processes with with daemon shutdown.

Document 2 contains a list of processes with daemon turned on.Im seeing alot of cwjava.exe processes, is it normal?

Should sm_server processes be stopped when I stopped the daemons? Is this as bug (CSCsx23656-DFM3.2: sm_server does not stop when daemons are stopped. ?)

Which processes can I kill in order to stop the snmp polling?

Cisco Employee
New Member

Re: Auth Failure Traps

You mean reboot the server?

What is the root cause?

Cisco Employee

Re: Auth Failure Traps

Yes, I mean reboot the server.  The root cause is that the DFM polling processes are not shutting down when Daemon Manager goes down.

New Member

Re: Auth Failure Traps

I've applied the patch and rebooted the server but the auth failure re-appeared.


C:\Program Files\CSCOpx\bin>perl CSCtb87449-0.pl

The patch is getting installed.....

The CiscoWorks Daemon Manager service is stopping..
The CiscoWorks Daemon Manager service was stopped successfully.

The CiscoWorks Daemon Manager service is starting.
The CiscoWorks Daemon Manager service was started successfully.


Generating the information file for the patch
Patch Installation was successful...

I notice that after rebooting the server, the auth failure went away for a few minutes, so its definitely the CW server that sending out uneccessary polling!

Attached is the server processes.

Is there another patch?

Thanks


Cisco Employee

Re: Auth Failure Traps

If you shutdown Daemon Manager, is the server still polling these devices?

New Member

Re: Auth Failure Traps

Prior the reboot the CW server, when I shutdown daemon, I can still see the polling.

Now when I shutdown daemon, the polling stopped.

When I re-enable daemon, polling starts again.

Ive atatched the server processes when daemon enabled and disabled.

Thanks Joe,

Cisco Employee

Re: Auth Failure Traps

To be clear, the polling is generating authFail traps still, or are the devices being polled with the correct SNMP credentials, and you just want polling to stop?

New Member

Re: Auth Failure Traps

Hi Joe,

Yes, the polling is still generating  authFail traps.

Yes, the devices being polled with the  correct SNMP credentials.

Yes I want the polling to  stop.

There are many cwjava.exe and sm_server.exe process when Daemon is enabled. Is this normal?

Can you see any issues with the server processes.

Cisco Employee

Re: Auth Failure Traps

Polling with the correct credentials will not cause authFailures.  If you want the polling to stop altogether, just remove the devices in question from DCR.  Yes, when Daemon Manager is running, there should be four sm_server processes, and quite a few cwjava processes.

New Member

Re: Auth Failure Traps

How do I verify whether the device has correct credentials apart from running one rme snmp credential checks?

Attached is the authentication error on kiwi syslog (1- kiwisyslog error.JPG)

When i verified the snmp credential check the read., write and ssh are all oK. 2 - snmp credential verified on RME.JPG

I want to stop all polling on ciscoworks, can you give me a list of the polling availbe on CW?

Cisco Employee

Re: Auth Failure Traps

If you remove the devices from DCR, and stop running Common Services Discovery, nothing in LMS will poll the devices.  Periodic polling is available from Common Services (with Discovery and DCR device status polling), RME, DFM, Campus, IPM, and HUM.  But all of those apps feed off of DCR to determine what devices to poll.

New Member

Re: Auth Failure Traps

Can you direct me to a link so I can download and upgrade the latest software for all of my CW apps?

Cisco Employee

Re: Auth Failure Traps

What versions of the LMS applications do you currently have installed?

New Member

Re: Auth Failure Traps

I'd like to upgrade all the apps to the latest/stable version, could you

please advise?

Name                                             Version   
CiscoWorks Common Services        3.3.0   
Campus Manager                            5.2.0   
CiscoView                                      6.1.9   
CiscoWorks Assistant                     1.2.0   
Device Fault Manager                      3.2.0   
Internetwork Performance Monitor     4.2.0   
Integration Utility                            1.9.0   
LMS Portal                                    1.2.0   
Resource Manager Essentials        4.3.1

Thanks Joe

Cisco Employee

Re: Auth Failure Traps

New Member

Re: Auth Failure Traps

Hi Joe,

The root cause of authentication failure messages was due to dfmserver. When I stop it, the message disappeared.

Process:DfmServer
Path:C:\PROGRA~1\CSCOpx\objects\smarts\bin\CS_sm_server.exe
Flags:
Startup:Started automatically at boot.
Dependencies:DfmBroker

Before applying the patch, when I shutdown dfmserver, I could still see the polling. After applying the patch, the polling stop.

There are only 2 patches for DFM. I have also applied fix CSCta56151.

Patches installed

Patch NameVersionInstalled Date
CSCtb87449-0002 Mar 2010, 11:28:07 WST
CSCta56151-0004 Mar 2010, 14:18:46 WST

Any more tips Joe?

Cisco Employee

Re: Auth Failure Traps

No, there is no known issue here.  It could be that when you changed the community strings in DCR, DFM was not updated (because of the bug you were encountering previously).  Remove the devices which are generating the authFail from DFM.  Make sure that DfmServer and DfmServer1 are running when you do this.  Then, re-add them, and that should hopefully sync everything.  DFM will still be polling the devices, but the authFail messages should stop.

New Member

Re: Auth Failure Traps

Thanks Joe, you're a legend.

The errors have gone. I had to reinitialize the dfm db.

1542
Views
10
Helpful
29
Replies
CreatePlease to create content