After i changed snmp strings on our network devices , I see a list of devices with Auth Failure Traps on Syslog server.
Ive check the snmp credential strings on CW for each device and they're correct.
This is the error message on my syslog server:
mm-dd-yyyy 11:23:16 Local0.Info 10.1.1.1 10.1.1.2.150 4 0 Authentication failure 10.1.1.254(CiscoWorks) 1 10.1.1.254(CiscoWorks)
This message wasnt there before i re-new the snmp community string. After I chnage the snmp string on my routers and switches, I a lots of traps on my syslog server.
How can I stop this?
Thank you for your help
If you are absolutely sure DCR is correct for these devices, then check your CS Discovery SNMP settings under Common Services > Device and Credentials > Device Discovery > Discovery Settings. Make sure those community strings are correct as well. If you still can't figure it out, start a sniffer trace filtering on udp/161 traffic between the LMS server and the device, and check to see what SNMP objects are being polled using the incorrect community string. That should help narrow down the problem application.
Thanks for your help.
I used Ethereal to sniff the packets. The result shows that CW is polling using the OLD snmp string even though I have updated the DCR database with the new strings (ive verified the snmp string with device credential and doublecheck in DCR/exports to convirm the device credentialg - DCR db has the new strings). My question is why it uses the old string to poll? I've deleted the whole database and import it again but still seeing "snmp authentication failure" on syslog server.
How can I track which aplication is doing the snmp polling (I have RME, IPM, DFM and CS all on one server)
How can I stop the snmp polling completely? Ive stopped daemon and change aaa to local but no success.
If you've stopped Daemon Manager, and the polling continues, then the problem is not LMS. You must have some other application installed on the server which is doing the polling.
There's no other applciation, only Ciscoworks on this server. Etherreal showing missing MIB (see atatachment)
All I did was changing the snmp community string on routers and switches and removed the one one.
What else can I check?
exactly where do you see the authentication failure ? Is it on the LMS server? In which application? Is it on another server, which application do you use to get this message?
If you have stopped the LMS daemons (net stop crmdmgtd on windows or \etc\init.d\dmgtd stop on solaris), open 2 terminal sessions to one of the devices and issue the following commands in the first sessions:
debug snmp packets
use the second session to easily disable the debugging if the message ouput in the first session is too much. Use this command:
and you should see the source of the request. Is it still pointing to LMS?
Thank you for your rhelp.
I see the 'authentication failure message' on the syslog server, not on LMS server. Im using Kiwi syslog service manager to capture these messages.
As advised, tos atrtw ith I enable daemon and performed the debug command, I could see the source of the request (Ciscoworks server) after I entered 'debugged snmp packets'
Feb 25 19:36:45.375 AWST: SNMP: Packet received via UDP from 10.1.1.1 (CWserver) on Serial0/0/1
Feb 25 19:36:45.379 AWST: SNMP: Queuing packet to 10.1.1.2 (syslogserver)
Feb 25 19:36:45.379 AWST:
Outgoing SNMP packet
Feb 25 19:36:45.379 AWST: v1 packet
Feb 25 19:36:45.379 AWST: community string: readstring
Feb 25 19:36:45.379 AWST: SNMP: V1 Trap, ent snmpTraps, addr 192.168.1.1 (remote router loopback interface), gentrap 4, spectrap 0
lsystem.5.0 = 10.1.1.1
ciscoMgmt.422.214.171.124.0 = 1
ciscoMgmt.4126.96.36.199.0 = 10.1.1.1
Feb 25 19:36:45.631 AWST: SNMP: Packet sent via UDP to 10.1.1.2
How can I stop this?
Based on the packets you captured previously, it could be DFM or HUM doing the polling. Try shutting down DfmServer and DfmServer1, and see if the polling stops. If not, shutdown UPMProcess.
But just to be clear, if you do "net stop crmdmgtd" does the polling stop?
I did "net stop crmdmgtd" but the polling didnt stop. I have only 1 ciscoworks server and no other application running in the background other than Ciscoworks.
Correction, I mentioned CW used old snmp string, I was wrong. It uses a new string when polling, but I dont know why it giving me a authentication failure. I've check thwe string on my routers and switches and again on CW.
I shutdown dfmserver and dfmserver 1 but still see the polling, my server doesnt have HUM so there's no UPMprocess option to shutdown
Every minute I get about 100 traps, I have 1000+ routers and switches, so basically sonner or later it going to killing my syslog server.
Attached is the screen capture of the repated polling message on my syslog server
Thank you for your input.
Document 1 contains a list of processes with with daemon shutdown.
Document 2 contains a list of processes with daemon turned on.Im seeing alot of cwjava.exe processes, is it normal?
Should sm_server processes be stopped when I stopped the daemons? Is this as bug (CSCsx23656-DFM3.2: sm_server does not stop when daemons are stopped. ?)
Which processes can I kill in order to stop the snmp polling?
You need to apply the fix for CSCta56151 from http://tools.cisco.com/support/downloads/go/ImageList.x?relVer=3.2.0&mdfid=282640771&sftType=CiscoWorks+Device+Fault+Manager+Patches&optPlat=Windows&nodecount=2&edesignator=null&modelName=CiscoWorks+Device+Fault+Manager+3.2&treeMdfId=268439477&treeNa... . Then reboot, and this problem should go away.
I've applied the patch and rebooted the server but the auth failure re-appeared.
C:\Program Files\CSCOpx\bin>perl CSCtb87449-0.pl
The patch is getting installed.....
The CiscoWorks Daemon Manager service is stopping..
The CiscoWorks Daemon Manager service was stopped successfully.
The CiscoWorks Daemon Manager service is starting.
The CiscoWorks Daemon Manager service was started successfully.
Generating the information file for the patch
Patch Installation was successful...
I notice that after rebooting the server, the auth failure went away for a few minutes, so its definitely the CW server that sending out uneccessary polling!
Attached is the server processes.
Is there another patch?
Prior the reboot the CW server, when I shutdown daemon, I can still see the polling.
Now when I shutdown daemon, the polling stopped.
When I re-enable daemon, polling starts again.
Ive atatched the server processes when daemon enabled and disabled.
To be clear, the polling is generating authFail traps still, or are the devices being polled with the correct SNMP credentials, and you just want polling to stop?
Yes, the polling is still generating authFail traps.
Yes, the devices being polled with the correct SNMP credentials.
Yes I want the polling to stop.
There are many cwjava.exe and sm_server.exe process when Daemon is enabled. Is this normal?
Can you see any issues with the server processes.
Polling with the correct credentials will not cause authFailures. If you want the polling to stop altogether, just remove the devices in question from DCR. Yes, when Daemon Manager is running, there should be four sm_server processes, and quite a few cwjava processes.
How do I verify whether the device has correct credentials apart from running one rme snmp credential checks?
Attached is the authentication error on kiwi syslog (1- kiwisyslog error.JPG)
When i verified the snmp credential check the read., write and ssh are all oK. 2 - snmp credential verified on RME.JPG
I want to stop all polling on ciscoworks, can you give me a list of the polling availbe on CW?
If you remove the devices from DCR, and stop running Common Services Discovery, nothing in LMS will poll the devices. Periodic polling is available from Common Services (with Discovery and DCR device status polling), RME, DFM, Campus, IPM, and HUM. But all of those apps feed off of DCR to determine what devices to poll.
I'd like to upgrade all the apps to the latest/stable version, could you
CiscoWorks Common Services 3.3.0
Campus Manager 5.2.0
CiscoWorks Assistant 1.2.0
Device Fault Manager 3.2.0
Internetwork Performance Monitor 4.2.0
Integration Utility 1.9.0
LMS Portal 1.2.0
Resource Manager Essentials 4.3.1
The only upgrade currently available is Campus Manager 5.2.1. You can download that from http://tools.cisco.com/support/downloads/go/Model.x?mdfid=282641773&mdfLevel=Software%20Version/Option&treeName=Network%20Management&modelName=CiscoWorks%20Campus%20Manager%205.2&treeMdfId=268439477 .
The root cause of authentication failure messages was due to dfmserver. When I stop it, the message disappeared.
|Startup:||Started automatically at boot.|
Before applying the patch, when I shutdown dfmserver, I could still see the polling. After applying the patch, the polling stop.
There are only 2 patches for DFM. I have also applied fix CSCta56151.
|Patch Name||Version||Installed Date|
|CSCtb87449-0||0||02 Mar 2010, 11:28:07 WST|
|CSCta56151-0||0||04 Mar 2010, 14:18:46 WST|
Any more tips Joe?
No, there is no known issue here. It could be that when you changed the community strings in DCR, DFM was not updated (because of the bug you were encountering previously). Remove the devices which are generating the authFail from DFM. Make sure that DfmServer and DfmServer1 are running when you do this. Then, re-add them, and that should hopefully sync everything. DFM will still be polling the devices, but the authFail messages should stop.