Inventory Collection Failed on Cisco 3550 Switches LMS 3.0

[dionjiles]

Problem Details: I have 75 devices that fails when performing inventory collection. The

Message ID I am receiving is RICS0001 (Cannot successfully collect Inventory Information

for device) Internal Error. After trying to collect inventory again it happened again,

here is the information I was able to collect for one of the devices in the IC_Server.log

I have a TACService Request opened, but its been a couple of days since I heard anything. Just wondering has anyone else encountered issues with these devices. Wasn't a problem for me when I on LMS 2.5.1

Attached is the log I ran after setting the log level to fatal.

[dionjiles]

Please see attached.

Yes that is one of the peasants thats giving me issues.

The funny thing I have a number of 3550 48Switch devices at another location that I am not having issue with adding to Inventory collection.

If you want I can send you the running config from one of those devices as well.

[Joe Clarke]

This could be an IOS bug. Take a look at one of the working 3550s, and compare the IOS version and config to see if there are any obvious differences. If there are none, you might try restarting a failing switch to see if that gets around the problem.

[dionjiles]

I will have my Network Engineer look into the configs to see if he can determine what's going on. Will keep you updated.

[fasl01]

The code running on my 3550-48s is 3550-ipservices59-mz.122-25.SEB4. It is only affecting the 3550-48 port switches it looks.

[Joe Clarke]

First, you need to get the patch for the known bug. That may resolve your problem (it should). If you still see failures, then additional troubleshooting can begin.

[fasl01]

Will do. I am waiting on TAC to respond to the service request I opened. I will update once I get the patch and install it.

Thanks

[dionjiles]

Good morning,

I compared the configs on the working and failed switches and the only thing i noticed was a access list permit statement for the CiscoWorks Server IP address the failed switched had the old CiscoWorks server IP address so I changed it and also changed the R/W community string.

The collection still failed but I was able to get this info from the log.

[ Tue Oct 30 15:35:00 EDT 2007 ],DEBUG,[Thread-22],com.cisco.nm.rmeng.inventory.ics.core.DevMgmtStatusUtil,462,QQQWQQ2: update Dev_Mgmt_Status set Sys_Object_Id = '.1.3.6.1.4.1.9.1.367',Dev_Status = -1,Inventory_Collection_Status = 3,AG_Collected = '',Failure_Reason = 'Device sensed, but collection failed ',Corrective_Action = 'Device side issue',Last_Updated_Time = '2007-10-30 15:35:00.238' where NetworkElementID = 4117

Device side issue concerns me....we will continue to investigate as to what is going on. I'm nervous about restarting the switches, this may have to be an after-hours project.

[Joe Clarke]

What about the versions of code? The non-working switch is running an older 12.1 version. Is the working switch running a newer version?

[dionjiles]

Attached are the show version outputs.

File sp00wsw.txt is the failed device

Files idjp3sw.txt is the successful device

[Joe Clarke]

So the working one is running a newer version of code. This may very well be a device side problem in the ENTITY-MIB. A sniffer trace of inventory collection from both devices should confirm the problem. Moral of the story will most likely be you need to upgrade the failing device.

[dionjiles]

That is correct. When you say sniffer trace of inventory collection from both devices can I run one on the failed device even though it's failing collection in CiscoWorks.

Sounds like its a good time to start learning the Software Mgmt piece in RME to for Software Distribution.

[Joe Clarke · Accepted solution]

You can always runs a sniffer trace. This is really the best way to diagnose inventory problems when the stack trace contains a MIB module name.

[dionjiles]

Thanks.....I will pass this info along to the Network Engineers.

[dionjiles]

I have one last question regarding this issue. When we were on LMS 2.5.1 the same devices that are failing now we didn't experience issues with Inventory collection. Is Ciscoworks not backward compatible? I'm sure the patch works for the IOS bug, but from a device standpoint when I ran the WindowCL tool I was able to collect info. Sorry to be a pest but just trying to gather as much information as I can so I can pass along to the Engineers.

[Joe Clarke · Accepted solution]

We are constantly evolving our device packages. We may be polling for newer objects in newer device packages, and tickling some device-side bugs. As I said, the sniffer trace for a working and non-working device would give a more complete picture as to why one is failing. Something must be different in the ENTITY-MIB data one device is returning.