Access Point CPU at 100% due to inventory collection

[Martin Ermel]

I have a customer which reports a weekly arising problem with his APs. A sniffer trace pointed him to LMS 2.5.1 (solaris). At the time in question LMS performs its system scheduled inventory collection which let CPU utilization jump up to 100 % - which is disruptive for the AP...

A manual started inventory collection on an AP let the snmp process consume all the CPU (100%).

Does anybody else observed this behaviour?

Customer reports this not for specific APs or IOS versions.

Currently I have not the possibility to test this in lab..

MArtin

[Joe Clarke]

I seem to recall hearing this before, but without more specific details about what objects are being polled when the CPU spikes on the AP, it is difficult to locate an exact bug. A sniffer trace of the SNMP traffic to the AP along with an exact time of when CPU spikes would help determine where the problem lies.

[Martin Ermel]

Thanks Joe, tomorrow I am on site at the customer and try to get as much info as I can - which will not be so easy because of different responsibilities...

[Martin Ermel]

I just got the IC_Server log and found the following- but could this jump up the CPU?? Is this a java error of the device package or just a timeout issue??

I will add the log as an attachment.

thanks,

MArtin

[nhabib · Accepted solution]

This could be similar to CSCsa50951:

packets dropped and high cpu while doing snmp walk on CISCO-FLASH-MIB

Problem:

--------------

IOS AP1200 is seen to get stuck at high cpu, and temporarily fails to respond during the time objects within the CISCO-FLASH-MIB are being polled.

Symptom:

----------------

it is seen that when doing an snmpwalk on the mib, the AP fails to respond back to ciscoFlashFileChecksum and its CPU gets pegged at 100%

at this time, the AP has shown to fail to respond temporarily.

Workaround:

-------------------

use the following view for your community strings (i.e replace public/private below with your RO/RW commstrings)

snmp-server view skipChecksum iso included

snmp-server view skipChecksum ciscoFlashFileEntry excluded

snmp-server community public view skipChecksum RO

snmp-server community private view skipChecksum RO

[Martin Ermel]

Thanks Nadim, you were right - I was hitting that bug; further investigations showed, that the AP did respond to the CISCO-FLASH-MIB, but its CPU was up to 100% so the AP couldn?t do its usual job (and the InvColl job ended with status failed)

I tested this also with to SW versions: 12.3.8-JEA and 12.3.7-JA4; with both the SNMP-Engine process used up to 70-80% CPU while running an "Inventory Collection" Job on that device without a service interuption. But enabling "debug snmp packets" on the AP let the SNMP-Engine process consume all the CPU while an "Inventory Collection" was running and as a consequence it faild to respond temporarily.

So - what I didn?t know - "debug snmp packets" enabled at the wrong time could be disruptive for the services on a device. A customer also reported service failures (STP Loops) after he enabled "debug snmp packets" on a cat4006 (SUP4)... - I think this could be interesting for some people..

thanks,

MArtin