Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

SNMP CPU Hog problems

I have CiscoWorks 2000 on my network. Every 4 hours, it collects inventory and configs from a pair of 7507 routers. When I view the router logs, I am getting SNMP CPU hog messages for the get next procedure. Apparently, CiscoWorks is causing performance problems on my network by maxing out the cpu on these routers during mib collection (hard to believe).

Has anyone seen this before? Any advice? Right now, CiscoWorks has been shut down totally, to prevent any future problems.

10 REPLIES
Blue

Re: SNMP CPU Hog problems

You may be running into a bug. Try doing a CCO Bug Toolkit search to see if you find anything on this.

New Member

Re: SNMP CPU Hog problems

New Member

Re: SNMP CPU Hog problems

Well, the URL given in the previous post provides some general help - specifically if the problem you're running into is caused by queries into the routers' ARP and routing table - which CiscoWorks generally does not do, per se (unless you've configured it to do so).

First off, what is the general baseline CPU load of the routers in question? If you're baseline is 90 percent, then I don't think you're running into a CWorks problem, you've got a bigger issue.

Second: Why the heck are you doing inventory collection and config polls every four hours? Is your network changing that often? Instead, I'd suggest doing inventory sweeps once a week or so - during off hours - and config pulls on a far less frequent basis as well. (once a day? once a week? depends on your business policies).

Third: thing you can do w/CWorks is configure it to pull configuration(s) only after it receives a config trap from a managed device. Something to consider.

Another point to consider: Are you running any other network management tools? Specifically, Netview, Openview, MRTG, Unicenter, Cricket, etc? If one of those tools hits your routers at the same time they're being hit for config information - it might appear the problem is with CWorks when it's really about the other tool(s). You might want to check that via the DEBUG command.

OBLIGATORY WARNING AND ALERT: If you run DEBUG, you do so at your OWN RISK!! Since your router is already under duress, using DEBUG commands to isolate what specific device is polling your router (and asking what) poses significant and measurable risk to the stability of your router and network even if caution is taken!

SUGGESTION: If you do use DEBUG to isolate, be sure to have console and terminal logging either DISABLED or set to a logging level higher then DEBUG.

Last, another thing to consider, besides excluding the arp and route table, is to use ACL's to permit only specific devices poll your router(s) via SNMP to ensure only those devices you know about can poll them.

As a comparison, I have CWorks polling the routers (250+) my team manages which run the gamut from 2612's all the way up to 7507's and MSFC/2s. I've never had a problem caused by CWorks polling for configuration and inventory information.

New Member

Re: SNMP CPU Hog problems

You are most likely running into an IOS rev issue and a MIB walk issue as described. We've seen this from time to time on devices of a variety of platforms. Updating the IOS solved the issue.

What ever you do, change the default SNMP strings, and apply an access list against the strings. Depending on your needs, you can also 'log' the deny to track down who is running snmp discoveries.

Here is a detailed description on the ANI server internals which may be helpful background. http://www.cisco.com/warp/public/477/Campus/ani.shtml

regards

New Member

Re: SNMP CPU Hog problems

It get's better. Not only does CiscoWorks max out the cpu, it generates CMD errors, HSRP flapping, and interface resets (essentially brings down the 7507). I know have my CiscoWorks server turned completely off because of the danger to my network.

The 7507 is a seed device, so this, I'm sure is a factor.

New Member

Re: SNMP CPU Hog problems

I assume a TAC case was opened...

What is the IOS rev on the 7507?

Checking on the similar devices we have... they are on old code...one has been up for 1 1/2 years in a pair HSRP arrangement, its other pair was up for 6 months.

Cisco Internetwork Operating System Software

IOS (tm) RSP Software (RSP-JSV-M), Version 11.2(16)P, RELEASE SOFTWARE (fc1)

New Member

Re: SNMP CPU Hog problems

I am running 12.1(8b)E7

I am concerned that even if I correct the one problem here, the CPU HOG issue is going to reappear elsewhere. It's a bandaid, not a fix. I have tried 3 IOS versions.

New Member

Re: SNMP CPU Hog problems

WAC-A-Mole is the name of the game. This is a TAC case for sure... We've come across the CPU-HOG error mostly on 2501's running stoneage code. If I base my knowledge on what I've seen, and read, this would appear to be a model specific issue. We've got devices of other models up and down coded from the one you have running.

I'd point to TAC for a possible bug fix. For your ref, I checked for 7500 in our environment. They're not while the 7507s were back a rev, the 7513's were more current:

On two of our 7513s that have been very stable, they are running the same IOS code: here is a show ver:

Cisco Internetwork Operating System Software

IOS (tm) RSP Software (RSP-JSV-M), Version 12.0(10), RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2000 by cisco Systems, Inc.

Compiled Tue 21-Mar-00 00:34 by phanguye

Image text-base: 0x60010930, data-base: 0x60EF6000

ROM: System Bootstrap, Version 11.1(8)CA1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc

1)

BOOTFLASH: GS Software (RSP-BOOT-M), Version 11.1(8)CA1, EARLY DEPLOYMENT RELEAS

E SOFTWARE (fc1)

xxxxxxxxxx uptime is 23 weeks, 6 days, 21 hours, 0 minutes

System restarted by reload

System image file is "slot0:rsp-jsv-mz.120-10.bin"

cisco RSP4 (R5000) processor with 131072K/2072K bytes of memory.

R5000 processor, Implementation 35, Revision 2.1 (512KB Level 2 Cache)

Last reset from power-on

G.703/E1 software, Version 1.0.

G.703/JT2 software, Version 1.0.

X.25 software, Version 3.0.0.

SuperLAT software (copyright 1990 by Meridian Technology Corp).

Bridging software.

TN3270 Emulation software.

Chassis Interface.

1 FSIP controller (8 Serial).

1 TRIP controller (2 Token Ring).

6 VIP2 controllers (5 FastEthernet)(8 Ethernet)(32 Serial)(1 Fddi).

8 Ethernet/IEEE 802.3 interface(s)

5 FastEthernet/IEEE 802.3 interface(s)

2 Token Ring/IEEE 802.5 interface(s)

40 Serial network interface(s)

1 FDDI network interface(s)

123K bytes of non-volatile configuration memory.

20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).

8192K bytes of Flash internal SIMM (Sector size 256K).

Slave in slot 7 is running Cisco Internetwork Operating System Software

IOS (tm) RSP Software (RSP-DW-M), Version 12.0(10), RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2000 by cisco Systems, Inc.

Compiled Tue 21-Mar-00 00:37 by phanguye

Slave: Loaded from system

Slave: cisco RSP4 (R5000) processor with 131072K bytes of memory.

Best of luck...

New Member

Re: SNMP CPU Hog problems

I used the snmp parameters as defined in this article

http://www.cisco.com/warp/customer/477/SNMP/ipsnmphighcpu.html

I also took the router out of the seed list and turned inventory polling off.

Even with these changes, if I bring CiscoWorks back online, cmd errors, hsrp flapping, and snmp cpu hog messages appear on the router. It will eventually bring the 7507 down along with part of my core network. Needless to say, I have not had network management software running for quite some time as a result of this. I wish I had a solution -the TAC has given me next to no guidance on this.

New Member

Re: SNMP CPU Hog problems

We have had a similar problem due to the ANI Server discovery. When the ANI Server tried to discover our 7500 WAN Consolidation Routers, we were causing SNMP CPU spikes that caused a EIGRP reconvergence. In order to solve this problem, we explicitly exclude the routers' IP addresses from the ANI Server discovery. We then had to define seed devices an all sides of the routers.

477
Views
0
Helpful
10
Replies
CreatePlease login to create content