Very High Utilization of CPU on Cisco 4506 Switch

[right_guy12]

Hi!

Very high utilization upto 99% is being experienced on a cisco 4506 switch. I have not been able to acertain the cause for the same. Although I have noticed that 2 particular interfaces(both running on a speed of 100Mbps) of which 1 is connected to a Veritas backup server and the other one is connected to an Oracle machine, have a very high input/output rate( 4035 packets/sec and 6271 packets/sec respectively). I suspect this may be the cause of high utilization of the switch.

Also theres a process called "Cat4k Mgmt LoPri" which has a very high utilization. I dont much about this process, what does it do? etc.

Below is the output from Show Processes Cpu command for the above mentioned process

88.63% 80.48% 72.39% 0 Cat4k Mgmt LoPri

Also the speed for the interface to which Veritas backup server is connected cannot be forced to be 1000Mbps 'coz as I understand, veritas doesnt support it.

The switch is running on IOS Version 12.2(18)EW3 and has WS-X4515 SUP module.

Please help me out!

[pkhatri]

Hello,

Do you have a service-policy with a lot of class-maps attached to one of your interfaces that has a high input/output rate ? If so, you may have hit bug CSCeg38623, which was first found in version 12.2(18)EW.

Issue the 'show platform health' command and if it indicates that the 'K2AclCamMan kx stats" is the job that takes most of the CPU, I am sure you have the above bug

There is no stated workaround for this bug. You would have to upgrade the software.

Hope that helps.

Pls remember to rate all posts.

Regards,

Paresh.

[right_guy12]

Hi! Paresh,

Thanks for the reply.

The answer to your questions is no. The 'K2AclCamMan kx stats' job was not taking most of the CPU. Its % CPU Target is 1 and %CPU Actual is 0.

Any other possible way out?

Regards.

[ankurbhasin]

Hi Friend,

When you issue the show processor cpu command, the output showing that the "Cat4k Mgmt LoPri" process is using the CPU indicates that the packets are being process switched and not hardware switched.

If you do not have IPX or AppleTalk routing enabled, but still see Cat4k Mgmt LoPri using high CPU, then you may have to troubleshoot which packets are sent to the CPU for processing.

Have you disabled fast switching on your switch interfaces?

Regards,

Ankur

[right_guy12]

Hi Ankur!

Thanks for the reply!

Fast switching is not enabled on the switch. And I dont have IPX or Appletalk protocols in use.

Can you help me out in the trouble-shoooting of the packets which are being sent to CPU for processing?

Regards.

[ankurbhasin]

Hi Friend,

If fast switching or hardware switching is not enabled on the switch all the packets will be process or CPU switch and will result in CPU going high.

Can you please post your "sh run" from the switch.

Regards,

Ankur

[right_guy12]

Hey Ankur,

Thanks.

I am sorry, I wont be able to provide you with the configuration, although I may be able to give you any specific information that you might you be looking for.

Regards.

[Georg Pauwen]

Hello,

in addition to the other posts, can you post the output of ´show platform health´ as previously mentioned by Paresh ?

Also, do you see any ´HOSTFLAPPING´ messages in the log of the switch ? How is the Veritas server connected ? With one or two NICs ?

Regards,

GP

[right_guy12]

Hi, Gp!

There are no 'HOSTFLAPPING' messages in the system log. The veritas server is connected through one NIC only, the other NIC is connected to standby 4506 switch.

[ankurbhasin]

Hi Friend,

As I said earlier I am sure your packets are getting process switched instead of hardware switched whcih is lettting the CPU hitting high. Can you confirm if CEF is enabled or not?

sh ip cef

Regards,

Ankur

[right_guy12]

Hi!

I was having trouble attaching files...thats why couldnt post the output of 'sh ip cef' earlier.

Here it is attached alongwith.

Thanks.

[Prashanth Krishnappa]

I looked at the sh platform health output and I do not see any indication of the switch being at 99% CPU. Use the following tech tip to troubleshoot

http://www.cisco.com/warp/public/473/cat4500_high_cpu.html

In the output of "sh platform health", I see that the process "K2L2 Address Table R" is consuming little more cycles than it should. Thisprocess is related to L2 entries and adjacency updates. The learning of MAC addresses on this box is done in software, so if we keep on learning too many mac addresses you will see a spike of CPU on this process. Check to see if there are any links going up/down regularly. We do not log link updown events by default in the 4500 and you will need to enable "logging event link-status" under all the interfaces.

Also, disable redirects using "no ip redirects" under all VLAN interfaces.

[right_guy12]

I have checked and there are no links which are going up and down regularly.

And also "no ip redirects" is already in use under all VLAN interfaces.

[ankurbhasin]

Hi Friend,

Ok post me the output from

"sh ip cef" and "sh run int " int number on which you have your varitas server and the interface which is facing the slow response.

Regards,

Ankur

[right_guy12]

Hi Ankur!

Here is the output of the "sh run int " command

interface GigabitEthernet5/27

description ******** Backup Server NB01 *********

switchport access vlan 2

speed 100

duplex full

no cdp enable

spanning-tree portfast

end

vlan2 is the vlan in which all the servers are connected.

Another interface on which an Oracle server is connected has the same config. That interface also has a high input/output rate of packets.