06-30-2013 08:33 PM - edited 03-07-2019 02:09 PM
Hello,
We need help to troubleshoot high CPU on Cisco 4948E switch. IOS image is cat4500e-lanbase-mz.122-54.SG1.bin
Tried finding the root cause using Cisco documentation for High CPU on 4948 switches but unable to find anythin specific.
Below is the output of show cpu | ex 0.00 command.
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
55 4533312041368514536 331 4.95% 4.84% 4.83% 0 Cat4k Mgmt HiPri
56 414621502 831220970 498 93.44% 93.64% 93.71% 0 Cat4k Mgmt LoPri
92 98582 185950416 0 0.07% 0.03% 0.02% 0 UDLD
116 63326430 57579164 1099 0.63% 0.60% 0.59% 0 Spanning Tree
The show platform health command shows K5 L2 Hardware Address process high actual cpu as well, while CpuMan Review seems to be normal.
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
K5 L2 Hardware Addre 2.00 92.88 20 10 100 500 126 118 93 3615:30
K5CpuMan Review 30.00 6.68 30 7 100 500 7 6 5 5362:24
We have checked the switch logs, no relevant logs available.
As far as toploogy is concerned, this switch is connected to another 4948E switch via etherchannel. Both switches are also connected to seperate firewalls & routers for redundancy.
Traffic on switch interface is normal & MAC addresses learnt are less than 10 per switch. No MAC flapping logs seen on switch.
Let me know if any other details are required.
Solved! Go to Solution.
06-30-2013 09:09 PM
Nitin,
Have you gone through the link mentioned below:
Ref:
https://supportforums.cisco.com/message/628786#628786
'Cat4k Mgmt LoPri' schedules all cat4k platform-dependent processes, which
exceed the CPU target utilization.
You can see target and actual CPU utilization in the output of 'show platform health'
'Cat4k Mgmt HiPri' manages processes which are staying under CPU target
utilization. Misbehaving processes are handled by 'Cat4k Mgmt LoPri'.
NOTE: The common misconception is that high CPU utilization indicates the
depletion of resources on a device and the threat of a crash. A capacity issue is one of
the symptoms of high CPU utilization on Cisco IOS routers. However, a capacity issue is
almost never a symptom of high CPU utilization with hardware-based forwarding switches like
the Catalyst 4500. The Catalyst 4500 is designed to forward packets in the hardware
application-specific integrated circuit (ASIC) and reach traffic-forwarding
speeds of up to 102 million packets per second (Mpps).
>>These two processes aggregate multiple platform-specific processes which
perform the essential management functions on the switch. These
processes process control plane as well as data packets that need to be
software-switched or processed.
Please follow the steps mentioned in the above link to findout the root cause. In case if you face any difficulty do let me know .
HTH
Regards
Inayath.
*Plz rate usefull posts.
06-30-2013 10:51 PM
Nitin,
I see that the process causing the high CPU specifically is K5 L2 Hardware Address. This process is associated with MAC address learning. Please send me the following outputs: . Network topology . Show platform cpu packet statistics all Also we need to find out why mac address table is updated so frequently (excessive mac moving/re-learning). One way is enable mac notification: MAC notification can be enabled the following global config lines, using the maximum buffer size: -------------------------------------------------- . mac address-table notification change . mac address-table notification mac-move . mac address-table notification change history-size 500 . mac address-table notification change interval 10 Please configure this, wait 30 seconds or so (the switch may generate logging messages if there are MAC flaps also), then grab 'sh mac address-table notification change' Thank you in advance.
Regards
Inayath
06-30-2013 09:09 PM
Nitin,
Have you gone through the link mentioned below:
Ref:
https://supportforums.cisco.com/message/628786#628786
'Cat4k Mgmt LoPri' schedules all cat4k platform-dependent processes, which
exceed the CPU target utilization.
You can see target and actual CPU utilization in the output of 'show platform health'
'Cat4k Mgmt HiPri' manages processes which are staying under CPU target
utilization. Misbehaving processes are handled by 'Cat4k Mgmt LoPri'.
NOTE: The common misconception is that high CPU utilization indicates the
depletion of resources on a device and the threat of a crash. A capacity issue is one of
the symptoms of high CPU utilization on Cisco IOS routers. However, a capacity issue is
almost never a symptom of high CPU utilization with hardware-based forwarding switches like
the Catalyst 4500. The Catalyst 4500 is designed to forward packets in the hardware
application-specific integrated circuit (ASIC) and reach traffic-forwarding
speeds of up to 102 million packets per second (Mpps).
>>These two processes aggregate multiple platform-specific processes which
perform the essential management functions on the switch. These
processes process control plane as well as data packets that need to be
software-switched or processed.
Please follow the steps mentioned in the above link to findout the root cause. In case if you face any difficulty do let me know .
HTH
Regards
Inayath.
*Plz rate usefull posts.
06-30-2013 10:24 PM
Hello Inayath,
Thanks for your inputs.
I have already tried troubleshooting the issue using the document you provided but found nothing specific to the outputs I am getting.
Only K5 L2 Hardware Address is showing high cpu rest all are OK. Not too sure how to further troubleshoot this to find the exact root cause.
One more thing I want to highlight is that the CPU usage is showing high on both 4948E switches interconnected via etherchannel. The high CPU is observed since last 3 days but no changes have been made on the switches recently.
Need your help to find out the exact root cause so that this can be corrected.
Regards.
06-30-2013 10:51 PM
Nitin,
I see that the process causing the high CPU specifically is K5 L2 Hardware Address. This process is associated with MAC address learning. Please send me the following outputs: . Network topology . Show platform cpu packet statistics all Also we need to find out why mac address table is updated so frequently (excessive mac moving/re-learning). One way is enable mac notification: MAC notification can be enabled the following global config lines, using the maximum buffer size: -------------------------------------------------- . mac address-table notification change . mac address-table notification mac-move . mac address-table notification change history-size 500 . mac address-table notification change interval 10 Please configure this, wait 30 seconds or so (the switch may generate logging messages if there are MAC flaps also), then grab 'sh mac address-table notification change' Thank you in advance.
Regards
Inayath
06-30-2013 10:55 PM
Did you debug the packets?
debug platform packet all receive buffer (Captures all packets sent to CPU in a circular buffer)
show platform CPU packet buffered
If yes please provide me above outputs?
Regards
Inayath
07-01-2013 09:46 PM
Hello Inayath,
I am really sorry as I cannot debug packets since the switch is in production carrying customer traffic.
However, seems like I have got a possible cause which might have caused this spike in CPU.
Although there are no mac flaps seen in the switch logs. However, as per your last comment that this might be related to excessive mac moving/re-learning. I checked the MAC address table of the switch several times in a minute & found that one MAC address was indeed moving on different switch interfaces for a particular VLAN.
This MAC address belong to VRRP type & thus I checked the routers connected on both the switches. I found that the sub-interface configuration for that particular VLAN on the router has been misconfigured. There are logs on routers for dupilcate IP address on that particular subinterface, subinterfaces on both routers have been configured with same physical IP address.
This sub-interface were created on the routers about the same time when the CPU spiked up on the switches.
We have asked the team who manage the router to correct the VRRP & will then check the switch CPU. By all means it shoud be back to normal again.
I will keep you updated about the same. Many thanks for your immense help on this.
Regards.
07-01-2013 11:23 PM
Nitin,
Thanks for the update. Yes that was the culprit.
Anyways hope that resolve kindly keep us updated. Also feel free to ask in case any further help is required.
HTH
Regards
Inayath
****************Plz rate all usefull posts.******************
07-02-2013 09:23 PM
Hello All,
For enveryone's information, I confirm that the high CPU usage issue faced by me has been resolved.
Root cause was wrong VRRP configuration on connected routers due to which Duplicate IP error messages were generating on Router. Also, on switch we had excessive MAC moving/re-learning causing the issue.
Same was corrected on rotuers which resolved the issue.
Once again many thanks to Inayath for pointing me towards the right direction.
Regards.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide