catalyst 3750 high utilization

[jackson.ku]

Hi,

We have one Catalyst 3750 with higher CPU utilization. ( CPU utilization for five seconds: 36%/30%; one minute: 27%; five minutes: 28% ) The CPU utilization for the other Catalyst 3750 is below 10%

We execute "sh proc cpu sort | exc 0.00" many times and find process "Check heaps" with higher utilization :

switch#sh proc cpu sort
CPU utilization for five seconds: 36%/30%; one minute: 27%; five minutes: 28%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   4     1581593     71425      22143  8.94%  1.19%  0.91%   0 Check heaps     
  88      219985     14589      15078  0.79%  0.15%  0.12%   0 HULC Tcam Memory
 207       76592    525547        145  0.15%  0.11%  0.07%   0 IP Input        
 289         100       315        317  0.15%  0.10%  0.02%   1 Virtual Exec

The IOS version is 122-55.SE8. Please help to give me some advise to find the higher utilization issue.

Best Regards,

[Leo Laohoo]

28% CPU is not high. 

[jackson.ku]

Hi, Thanks for your reply, but the CPU utilization of the other Catalyst 3750 is below 10% ( almost the same configuration and same environment ) So we need to find the reason why this CPU utilization is higher than others.

[InayathUlla Sharieff]

Jackson,

Could you please have a look on my document which have been created for troubleshooting the high cpu on 3750 and see if that helps. Else capture the ouptuts which are been suggested and we will have a look:

https://supportforums.cisco.com/document/12298401/troubleshooting-high-cpu-3750

HTH

Inayath

[jackson.ku]

Dear All, please reference attached file ( the show command result ). Thanks a lot....

[InayathUlla Sharieff]

Can you also send this:

- show sdm prefer
- show platform tcam util asic all
- show ip route summary
- show int | inc is up|drop

debug platform cpu-queues software-fwd-q

(Please be cautious using the debug, only run for a few seconds, then
'un all')

debug platform cpu-queues cpu heartbeat  

(Please be cautious using the debug, only run for a few seconds, then
'un all')

[jackson.ku]

Hi,

attached files are show command result. ( I have not collect debug result because this switch is in production and not allow any performance impact )

I also upload the running-config file. ( We used IPSLA, EEM, PBR to trace link status and change ip route dynamically )

This switch platform is Catalyst 3750G-12S, we have another one Catalyst 3750X-12S switch with almost same configuration, but the CPU utilization below 10%. I compare the "show controller cpu-interface" result, the queue "sw forwarding" for Catalyst 3750G-12S always increased, but the queue "sw forwarding" for Catalyst 3750X-12S not increased. Maybe it is why the Catalyst 3750G-12S CPU utilization higher than Catalyst 3750X-12S?

Is it the platform performance issue? If not, how to find the root cause?

[InayathUlla Sharieff]

Hi,

Below link might help something for you:

https://supportforums.cisco.com/discussion/11532116/high-cpu-usage-3750-x-stack

HTH

[jackson.ku]

Hi,

I know the IP packet with option will forward by software, is there other traffic or function that cause traffic not forward via hardware and it is done by software?
 ( for example : PBR, IPSLA .. )

Maybe I can confirm with customer before I enable "debug platform cpu-queues software-fwd-q"?

Best Regards,

Jackson Ku

[InayathUlla Sharieff]

If you look at the process cpu the utilization is mainly by ip input process., Here is below explanation which I can think of right now:

The IP input process is merely traffic related. More specific layer 3
traffic. Normally layer 3 traffic goes to the CPU to be forwarded that is
why you will have an increase of the utilization.

When I say layer 3 traffic I mean traffic that need to be forwarded based on
its IP address. Traffic that is not forwarded via hardware and it is done
via software. Traffic that needs to be evaluated on the ACLs and routing
protocols instances. The SDM template set to default is a layer 2 template.
That means that the switch is not prepare for heavily loaded layer 3 traffic
forwarding.

What we can try right now is to set the SDM template to routing. You can do
that using the following command:

switch>enable
switch#config t
switch(config)#sdm prefer routing

After entering the command you will need to reload the device to make it
take effect.

Please apply these commands and let me know how it goes. However keep in
mind that IP input means that you are forwarding a lot of layer 3 (IP
address based) packets that are loading your switch too much.

HTH

Inayath

*Plz dont forget to rate all usefull posts.

[jackson.ku]

Hi, The current sdm template was "aggregate routing". Thanks a lot.

[Walter Astori]

Also show stacks can be used for diagnostic

[jackson.ku]

Hi, I got some debug platform cpu-queues software-fwd-q sample collect from lab, please help to mention me how to investigate it. Thanks a lot.

[InayathUlla Sharieff]

IP_SA:192.168.1.77 and 
Destination ip address: 10.1.1.171 

As only this is th destination i paddress which I can see the src is punting the traffic.

Shut down the 192.168.1.77 and see if the cpu gets reduced?

[jackson.ku]

Hi, the debug result was captured at our lab, not at real production network. Is there any doc that can help us to investigate it? ( we should arrange off-hour to capture the debug information )