CPU Load 98% after changed MTU size

Unanswered Question
Mar 6th, 2009
User Badges:

Hi,

I am having problem here. Previously, when the MTU was 1508...The cpu load was <5%. When changed to 1512, the load was increasing 70%. Recently, changed to 1522, the load is >95%. Not sure what's going with the configuration. But I can a lot ques dropping. Should I put on the port-channel 1? For more details configuration, please refer to attached file

hold-queue 1024 in

hold-queue 1024 out

# show proce cpu

CPU utilization for five seconds: 96%/38%; one minute: 96%; five minutes: 96%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process



# show proce cpu | include Tag

165 150300 11396947 13 0.07% 0.01% 0.00% 0 Tag Control

193 2608887200 237733062 10974 52.23% 52.71% 52.44% 0 Tag Input

304 4692 36566 128 0.00% 0.00% 0.00% 0 Tagcon Addr

Any idea guys. Thanks lot



Attachment: 
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
shivlu jain Fri, 03/06/2009 - 02:53
User Badges:
  • Silver, 250 points or more

could you tell me which router and ios you are using. Are you using sub interfaces on the router and mpls on router.


regards

shivlu jain

lawwm Fri, 03/06/2009 - 07:45
User Badges:

Hi,

cisco CISCO7609 (R7000) processor (revision 1.2) with 983008K/65536K bytes of memory.

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)

BOOTLDR: s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(18)SXE5, RELEASE SOFTWARE (fc1)

System image file is "disk0:s72033-advipservicesk9_wan-mz.122-18.SXE5.bin"

2 physical Ge interfaces bundle to port-channel.


If not mistaken, by increasing mtu size shouldn't increase the load.


You can refer configure attached.


Thanks.

autobot130 Fri, 03/06/2009 - 08:31
User Badges:

I had this same exact problem on the cat 6500 running SXF10. Where I enabled MPLS and it caused CPU to spike. So to check what is getting forwarded to the MSFC perform the following to see a packet capture of what is hitting the CPU. So you need to run a SNIFFER capture.


Note this is only for the 6500 platform... once you check what's hitting the CPU (eg: fragmentation, icmp type 3 code 4's, etc al)...


Here are the instructions to setup inband span (which monitors traffic sent

to the MSFC):


Router#monitor session 1 source interface fa 3/3 !--- Use any interface that

is administratively shut down.

Router#monitor session 1 destination interface fa 3/2 !-- connected to sniffer port


Now, go to the SP console. Here is an example:


Router#remote login switch

Router-sp#test monitor add 1 rp-inband rx <--- check the syntax as it varies

from one IOS to the next so use ?


Verify monitor session:


Router-sp#test monitor show session 1

Ingress Source Ports: 3/3 15/1

Egress Source Ports: 3/3

Ingress Source Vlans:

Egress Source Vlans:

Filter Vlans:

Destination Ports: 3/2


Go back to the RP and verify the monitor session as well:


Router#show monitor

Session 1

---------

Type : Local Session

Source Ports :

Both : Fa3/3

Destination Ports : Fa3/2

SP console:

Router-sp#test monitor session 1 show

Ingress Source Ports: 3/3 15/1

Egress Source Ports: 3/3

Ingress Source Vlans:

Egress Source Vlans:

Filter Vlans:

Destination Ports: 3/2


To remove the inband span from sp do test monitor session 1 del and from the rp do no mon sess all

===============================

So my exact issue was two parts, CPU spike up 99% when WCCP was enabled with MPLS tag switching at the same time. Rebooting didnt help... I ended up changing from SXF10 to SXF14 IOS just in case but I still had high CPU after upgrade. But the major difference was when I removed the ACL tied to WCCP and removing the ACL completely and re-pasting into the configs for WAAS and then reapplying. But in my sniffer capture I did notice a lot of ip fragmentation hitting the MSFC processor. Now my MTU is 1522 and works fine.

lawwm Fri, 03/06/2009 - 17:35
User Badges:

autobot130,

I really appreciated your solution. I will try it out. Actually my scenario is


same location (1 year)

7609<-->12410

1GE


due to additional traffic, the traffic hit 1.2G caused some of packet dropped


I was forced to bundle 2GE from(by vendor)


7609<-->12140 with 1508

2GE

All the while, it's fine. Until my users had file complaints that some emails attachment cannot go thru.


Even it's locate LAN, MPLS is enabled. I calc. 8 extra headers are needed. I tried to add a bit 1512 and test with users(improvement). The CPU load was >70%. I tried add 1522, whoo 98%.




I think part of the solution is to sniff it. It's remote site, I will try to do it.


Anyone of you encounter?


Thanks a lot autobo




Actions

This Discussion