Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

ME 3400 Switch High CPU Utilization due to ARP input problem

We have Cisco ME3400 switch, with configuration described below.

GE0/1 port is uplink:

interface GigabitEthernet0/1

description ***MATED-IUB_C&U-Plan***

port-type nni

switchport trunk allowed vlan 331,450,4085

switchport mode trunk

GEo/2 is configured as below:

interface GigabitEthernet0/2

description ***From NIB***

port-type nni

switchport access vlan 4085

FA0/3 and FA0/15 are configured as below:

interface FastEthernet0/15 (and 3)

description ***NODE-B for rnc07***

port-type nni

switchport access vlan 4085

Now the problem is the some time the CPU Utilization is increased abruptly.

The spanning Tree configuration is biven below:

spanning-tree mode rapid-pvst

spanning-tree extend system-id

and IP Route is given below:

interface Vlan331

ip address 10.135.232.148 255.255.255.240

!

interface Vlan450

ip address 10.72.228.194 255.255.255.252

!

interface Vlan550

ip address 10.72.228.202 255.255.255.252

shutdown

!

interface Vlan4085

ip address 10.72.234.254 255.255.255.0

!

ip http server

no ip http secure-server

ip classless

ip route 0.0.0.0 0.0.0.0 10.135.232.158

ip route 10.72.224.96 255.255.255.240 10.72.228.193

ip route 10.72.228.64 255.255.255.240 10.72.228.193

!

At the time of High cpu utilization we get:

rjrncbb1#  show processes cpu history sorted 1min

CPU utilization for five seconds: 98%/29%; one minute: 98%; five minutes: 98%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   7    90596994  55277982       1638 64.21% 64.67% 64.73%   0 ARP Input       

195        4130      3888       1062  0.15%  0.25%  0.08%   1 Virtual Exec 

and interface summary is :

rjrncbb1#show interfaces summary

*: interface is up

IHQ: pkts in input hold queue     IQD: pkts dropped from input queue

OHQ: pkts in output hold queue    OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec)          RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec)          TXPS: tx rate (pkts/sec)

TRTL: throttle count

  Interface              IHQ   IQD  OHQ   OQD  RXBS RXPS  TXBS TXPS TRTL

------------------------------------------------------------------------

  Vlan1                    0     0    0     0     0    0     0    0    0

* Vlan331                  0     0    0     0     0    1     0    0    0

  Vlan337                  0     0    0     0     0    0     0    0    0

* Vlan450                  0     0    0     0     0    1     0    0    0

  Vlan550                  0     0    0     0     0    0     0    0    0

* Vlan4085                17  5533    0     0 1421000  2943 76000  151    0

  FastEthernet0/1          0     0    0     0     0    0     0    0    0

  FastEthernet0/2          0     0    0     0     0    0     0    0    0

* FastEthernet0/3          0     0    0     0 50000   50 2484000  4529    0

  FastEthernet0/4          0     0    0     0     0    0     0    0    0

* FastEthernet0/5          0     0    0     0  4000    8 2463000  4498    0

  FastEthernet0/6          0     0    0     0     0    0     0    0    0

  FastEthernet0/7          0     0    0     0     0    0     0    0    0

  Interface              IHQ   IQD  OHQ   OQD  RXBS RXPS  TXBS TXPS TRTL

------------------------------------------------------------------------

  FastEthernet0/8          0     0    0     0     0    0     0    0    0

  FastEthernet0/9          0     0    0     0     0    0     0    0    0

  FastEthernet0/10         0     0    0     0     0    0     0    0    0

  FastEthernet0/11         0     0    0     0     0    0     0    0    0

  FastEthernet0/12         0     0    0     0     0    0     0    0    0

  FastEthernet0/13         0     0    0     0     0    0     0    0    0

  FastEthernet0/14         0     0    0     0     0    0     0    0    0

* FastEthernet0/15         0     0    0     0 6414000  9113 15109000  13987    0

  FastEthernet0/16         0     0    0     0     0    0     0    0    0

  FastEthernet0/17         0     0    0     0     0    0     0    0    0

  FastEthernet0/18         0     0    0     0     0    0     0    0    0

  FastEthernet0/19         0     0    0     0     0    0     0    0    0

  FastEthernet0/20         0     0    0     0     0    0     0    0    0

  FastEthernet0/21         0     0    0     0     0    0     0    0    0

  FastEthernet0/22         0     0    0     0     0    0     0    0    0

* FastEthernet0/23         0     0    0     0  3000    6 2316000  4493    0

  FastEthernet0/24         0     0    0     0     0    0     0    0    0

* GigabitEthernet0/1       0     0    0     0 40281000  16323 23938000  36323    0

* GigabitEthernet0/2       0     0    0     0 16321000  27183 26887000  28656    0

NOTE:No separate counters are maintained for subinterfaces

     Hence Details of subinterface are not shown

We have also tried Loopguard feature enabled on all ports:

but after some period same problem is repeated.

version information is below:

Cisco IOS Software, ME340x Software (ME340x-METROIPACCESSK9-M), Version 12.2(25)SEG3, RELEASE SOFTWARE (fc2)

Copyright (c) 1986-2007 by Cisco Systems, Inc.

Compiled Wed 25-Jul-07 22:56 by amvarma

Kindly help me on this issue at most urgent.

Thank you all.

8 REPLIES
VIP Purple

ME 3400 Switch High CPU Utilization due to ARP input problem

Hi Sandip,

High CPU utilization in the Address Resolution Protocol (ARP) Input process occurs if the router has to originate an excessive number of ARP requests. The router uses ARP for all hosts, not just those on the local subnet, and ARP requests are sent out as broadcasts, which causes more CPU utilization on every host in the network. ARP requests for the same IP address are rate-limited to one request every two seconds, so an excessive number of ARP requests would have to originate for different IP addresses. This can happen if an IP route has been configured pointing to a broadcast interface. A most obvious example is a default route such as:

ip route 0.0.0.0 0.0.0.0 Fastethernet0/0

In this case, the router generates an ARP request for each IP address that is not reachable through more specific routes, which practically means that the router generates an ARP request for almost every address on the Internet. For more information about configuring next hop address for static routing,

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml

or

Alternatively, an excessive amount of ARP requests can be caused by a malicious traffic stream which scans through locally attached subnets. An indication of such a stream would be the presence of a very high number of incomplete ARP entries in the ARP table. Since incoming IP packets that would trigger ARP requests would have to be processed, troubleshooting this problem would essentially be the same as troubleshooting high CPU utilization in the

IP Input process

In addition to the other post would like to suggest to check out whether ip proxy-arp is enabled on any of the interfaces.

if yes try to disable the same on them and check out ...

it also proves to be a possible cause for the same..

http://www.cisco.com/warp/public/63/highcpu_processes.html#arp

Regards

Please rate if it helps.

New Member

ME 3400 Switch High CPU Utilization due to ARP input problem

I've changed 0.0.0.0 route to 10.0.0.0

ip route 10.0.0.0 255.0.0.0 10.135.232.158

ip route 10.72.224.96 255.255.255.240 10.72.228.193

ip route 10.72.228.64 255.255.255.240 10.72.228.193

But still the problem is the same..the cpu utilization increases abruptly due to ARP Input....below is the summary

rjrncbb1#show processes cpu sorted 1min

CPU utilization for five seconds: 72%/27%; one minute: 83%; five minutes: 43%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   7      127110    148856        853 39.93% 47.42% 24.51%   0 ARP Input       

189        1183       419       2823  0.31%  0.34%  0.19%   1 Virtual Exec    

200        1198         7     171142  0.00%  0.15%  0.17%   0 hulc running con

146         466      1273        366  0.00%  0.15%  0.10%   0 IP Input        

   4         295        31       9516  0.00%  0.04%  0.01%   0 Check heaps     

108         193       137       1408  0.00%  0.03%  0.02%   0 HRPC qos request

  99          66      5520         11  0.00%  0.01%  0.00%   0 Hulc LED Process

  28          59         5      11800  0.00%  0.01%  0.00%   0 Per-minute Jobs 

107          56        65        861  0.00%  0.01%  0.00%   0 HQM Stack Proces

please help me ...

I've also observed, a drop in Input que....at the time of high cpu utilization.

Cisco Employee

ME 3400 Switch High CPU Utilization due to ARP input problem

Hi Sandip,

how often do you see ARP input process going up? can you add 'show process cpu history'?

How many hosts are normally directly connected to this device?

can you also take 3 instances of 'show ip traffic' with 2-3 seconds interval between each of them and then take 3 more with same interval when you see high cpu.

take also 2 instances of 'sh interfa | i is up|drop|put rate|thro' with 1 second interval and again 2 more when CPU is high.

Riccardo

New Member

Re: ME 3400 Switch High CPU Utilization due to ARP input problem

The hosts are connected to various locations, but connected to switch trhough single GE interface 0/2.

And some 4 to  5 hosts are connected to single FE port 0/15.

Well as per last two days observation, the ARP input process going up on aroung 15:00 in the day time.

But when I gave the following command,

ARP Timeout 30 (under Interface VLAN 4085).

And the show ip arp summary shows 96 ARP entries with 25 of them are incomplete status

Can you tell me whether this command worked for me or there might be some other reason of the ARP Processes suddenly gets down?

VIP Purple

ME 3400 Switch High CPU Utilization due to ARP input problem

HI Sandip,

please paste the output of these command:

sh ip arp summary

show interfaces accounting | i Ethe|ARP|Vlan

first we should know that, from where these traffic  is generating?

Regards

New Member

Re: ME 3400 Switch High CPU Utilization due to ARP input problem

Pls find the response to the above command below:

rjrncbb1#show ip arp summary

99 IP ARP entries, with 34 of them incomplete

rjrncbb1#show interfaces accounting | i Ethe|ARP|Vlan

Vlan1

Vlan331

                     ARP          4        240         46       2760

Interface Vlan337 is disabled

Vlan450

                     ARP          1         60          6        360

Interface Vlan550 is disabled

Vlan4085

                     ARP   17371462 1042318684     541831   32509860

Interface FastEthernet0/1 is disabled

FastEthernet0/2

FastEthernet0/3 ***NODE-B for rnc07***

FastEthernet0/4 ***NODE-B for rnc08***

FastEthernet0/5 ***connected_to_BSS-Switch***

FastEthernet0/6 ***connected_to_BSS-Switch***

FastEthernet0/15 ***NODE-B for rnc07***

GigabitEthernet0/1 ***MATED-IUB_C&U-Plan***

GigabitEthernet0/2 ***From NIB***

Pls guide me what to do next???

VIP Purple

ME 3400 Switch High CPU Utilization due to ARP input problem

HI Sandip,

I think the ARP input is coming from VLAN4085

paste the output of this command:show buffers input-interface vlan 4085

questions:

When this problem occurs(Is there any time slot , when it comes high??? like when u r doing backup or  ??)

follow this document:

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af6.shtml

Regards

please rate if it helps.

New Member

Re: ME 3400 Switch High CPU Utilization due to ARP input problem

rjrncbb1#show buffers input-interface vlan 4085

  Header DataArea  Pool Rcnt  Size Link  Enc    Flags          Input      Output

249E780  741A574 RxQFB    1    60    1    1      210         Vl4085        None

275D400  74C35E4 RxQFB    1    60    1    1      210         Vl4085        None

27605B0  74CBA14 RxQFB    1    60    1    1      210         Vl4085        None

2760900  74CC2E4 RxQFB    1    60    1    1      210         Vl4085        None

2761CE0  74CF7C4 RxQFB    1    60    1    1      210         Vl4085        None

2763410  74D3574 RxQFB    1    60    1    1      210         Vl4085        None

2767300  74DDCE4 RxQFB    1    60    1    1      210         Vl4085        None

2767650  74DE5B4 RxQFB    1    60    1    1      210         Vl4085        None

2768A30  74E1A94 RxQFB    1    60    1    1      210         Vl4085        None

276A160  74E5844 RxQFB    1    60    1    1      210         Vl4085        None

276A4B0  74E6114 RxQFB    1    60    1    1      210         Vl4085        None

276C920  74EC204 RxQFB    1    60    1    1      210         Vl4085        None

276D9B0  74EEE14 RxQFB    1    60    1    1      210         Vl4085        None

276EA40  74F1A24 RxQFB    1    60    1    1      210         Vl4085        None

276ED90  74F22F4 RxQFB    1    60    1    1      210         Vl4085        None

2770B60  74F7244 RxQFB    1    60    1    1      210         Vl4085        None

2771BF0  74F9E54 RxQFB    1    60    1    1      210         Vl4085        None

2773670  74FE4D4 RxQFB    1    60    1    1      210         Vl4085        None

2797A20  755EB04 RxQFB    1    60    1    1      210         Vl4085        None

27A0890  75765F4 RxQFB    1    60    1    1      210         Vl4085        None

27A7290  7587FF4 RxQFB    1    60    1    1      210         Vl4085        None

27A9700  758E0E4 RxQFB    1    60    1    1      210         Vl4085        None

290D8A0  75D2E74 RxQFB    1    60    1    1      210         Vl4085        None

2911AE0  75DDEB4 RxQFB    1    60    1    1      210         Vl4085        None

291DE50  75FE6A4 RxQFB    1    60    1    1      210         Vl4085        None

291EB90  76009E4 RxQFB    1    60    1    1      210         Vl4085        None

2927A00  76184D4 RxQFB    1    60    1    1      210         Vl4085        None

292FE80  762E554 RxQFB    1    60    1    1      210         Vl4085        None

293B160  764C134 RxQFB    1    60    1    1      210         Vl4085        None

293DFC0  7653C94 RxQFB    1    60    1    1      210         Vl4085        None

2941B60  765DB34 RxQFB    1    60    1    1      210         Vl4085        None

2946790  766A5E4 RxQFB    1    60    1    1      210         Vl4085        None

294DB80  767DA54 RxQFB    1    60    1    1      210         Vl4085        None

294F2B0  7681804 RxQFB    1    60    1    1      210         Vl4085        None

2950690  7684CE4 RxQFB    1    60    1    1      210         Vl4085        None

295A8E0  769FCB4 RxQFB    1    60    1    1      210         Vl4085        None

30207C0  7B39EDC  RxQ8    1    60    1    1      200         Vl4085        None

3022590  7B3EE2C  RxQ8    1    60    1    1      200         Vl4085        None

30232D0  7B4116C  RxQ8    1    60    1    1      200         Vl4085        None

3024D50  7B457EC  RxQ8    1    60    1    1      200         Vl4085        None

3026130  7B48CCC  RxQ8    1    60    1    1      200         Vl4085        None

3027510  7B4C1AC  RxQ8    1    60    1    1      200         Vl4085        None

3028C40  7B4FF5C  RxQ8    1    60    1    1      200         Vl4085        None

3029CD0  7B52B6C  RxQ8    1    60    1    1      200         Vl4085        None

302BDF0  7B5838C  RxQ8    1    60    1    1      200         Vl4085        None

302C7E0  7B59DFC  RxQ8    1    60    1    1      200         Vl4085        None

30BC0D0  7BAB01C  RxQ3    1    60    1    1      210         Vl4085        None

30BCE10  7BAD35C  RxQ3    1    60    1    1      210         Vl4085        None

30BF280  7BB344C  RxQ3    1    60    1    1      210         Vl4085        None

30BF5D0  7BB3D1C  RxQ3    1    60    1    1      210         Vl4085        None

30C4550  7BC109C  RxQ3    1    60    1    1      210         Vl4085        None

30C48A0  7BC196C  RxQ3    1    60    1    1      210         Vl4085        None

30C5FD0  7BC571C  RxQ3    1    60    1    1      210         Vl4085        None

30C6320  7BC5FEC  RxQ3    1    60    1    1      210         Vl4085        None

30C6670  7BC68BC  RxQ3    1    60    1    1      210         Vl4085        None

30C69C0  7BC718C  RxQ3    1    60    1    1      210         Vl4085        None

30C6D10  7BC7A5C  RxQ3    1    60    1    1      210         Vl4085        None

30C7060  7BC832C  RxQ3    1    60    1    1      210         Vl4085        None

30C73B0  7BC8BFC  RxQ3    1    60    1    1      210         Vl4085        None

30C7A50  7BC9D9C  RxQ3    1    60    1    1      210         Vl4085        None

30C7DA0  7BCA66C  RxQ3    1    60    1    1      210         Vl4085        None

30C80F0  7BCAF3C  RxQ3    1    60    1    1      210         Vl4085        None

30C8440  7BCB80C  RxQ3    1    60    1    1      210         Vl4085        None

30C8790  7BCC0DC  RxQ3    1    60    1    1      210         Vl4085        None

30C8AE0  7BCC9AC  RxQ3    1    60    1    1      210         Vl4085        None

30C9180  7BCDB4C  RxQ3    1    60    1    1      210         Vl4085        None

30C9820  7BCECEC  RxQ3    1    60    1    1      210         Vl4085        None

30CB5F0  7BD3C3C  RxQ3    1    60    1    1      210         Vl4085        None

30CC680  7BD684C  RxQ3    1    60    1    1      210         Vl4085        None

30D0C10  7BE215C  RxQ3    1    60    1    1      210         Vl4085        None

30D3080  7BE824C  RxQ3    1    60    1    1      210         Vl4085        None

  Header DataArea  Pool           Rcnt  Size  Original   Flags   caller_pc

This problem might occur due to some wrong configuration on the other end of router, Gig 0/1, RNC end. and there is no any fix time slot..

2847
Views
10
Helpful
8
Replies
CreatePlease to create content