cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
8868
Views
0
Helpful
16
Replies

6509E-VSS high CPU process due to ARP input

eek
Level 1
Level 1

We are experiencing with high CPU input due to ARP input between 20:30 and 22:30 every day

At this time we have a lot of backup operations. When I look the netflow report, I can't see anything anormal.

We are changing our backup server's NIC card from 1gig to 10Gig. The backup operation's traffic is high (approx 2Gbps level) but 6509 has to be handle this size of traffic.

We are using two 6509E in VSS mode

and our image version is  s72033-adventerprisek9_wan-mz.122-33.SXJ.bin

What the problem cause may be?

20:00

  show ip arp summary

  --------------

  2588 IP ARP entries, with 166 of them incomplete

   show process cpu sorted

--------------

  CPU utilization for five seconds: 11%/3%; one minute: 9%; five minutes: 10%

   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

     3         124       130        953  1.67%  0.13%  0.02%   6 SSH Process     

     9     7713920    452047      17064  1.19%  0.47%  0.37%   0 Check heaps    

   306    45535792 118795990        383  0.79%  0.73%  0.73%   0 Earl NDE Task  

   327    18046456 124766366        144  0.71%  0.54%  0.50%   0 IP Input       

   558     9185532  35575478        258  0.71%  0.36%  0.35%   0 Port manager per

    12   263195052 123093541       2138  0.47%  0.52%  0.55%   0 ARP Input      

20:30

  show ip arp summary

  --------------

  2587 IP ARP entries, with 165 of them incomplete

  show process cpu sorted

  --------------

  CPU utilization for five seconds: 88%/29%; one minute: 82%; five minutes: 83%

   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

    12   263956908 123363086       2139 42.39% 39.49% 40.28%   0 ARP Input      

   397    26557084  28759093        923  4.71%  4.33%  4.32%   0 XDR mcast      

     9     7721604    452477      17065  3.11%  0.52%  0.38%   0 Check heaps    

   306    45582404 118903810        383  2.79%  2.38%  2.00%   0 Earl NDE Task  

     3         168       130       1292  1.67%  0.13%  0.02%   6 SSH Process    

21:00

show ip arp summary

  --------------

  2589 IP ARP entries, with 167 of them incomplete

  show process cpu sorted

  --------------

  CPU utilization for five seconds: 81%/27%; one minute: 83%; five minutes: 79%

   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

    12   265046164 123738558       2141 37.91% 39.18% 37.08%   0 ARP Input      

   397    26664908  28858941        923  5.43%  4.55%  4.26%   0 XDR mcast      

   306    45645772 119037173        383  3.11%  2.09%  1.95%   0 Earl NDE Task  

     3         176       130       1353  1.67%  0.13%  0.02%   6 SSH Process    

   327    18089292 124960978        144  1.11%  0.71%  0.70%   0 IP Input       

21:30

  show ip arp summary

  --------------

  2589 IP ARP entries, with 168 of them incomplete

  --------------

  show process cpu sorted

  --------------

  CPU utilization for five seconds: 98%/40%; one minute: 83%; five minutes: 79%

   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

    12   266025804 124084184       2143 41.67% 37.30% 36.26%   0 ARP Input      

   397    26760700  28950445        924  4.07%  3.98%  3.92%   0 XDR mcast      

    24    18692320 127459398        146  1.75%  1.07%  0.83%   0 IPC Seat Manager

     3         160       130       1230  1.67%  0.13%  0.02%   6 SSH Process    

   327    18110704 125057205        144  1.27%  0.78%  0.70%   0 IP Input       

22:00

  show ip arp summary

  --------------

  2586 IP ARP entries, with 164 of them incomplete

  --------------

  show process cpu sorted

  --------------

  CPU utilization for five seconds: 23%/9%; one minute: 65%; five minutes: 72%

   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

    12   266948752 124452798       2144  7.27% 28.77% 32.71%   0 ARP Input      

     3         144       130       1107  1.67%  0.13%  0.02%   6 SSH Process    

   306    45782092 119315110        383  1.59%  2.04%  2.08%   0 Earl NDE Task  

   397    26863336  29062080        924  1.11%  3.87%  4.44%   0 XDR mcast      

   327    18132460 125159162        144  0.79%  0.66%  0.69%   0 IP Input        

16 Replies 16

Hi ,

You can check :

sh ip traffic | s ARP

Also you can use control-plane policy for ARP :

mls qos protocol ARP police

Dan

Hi, thanks for your answer,

When I look for ip traffic, I see that Receive Replies is increasing rapidly..

***

C6509-251-DC#clear ip traffic    

Clear "show ip traffic" counters [confirm]

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#sh ip traffic | s ARP

ARP statistics:

  Rcvd: 36 requests, 8199 replies, 3 reverse, 0 other

  Sent: 14 requests, 8 replies (0 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#sh ip traffic | s ARP

ARP statistics:

  Rcvd: 77 requests, 21586 replies, 11 reverse, 0 other

  Sent: 61 requests, 22 replies (0 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#sh ip traffic | s ARP

ARP statistics:

  Rcvd: 228 requests, 57298 replies, 24 reverse, 0 other

  Sent: 213 requests, 77 replies (2 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#sh ip traffic | s ARP

ARP statistics:

  Rcvd: 335 requests, 82834 replies, 31 reverse, 0 other

  Sent: 344 requests, 114 replies (5 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#

C6509-251-DC#sh ip traffic | s ARP

ARP statistics:

  Rcvd: 482 requests, 121919 replies, 46 reverse, 0 other

  Sent: 491 requests, 166 replies (6 proxy), 0 reverse

  Drop due to input queue full: 0

Erdal

At normal time (when CPU traffic is normal)

this ARP reply count is not increasing;

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 161 requests, 0 replies, 17 reverse, 0 other

  Sent: 33 requests, 57 replies (2 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 199 requests, 0 replies, 22 reverse, 0 other

  Sent: 38 requests, 75 replies (2 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 206 requests, 0 replies, 22 reverse, 0 other

  Sent: 44 requests, 78 replies (2 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 211 requests, 0 replies, 23 reverse, 0 other

  Sent: 46 requests, 78 replies (2 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 277 requests, 0 replies, 25 reverse, 0 other

  Sent: 49 requests, 137 replies (35 proxy), 0 reverse

  Drop due to input queue full: 0

C6509-251-DC#show ip traffic | section ARP

ARP statistics:

  Rcvd: 295 requests, 0 replies, 25 reverse, 0 other

  Sent: 61 requests, 143 replies (35 proxy), 0 reverse

  Drop due to input queue full: 0

Hi ,

You could apply the ARP rate limit, but first of all you must know where are they from :

show interfaces accounting | i Ethe|ARP|Vlan

The second column will show you the number of packets IN. I would try using accounting identify the source VLAN and also source interface.

Dan

Hi,

I get this output;

Vlan1 VL001(MGMT)

                Protocol    PktsIn   CharsIn   PktsOut  CharsOut

                   Other        160       9600          0          0

                      IP  388089272 282981891991  253012817 134920433824

                 DECMOP        418      32186        142      18318

                     ARP      47795    2960482      30521    3418352

Vlan2 VL002

                Protocol    PktsIn   CharsIn   PktsOut  CharsOut

                   Other         10        600          0          0

                      IP 1933102896 727656700757 4009256393 5467437676205

                 DECMOP        125       9625        142      18318

                     ARP   82061561 4923744674     496931   55656272

and export to excel and sorted, I see the ARP input is coming from VLAN2, I will try to get info when the problem occour (22:00 PM at GMT+2)


ProtocolPktsInCharsInPktsOutCharsOut
Vlan2ARP82061561492374467449693155656272
Vlan111ARP35143321113488562526300224
Vlan211ARP2656591593954010019611221952
Vlan102ARP1402058412432181562033472
Vlan110ARP1135486812880781638754256

unfurtunately I cannot see any arp traffic on Layer2 ports, how can i find the source of this arp packet?

Whould I have to enable anything to apply ARP rate limit or  "mls qos protocol ARP police" is enoug for

Hi,

I want to ask this, how can I clear this counters?

Regards..

Hi ,

Clear : clear counters

To enable arp rate-limit you need to enable "mls qos".

Dan

You might try , having in mind that you now know that the problem is on Vlan 2 , using :

show buffers input-interface vlan2

Dan

Hi,

I configured 

mls qos protocol ARP police 64000

but it was distrupt the treaffic, so i had to be remove command.

i thing i has to be configure the correct value, which value whould i use?

this is the show buffers command;

C6509-251-DC#show buffers input-interface vlan 2

  Header DataArea  Pool Rcnt  Size Link  Enc    Flags          Input      Output

47950894  802BCA4 Small    1    60    1    1      200            Vl2        None

47950C28  802BF64 Small    1    60    1    1      200            Vl2        None

479521A0  802CFE4 Small    1    60    1    1      200            Vl2        None

47952534  802D2A4 Small    1    60    1    1      200            Vl2        None

47953E40  802E5E4 Small    1    60    1    1      200            Vl2        None

4795659C  8030424 Small    1    60    1    1      200            Vl2        None

47956930  80306E4 Small    1    60    1    1      200            Vl2        None

4795823C  8031A24 Small    1    60    1    1      200            Vl2        None

4795A998  8033864 Small    1    60    1    1      200            Vl2        None

4795C9CC  8035124 Small    1    60    1    1      200            Vl2        None

4795E2D8  8036464 Small    1    60    1    1      200            Vl2        None

4795F128  8036F64 Small    1    60    1    1      200            Vl2        None

47964E30  803B6E4 Small    1    60    1    1      200            Vl2        None

47965C80  803C1E4 Small    1    60    1    1      200            Vl2        None

47966E64  803CFA4 Small    1    60    1    1      200            Vl2        None

4796A7A4  803FBA4 Small    1    60    1    1      200            Vl2        None

4796AECC  8040124 Small    1    60    1    1      200            Vl2        None

479704AC  8044324 Small    1    60    1    1      200            Vl2        None

47974180  80471E4 Small    1    60    1    1      200            Vl2        None

4797C978  804DA64 Small    1    60    1    1      200            Vl2        None

4797E618  804F064 Small    1    60    1    1      200            Vl2        None

479822EC  8051F24 Small    1    60    1    1      200            Vl2        None

479846B4  8053AA4 Small    1    60    1    1      200            Vl2        None

47989900  80579E4 Small    1    60    1    1      200            Vl2        None

4798C05C  8059824 Small    1    60    1    1      200            Vl2        None

4798D968  805AB64 Small    1    60    1    1      200            Vl2        None

4799248C  805E524 Small    1    60    1    1      200            Vl2        None

479956A4  8060BA4 Small    1    60    1    1      200            Vl2        None

47996FB0  8061EE4 Small    1    60    1    1      200            Vl2        None

479AC39C  8072424 Small    1    60    1    1      200            Vl2        None

479AC730  80726E4 Small    1    60    1    1      200            Vl2        None

479B197C  8076624 Small    1    60    1    1      200            Vl2        None

479B2EF4  80776A4 Small    1    60    1    1      200            Vl2        None

479B40D8  8078464 Small    1    60    1    1      200            Vl2        None

479B52BC  8079224 Small    1    60    1    1      200            Vl2        None

479B9DE0  807CBE4 Small    1    60    1    1      200            Vl2        None

479BAFC4  807D9A4 Small    1    60    1    1      200            Vl2        None

479BCC64  807EFA4 Small    1    60    1    1      200            Vl2        None

479BCFF8  807F264 Small    1    60    1    1      200            Vl2        None

  Header DataArea  Pool           Rcnt  Size  Original   Flags   caller_pc

There is no "correct" value, the value is linked with how many hosts are behind this system.

Let's try:

int vlan 2

ip accounting mac-address input

exit

show int vl 2 mac-accounting

Dan

When i use this command i cant see any anomal trafic at input way, i see anarmous traffic on output side.

When I track this ip address, i saw that this is out backup servers mac address...

directionmacpacketsbyteslast seen
output    0000.c9f2.f38a(66 )3831844 5813M last: 8340ms ago
output    0000.c9f2.f38e(70 )3745234 5681M last: 6252ms ago
output    e41f.1331.4890(1  )6763133061779 last: 6248ms ago
output    001a.64c1.d544(46 )3394110921794 last: 6252ms ago
output    5cf3.fc0b.e724(155)310558412955 last: 972ms ago
output    e41f.13b3.b6f8(21 )2898614008037 last: 6248ms ago

Yes, backup server can be able to get great trafficbut how the 6500 can't be able to handle this size of traffic?

As I understood until now , you problem is INPUT traffic , explicitly ARP Input on Vlan 2.

So I do not belive that the accounted traffic is the one that generates the issue.

Also until now , I've understand that you issue is interminttent , am I right ?

Dan

Hi,

I see that you have a great deal of traffic punted to the CPU at the time of the issue, you can try to use

"debug netdr capture" to see what packets are getting to the CPU, it will give you more info on the flow

https://supportforums.cisco.com/docs/DOC-15608

cheers

Dejan

Hi,

Yes our problem is occours specially at backup time, between 20:30 PM to 22:30 PM and, 00:30 AM to 02:30 AM.

Other than this time range, we have no problem. Also this CPU problem is not interrupting normal traffic at this specific time range..

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: