High CPU utilization due to high ARP Input process

Unanswered Question
Mar 17th, 2010
User Badges:

Hi,


One of the Cat 6500 VSS switches have been experiencing high cpu peaks for sometime. On analysis it was observed that it was due to high 'ARP Input' process. There are no static routes configured in this switch, no incomplete ARP entries or any inferences of DoS attack.


/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman"; mso-ansi-language:#0400; mso-fareast-language:#0400; mso-bidi-language:#0400;} /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman"; mso-ansi-language:#0400; mso-fareast-language:#0400; mso-bidi-language:#0400;}

DIST_SW>sh proc cpu

CPU utilization for five seconds: 98%/31%; one minute: 79%; five minutes: 37%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   5    39583544   2366177      16728  0.00%  0.24%  0.36%   0 Check heaps

   8   234378596 161314924       1452 61.43% 45.63% 17.09%   0 ARP Input

  20   1373772961296358745        105  0.00%  0.53%  0.53%   0 IPC Seat Manager


I suspect it could be due to "proxy-arp" turned on by default under the  interfaces and arranging to disable it. I've also started engaging server teams to verify if the subnet mask & default-gateway are configured correctly in all the servers along with any static routes configured pointing to a NIC as next-hop.


/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman"; mso-ansi-language:#0400; mso-fareast-language:#0400; mso-bidi-language:#0400;}

UK_PR_DIST_02>sh ip traffic | b ARP

ARP statistics:

  Rcvd: 1752882295 requests, 30808911 replies, 3228 reverse, 0 other

  Sent: 4517117 requests, 1450623731 replies (1437736195 proxy), 0 reverse

  Drop due to input queue full: 0


Apart from proxy-arp is there anything that I need to check?


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Aaron Harrison Wed, 03/17/2010 - 02:42
User Badges:
  • Super Bronze, 10000 points or more
  • Community Spotlight Award,

    Member's Choice, May 2015

Hi


Maybe try a packet cap to see what is generating all the ARP packets...


Aaron

vijayaram Wed, 03/17/2010 - 05:24
User Badges:

Hi, thanks for your reply. I had already sniffed the traffic and shared ip /mac addr of hosts innolved in ARP broadcasts. However server admin didnt find any anamoly with the NIC settings. Disabling proxy ARP is more of protecting switches from being hit by ARP storm.

Actions

This Discussion