Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

CISCO C3750E CPU utilization 100%

Dear Team,

I am experiencing an issue with my core switch C3750E (L3 switch) with the cpu utilization going 100% and in cpu processes I see the ARP input is using more than 61% of the cpu.

CPU utilization for five seconds: 99%/25%; one minute: 99%; five minutes: 99%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

  12  1313789569    66834997      19657 60.10% 61.13% 61.48%   0 ARP Input

  85    75586512    18520293       4081  1.76%  1.46%  1.44%   0 RedEarth Tx Mana

  84    27740856    23022207       1204  1.44%  1.00%  0.94%   0 RedEarth I2C dri

214    51056595   110992559        460  1.28%  1.65%  1.64%   0 IP Input

129    26284746     4806428       5468  0.96%  0.83%  0.82%   0 hpm counter proc

Please suggest what I can provide from my end from show run and help me out resolving this issue.

Warm Regards,

Nishant.


Everyone's tags (1)
25 REPLIES

CISCO C3750E CPU utilization 100%

Hi Nishant,

The process 'ARP input' indicates the switch is issuing a high number to ARP broadcasts for unlearnt IP addresses. These requests are normally rate-limited at one request every two seconds, so you must be getting a lot of requests for a great many different addresses!

Is any routing configured on this switch?

This could also be the an inidaciton of a malicious process on your network scanning for hosts.

Can you provide a 'sh run' and also 'sh arp | inc Incomplete' .

cheers,

Seb.

Community Member

Re: CISCO C3750E CPU utilization 100%

Dear Seb,

Please find the show run and show arp | inc Incomplete below :-

========================================================================

show arp | inc Incomplete

Internet  192.168.142.147         0   Incomplete      ARPA

Internet  192.168.149.23          0   Incomplete      ARPA

Internet  192.168.149.50          0   Incomplete      ARPA

Please let me know if any further details required.

Re: CISCO C3750E CPU utilization 100%

What is the output of:


sh ip arp inspection interfaces

sh ip arp inspection statistics

Community Member

CISCO C3750E CPU utilization 100%

Hi Seb,

The interfaces are intentionally into trusted.

Please find the required output below :-

sh ip arp inspection interfaces

Interface        Trust State     Rate (pps)    Burst Interval

---------------  -----------     ----------    --------------

Gi1/0/1          Untrusted               15                 1

Gi1/0/2          Untrusted               15                 1

Gi1/0/3          Untrusted               15                 1

Gi1/0/4          Untrusted               15                 1

Gi1/0/5          Trusted               None               N/A

Gi1/0/6          Trusted               None               N/A

Gi1/0/7          Trusted               None               N/A

Gi1/0/8          Trusted               None               N/A

Gi1/0/9          Trusted               None               N/A

Gi1/0/10         Untrusted               15                 1

Gi1/0/11         Untrusted               15                 1

Gi1/0/12         Untrusted               15                 1

Gi1/0/13         Untrusted               15                 1

Gi1/0/14         Untrusted               15                 1

Gi1/0/15         Untrusted               15                 1

Gi1/0/16         Untrusted               15                 1

Gi1/0/17         Untrusted               15                 1

Gi1/0/18         Untrusted               15                 1

Gi1/0/19         Untrusted               15                 1

Gi1/0/20         Trusted                 10                 1

Gi1/0/21         Untrusted               15                 1

Gi1/0/22         Untrusted               15                 1

Gi1/0/23         Untrusted               15                 1

Gi1/0/24         Untrusted               15                 1

Gi1/1/1          Untrusted               15                 1

Gi1/1/2          Untrusted               15                 1

Gi1/1/3          Untrusted               15                 1

Gi1/1/4          Untrusted               15                 1

Te1/1/1          Untrusted               15                 1

Te1/1/2          Untrusted               15                 1

sh ip arp inspection statistics

Source Mac Validation      : Disabled

Destination Mac Validation : Disabled

IP Address Validation      : Enabled

No active or enabled vlans on switch.

CISCO C3750E CPU utilization 100%

Does your network design for using DAI adhere to making all host connected interfaces 'untrusted' and links to to switches as 'trusted' .

Is your DAI process running any logging? 'sh ip arp inspection log'

If not can you configure it with 'ip arp inspection log-buffer entries 64', and run the above 'sh' command after a period of time has elapsed.

Community Member

CISCO C3750E CPU utilization 100%

Hi Seb,

the sh ip arp inspection log is already enabled from start.

Yes I can assure that we are using DAI process where hosts are connected to untrusted and the links connected to trusted.

I see the ARP entry incompletion in mac table is creating this.

I want to write an access list to deny all the incomplete entries.

Will this be possible ?

CISCO C3750E CPU utilization 100%

Can you show me the output from the DAI log?

The ARP entry is only marked as incomplete after an ARP broadcast has been made and failed to receive a response. If an ACL could be applied it would not have any effect.

Community Member

CISCO C3750E CPU utilization 100%

Dear Seb,

Please find the log detail below :-

Total Log Buffer Size : 32

Syslog rate : 5 entries per 1 seconds.

Smartlog is not enabled

No entries in log buffer.

CISCO C3750E CPU utilization 100%

Hi Nishant,

Since the DAI approach isn't presenting any information, I suggest you run tcpdump/wireshark on each VLAN in turn and filter for ARP packets. It shouldn't take too long to gain metrics on amount of traffic and the most prolific sources.

cheers,

Seb.

Community Member

CISCO C3750E CPU utilization 100%

Hi Seb,

Thanks a ton for your reply.

I did run wireshark before and found out that most of the traffic in the 142 vlan is broadcasting and trying to search many hosts which doesnt exist in my vlan. That is the reason my L3 core switch is unable to resolve the ARP requests and hence the ARP input is going high.

Could you please provide any solution on removing this broadcast. I also applied port security which doesnt allow traffic from one PC to another within the same Vlan that is 142.

Warm Regards,

Nishant.

Hall of Fame Super Blue

CISCO C3750E CPU utilization 100%

What IOS version are you running?

Community Member

CISCO C3750E CPU utilization 100%

Hi Leo,

My IOS version is 12.2(58)SE2.

Hall of Fame Super Blue

CISCO C3750E CPU utilization 100%

My IOS version is 12.2(58)SE2.

Ok, then this is "expected" when you have one of these IOS versions.

Unless you have required features only present from 12.2(58)SE and later, try using 12.2(55)SE8.

Community Member

CISCO C3750E CPU utilization 100%

Hi Leo,

Do you want me to downgrade the IOS version from 12.2(58)SE2 to 12.2(55)SE8 ?

990
Views
0
Helpful
25
Replies
CreatePlease to create content