CISCO C3750E CPU utilization 100%

nishhhant · ‎09-02-2013

Dear Team,

I am experiencing an issue with my core switch C3750E (L3 switch) with the cpu utilization going 100% and in cpu processes I see the ARP input is using more than 61% of the cpu.

CPU utilization for five seconds: 99%/25%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

12 1313789569 66834997 19657 60.10% 61.13% 61.48% 0 ARP Input

85 75586512 18520293 4081 1.76% 1.46% 1.44% 0 RedEarth Tx Mana

84 27740856 23022207 1204 1.44% 1.00% 0.94% 0 RedEarth I2C dri

214 51056595 110992559 460 1.28% 1.65% 1.64% 0 IP Input

129 26284746 4806428 5468 0.96% 0.83% 0.82% 0 hpm counter proc

Please suggest what I can provide from my end from show run and help me out resolving this issue.

Warm Regards,

Nishant.

Seb Rupik · ‎09-02-2013

Hi Nishant,

The process 'ARP input' indicates the switch is issuing a high number to ARP broadcasts for unlearnt IP addresses. These requests are normally rate-limited at one request every two seconds, so you must be getting a lot of requests for a great many different addresses!

Is any routing configured on this switch?

This could also be the an inidaciton of a malicious process on your network scanning for hosts.

Can you provide a 'sh run' and also 'sh arp | inc Incomplete' .

cheers,

Seb.

nishhhant · ‎09-02-2013

Dear Seb,

Please find the show run and show arp | inc Incomplete below :-

========================================================================

show arp | inc Incomplete

Internet 192.168.142.147 0 Incomplete ARPA

Internet 192.168.149.23 0 Incomplete ARPA

Internet 192.168.149.50 0 Incomplete ARPA

Please let me know if any further details required.

Seb Rupik · ‎09-02-2013

What is the output of:

sh ip arp inspection interfaces

sh ip arp inspection statistics

nishhhant · ‎09-02-2013

Hi Seb,

The interfaces are intentionally into trusted.

Please find the required output below :-

sh ip arp inspection interfaces

Interface Trust State Rate (pps) Burst Interval

--------------- ----------- ---------- --------------

Gi1/0/1 Untrusted 15 1

Gi1/0/2 Untrusted 15 1

Gi1/0/3 Untrusted 15 1

Gi1/0/4 Untrusted 15 1

Gi1/0/5 Trusted None N/A

Gi1/0/6 Trusted None N/A

Gi1/0/7 Trusted None N/A

Gi1/0/8 Trusted None N/A

Gi1/0/9 Trusted None N/A

Gi1/0/10 Untrusted 15 1

Gi1/0/11 Untrusted 15 1

Gi1/0/12 Untrusted 15 1

Gi1/0/13 Untrusted 15 1

Gi1/0/14 Untrusted 15 1

Gi1/0/15 Untrusted 15 1

Gi1/0/16 Untrusted 15 1

Gi1/0/17 Untrusted 15 1

Gi1/0/18 Untrusted 15 1

Gi1/0/19 Untrusted 15 1

Gi1/0/20 Trusted 10 1

Gi1/0/21 Untrusted 15 1

Gi1/0/22 Untrusted 15 1

Gi1/0/23 Untrusted 15 1

Gi1/0/24 Untrusted 15 1

Gi1/1/1 Untrusted 15 1

Gi1/1/2 Untrusted 15 1

Gi1/1/3 Untrusted 15 1

Gi1/1/4 Untrusted 15 1

Te1/1/1 Untrusted 15 1

Te1/1/2 Untrusted 15 1

sh ip arp inspection statistics

Source Mac Validation : Disabled

Destination Mac Validation : Disabled

IP Address Validation : Enabled

No active or enabled vlans on switch.

Seb Rupik · ‎09-02-2013

Does your network design for using DAI adhere to making all host connected interfaces 'untrusted' and links to to switches as 'trusted' .

Is your DAI process running any logging? 'sh ip arp inspection log'

If not can you configure it with 'ip arp inspection log-buffer entries 64', and run the above 'sh' command after a period of time has elapsed.

nishhhant · ‎09-02-2013

Hi Seb,

the sh ip arp inspection log is already enabled from start.

Yes I can assure that we are using DAI process where hosts are connected to untrusted and the links connected to trusted.

I see the ARP entry incompletion in mac table is creating this.

I want to write an access list to deny all the incomplete entries.

Will this be possible ?

Seb Rupik · ‎09-02-2013

Can you show me the output from the DAI log?

The ARP entry is only marked as incomplete after an ARP broadcast has been made and failed to receive a response. If an ACL could be applied it would not have any effect.

nishhhant · ‎09-03-2013

Dear Seb,

Please find the log detail below :-

Total Log Buffer Size : 32

Syslog rate : 5 entries per 1 seconds.

Smartlog is not enabled

No entries in log buffer.

Seb Rupik · ‎09-03-2013

Hi Nishant,

Since the DAI approach isn't presenting any information, I suggest you run tcpdump/wireshark on each VLAN in turn and filter for ARP packets. It shouldn't take too long to gain metrics on amount of traffic and the most prolific sources.

cheers,

Seb.

nishhhant · ‎09-03-2013

Hi Seb,

Thanks a ton for your reply.

I did run wireshark before and found out that most of the traffic in the 142 vlan is broadcasting and trying to search many hosts which doesnt exist in my vlan. That is the reason my L3 core switch is unable to resolve the ARP requests and hence the ARP input is going high.

Could you please provide any solution on removing this broadcast. I also applied port security which doesnt allow traffic from one PC to another within the same Vlan that is 142.

Warm Regards,

Nishant.

Leo Laohoo · ‎09-03-2013

What IOS version are you running?

nishhhant · ‎09-03-2013

Hi Leo,

My IOS version is 12.2(58)SE2.

Leo Laohoo · ‎09-03-2013

My IOS version is 12.2(58)SE2.

Ok, then this is "expected" when you have one of these IOS versions.

Unless you have required features only present from 12.2(58)SE and later, try using 12.2(55)SE8.

nishhhant · ‎09-03-2013

Hi Leo,

Do you want me to downgrade the IOS version from 12.2(58)SE2 to 12.2(55)SE8 ?