I am experiencing an issue with my core switch C3750E (L3 switch) with the cpu utilization going 100% and in cpu processes I see the ARP input is using more than 61% of the cpu.
CPU utilization for five seconds: 99%/25%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 1313789569 66834997 19657 60.10% 61.13% 61.48% 0 ARP Input
85 75586512 18520293 4081 1.76% 1.46% 1.44% 0 RedEarth Tx Mana
84 27740856 23022207 1204 1.44% 1.00% 0.94% 0 RedEarth I2C dri
214 51056595 110992559 460 1.28% 1.65% 1.64% 0 IP Input
129 26284746 4806428 5468 0.96% 0.83% 0.82% 0 hpm counter proc
Please suggest what I can provide from my end from show run and help me out resolving this issue.
The process 'ARP input' indicates the switch is issuing a high number to ARP broadcasts for unlearnt IP addresses. These requests are normally rate-limited at one request every two seconds, so you must be getting a lot of requests for a great many different addresses!
Is any routing configured on this switch?
This could also be the an inidaciton of a malicious process on your network scanning for hosts.
Can you provide a 'sh run' and also 'sh arp | inc Incomplete' .
Please find the show run and show arp | inc Incomplete below :-
show arp | inc Incomplete
Internet 192.168.142.147 0 Incomplete ARPA
Internet 192.168.149.23 0 Incomplete ARPA
Internet 192.168.149.50 0 Incomplete ARPA
Please let me know if any further details required.
The interfaces are intentionally into trusted.
Please find the required output below :-
sh ip arp inspection interfaces
Interface Trust State Rate (pps) Burst Interval
--------------- ----------- ---------- --------------
Gi1/0/1 Untrusted 15 1
Gi1/0/2 Untrusted 15 1
Gi1/0/3 Untrusted 15 1
Gi1/0/4 Untrusted 15 1
Gi1/0/5 Trusted None N/A
Gi1/0/6 Trusted None N/A
Gi1/0/7 Trusted None N/A
Gi1/0/8 Trusted None N/A
Gi1/0/9 Trusted None N/A
Gi1/0/10 Untrusted 15 1
Gi1/0/11 Untrusted 15 1
Gi1/0/12 Untrusted 15 1
Gi1/0/13 Untrusted 15 1
Gi1/0/14 Untrusted 15 1
Gi1/0/15 Untrusted 15 1
Gi1/0/16 Untrusted 15 1
Gi1/0/17 Untrusted 15 1
Gi1/0/18 Untrusted 15 1
Gi1/0/19 Untrusted 15 1
Gi1/0/20 Trusted 10 1
Gi1/0/21 Untrusted 15 1
Gi1/0/22 Untrusted 15 1
Gi1/0/23 Untrusted 15 1
Gi1/0/24 Untrusted 15 1
Gi1/1/1 Untrusted 15 1
Gi1/1/2 Untrusted 15 1
Gi1/1/3 Untrusted 15 1
Gi1/1/4 Untrusted 15 1
Te1/1/1 Untrusted 15 1
Te1/1/2 Untrusted 15 1
sh ip arp inspection statistics
Source Mac Validation : Disabled
Destination Mac Validation : Disabled
IP Address Validation : Enabled
No active or enabled vlans on switch.
Does your network design for using DAI adhere to making all host connected interfaces 'untrusted' and links to to switches as 'trusted' .
Is your DAI process running any logging? 'sh ip arp inspection log'
If not can you configure it with 'ip arp inspection log-buffer entries 64', and run the above 'sh' command after a period of time has elapsed.
the sh ip arp inspection log is already enabled from start.
Yes I can assure that we are using DAI process where hosts are connected to untrusted and the links connected to trusted.
I see the ARP entry incompletion in mac table is creating this.
I want to write an access list to deny all the incomplete entries.
Will this be possible ?
Can you show me the output from the DAI log?
The ARP entry is only marked as incomplete after an ARP broadcast has been made and failed to receive a response. If an ACL could be applied it would not have any effect.
Please find the log detail below :-
Total Log Buffer Size : 32
Syslog rate : 5 entries per 1 seconds.
Smartlog is not enabled
No entries in log buffer.
Since the DAI approach isn't presenting any information, I suggest you run tcpdump/wireshark on each VLAN in turn and filter for ARP packets. It shouldn't take too long to gain metrics on amount of traffic and the most prolific sources.
Thanks a ton for your reply.
I did run wireshark before and found out that most of the traffic in the 142 vlan is broadcasting and trying to search many hosts which doesnt exist in my vlan. That is the reason my L3 core switch is unable to resolve the ARP requests and hence the ARP input is going high.
Could you please provide any solution on removing this broadcast. I also applied port security which doesnt allow traffic from one PC to another within the same Vlan that is 142.
My IOS version is 12.2(58)SE2.
Ok, then this is "expected" when you have one of these IOS versions.
Unless you have required features only present from 12.2(58)SE and later, try using 12.2(55)SE8.