Re: ARP issue. Router CPU running over 70%

enriquebs · ‎12-13-2006

HI all,

We are having an issue in a LAN. The box is a Cisco 1721. The FastEthernet1 is facing a switch. The box is learning an IP address which dosen't exist, then it is trying to get the MAC all the time overloading the CPU. If we shut the interface the box responds good. Here is some information (debug and show commands).

Dec 13 14:17:37: IP ARP req filtered src 10.136.133.254 0016.4628.4ff9, dst 10.136.133.230 0000.0000.0000 it's

our address

Dec 13 14:17:37: IP ARP req filtered src 10.136.133.254 0016.4628.4ff9, dst 10.136.133.230 0000.0000.0000 it's

our address

Dec 13 14:17:37: IP ARP req filtered src 10.136.133.254 0016.4628.4ff9, dst 10.136.133.230 0000.0000.0000 it's

our address

Dec 13 14:17:37: IP ARP req filtered src 10.136.133.254 0016.4628.4ff9, dst 10.136.133.230 0000.0000.0000 it's

our address

Dec 13 14:17:37: IP ARP req filtered src 10.136.133.254 0016.4628.4ff9, dst 10.136.133.230 0000.0000.0000 it's

our address

and here is the show ip arp:

Internet 10.136.133.230 0 Incomplete ARPA

Thx a lot in advance.

Cheers,

sourabhagarwal · ‎12-13-2006

you mean IPs 10.136.133.254 and 10.136.133.230 doesn't belong to your network?

can you put output of show process when you are observing high CPU utilization.

dgahm · ‎12-13-2006

Your router is trying to route packets with a destination address of 10.136.133.230. Since that is on a connected network the router ARPs to learn the MAC address and gets no response. The packets will then be discarded. This is all normal router operation. There will always be some undeliverable packets. Are you sure this is what is overloading the router?

show process cpu sorted

What is the source address of the packets? A filtered debug ip packet can tell you (be sure and use an access list filter). Once you know that you could apply an access list to the inbound inteface (ip access-group in) to block these packets.

enriquebs · ‎12-14-2006

Here is the output:

EDU215#sh proc cpu sorted

CPU utilization for five seconds: 59%/36%; one minute: 56%; five minutes: 56%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

8 47146068 55678337 846 18.81% 19.55% 18.35% 0 ARP Input

115 4121784 15372007 268 1.47% 1.37% 1.37% 0 COLLECT STAT COU

116 1480 1586 933 1.22% 0.12% 0.03% 6 Virtual Exec

4 4163488 465793 8938 0.73% 0.13% 0.11% 0 Check heaps

5 62708 13196 4752 0.24% 0.03% 0.00% 0 Pool Manager

41 1037168 1552631 668 0.24% 0.22% 0.22% 0 DSL State Machin

2 323040 775901 416 0.16% 0.06% 0.06% 0 Load Meter

86 823092 7755064 106 0.16% 0.18% 0.21% 0 DHCPD Receive

50 34052352 26970193 1262 0.08% 0.02% 0.05% 0 IP Input

9 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

Both IP addresses belong to my network, I mean, the .254 is the fast1 IP address which is facing the LAN, the .230 is an unknow host (anyway that address match with the network address of the LAN /24).

Could be dealing with an ARP spoofing issue?

cheers,

enriquebs · ‎12-14-2006

or maybe a virus... because if you add up the % of the proccesses don't match with the % average in the first line...

enriquebs · ‎12-14-2006

A little bit more information (debug ip packets)

Dec 14 10:06:10: IP: tableid=1, s=10.10.4.78 (Tunnel6), d=10.136.133.230 (Vlan906), routed via RIB

Dec 14 10:06:10: IP: s=10.10.4.78 (Tunnel6), d=10.136.133.230 (Vlan906), g=10.136.133.230, len 83, forward

Dec 14 10:06:10: ICMP type=8, code=0

Dec 14 10:06:10: IP: s=10.10.4.78 (Tunnel6), d=10.136.133.230 (Vlan906), len 83, encapsulation failed

Dec 14 10:06:10: ICMP type=8, code=0

EDU215#u all

sourabhagarwal · ‎12-14-2006

can you try to locate IP 10.136.133.230 in your network. from logs it appears that this IP belongs to VLAN 906.

are you able to ping this IP from router?

if yes, try to locate the port which is learning it's MAC address and try to shut the port and see if cpu util comes down.

or, you can try to apply ACL to deny this IP and permit any on router fast ethernet interface and see if it helps.

enriquebs · ‎12-14-2006

So... after filtering the 10.10.4.78 with an ACL in the tunnel6 interface the CPU continues running over 50%...

sourabhagarwal · ‎12-14-2006

filter 10.136.133.230 IP which is in your network and then check the cpu usage.

enriquebs · ‎12-14-2006

CPU continues in the same way.

sourabhagarwal · ‎12-14-2006

what is the IOS running on your cisco 1721? can you share output of "show version" with us?

enriquebs · ‎12-14-2006

Sure! here is the output:

EDU215#sh ver

Cisco IOS Software, C1700 Software (C1700-IPBASE-M), Version 12.4(1c), RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 26-Oct-05 06:46 by evmiller

ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)

EDU215 uptime is 1 hour, 33 minutes

System returned to ROM by power-on

System restarted at 11:18:46 MET Thu Dec 14 2006

System image file is "flash:c1700-ipbase-mz.124-1c.bin"

Cisco 1721 (MPC860P) processor (revision 0x500) with 58329K/7207K bytes of memory.

Processor board ID FOC095049CX (4018852811), with hardware revision 0000

MPC860P processor: part number 5, mask 2

1 Ethernet interface

5 FastEthernet interfaces

1 ATM interface

32K bytes of NVRAM.

32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

sourabhagarwal · ‎12-14-2006

looks to me IOS bug. is it possible for you to upgrade IOS on this router?

check this URL if you have CCO account.

http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCsg48183&cco_product=IOS&fset=&swver=12.4&keyw=ARP&target=1c&train=

enriquebs · ‎12-14-2006

HI,

Suddenly the CPU returned to the normal behaviour (2%)... so maybe someone reloaded the switch facing the router interfaces...

thx everybody!!

cheers

sourabhagarwal · ‎12-14-2006

Good to hear that CPU is back to normal again :-)

during some research I found this bug. There are few others ARP related bugs oc 1700 platforms.

CSCsg48183 Bug Details

Symptoms: A router may unexpectedly send an ARP request from all its active

interfaces to the nexthop of the network of an SNMP server.

Conditions: This symptom is observed on a Cisco router that has the snmp-server host command enabled after any of the

following actions occur:

- You reload the router.

- A switchover of the active RP occurs.

- You enter the redundancy force-switchover main-cpu command.

your IOS is affected by this bug.

please check if this bug can cause problem to you again.

hope to help ... rate if it does ..