I'd be very interested to hear if anyone has experienced this problem before
Nov 7 10:39:58 10.4.50.249 Nov 07 2011 10:39:48: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470862340
Nov 7 10:40:08 10.4.50.249 Nov 07 2011 10:39:58: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470872340
Nov 7 10:40:24 10.4.50.249 Nov 07 2011 10:40:15: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470889340
Nov 7 10:40:28 10.4.50.249 Nov 07 2011 10:40:19: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470893340
Nov 7 10:40:54 10.4.50.249 Nov 07 2011 10:40:44: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470918340
Nov 7 10:41:23 10.4.50.249 Nov 07 2011 10:41:14: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS
Nov 7 10:41:29 10.4.50.249 Nov 07 2011 10:41:20: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470954340
Nov 7 10:41:56 10.4.50.249 Nov 07 2011 10:41:47: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS
Nov 7 10:42:12 10.4.50.249 Nov 07 2011 10:42:03: %ASA-7-711001: arp-req: request for 10.110.34.24 still pending
Nov 7 10:42:23 10.4.50.249 Nov 07 2011 10:42:14: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS
Nov 7 10:42:40 10.4.50.249 Nov 07 2011 10:42:31: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471025340
Nov 7 10:43:10 10.4.50.249 Nov 07 2011 10:43:01: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471055340
Nov 7 10:43:20 10.4.50.249 Nov 07 2011 10:43:11: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471065340
Nov 7 10:43:56 10.4.50.249 Nov 07 2011 10:43:47: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471101340
Has anyone seen anything like this before ?
It sounds like you've already done a lot of good troubleshooting. I would do 2 additional captures:
1. Setup an ARP capture on the OPS interface and see if the ASA is actually passing the ARP request down to the NIC:
capture arp ethernet-type arp int OPS
show capture arp detail
2. Setup an ASP drop capture to see if the ASA is dropping the ARP requests for any reason:
capture drop type asp-drop all
show capture drop
Does the cable between the OPS interface connect directly to the switchport that you mirrored? Or is there any other device in between like an inline IPS or load balancer?
Thanks for your suggestion, I've not used the capture command before.
Interesting. The ASA captures show the ARP requests being put on the wire, but yet they never show up in the switch port mirror capture.
Do you see any input drops on the switch port? How about any output drops on the ASA interface? If this is a Cisco switch I would suggest opening a TAC case to have a switching engineer take a look at this. The captures on the ASA are a reliable indicator of whether or not the packet was sent onto the network (as long as we don't see it dropped in the ASP drop capture).
I cleared the counters on the switchport 4 days ago. I see the 219 input errors since then.
GigabitEthernet4/10 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 0012.d9d2.d319 (bia 0012.d9d2.d319)
Description: Uplink to euedi-fw-gw2-Gi1/2
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 9w4d, output never, output hang never
Last clearing of "show interface" counters 4d00h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 452000 bits/sec, 213 packets/sec
5 minute output rate 163000 bits/sec, 57 packets/sec
493820809 packets input, 620784478408 bytes, 0 no buffer
Received 0 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
219 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
89842242 packets output, 12485193314 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
And on the firewall interface (I'm not sure if I cleared counters on this)
Interface GigabitEthernet1/2 "OPS", is up, line protocol is up
Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Media-type configured as RJ45 connector
Description: Interface to UK Operations network
MAC address d0d0.fd52.92ad, MTU 1500
IP address 10.110.34.254, subnet mask 255.255.255.0
89873929 packets input, 12497278308 bytes, 0 no buffer
Received 54703 broadcasts, 0 runts, 0 giants
955 input errors, 0 CRC, 0 frame, 955 overrun, 0 ignored, 0 abort
0 L2 decode drops
493908286 packets output, 620732665395 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
0 rate limit drops
input queue (blocks free curr/low): hardware (0/0)
output queue (blocks free curr/low): hardware (0/0)
Traffic Statistics for "OPS":
89873139 packets input, 10865741315 bytes
494057566 packets output, 611909242495 bytes
210105 packets dropped
1 minute input rate 51 pkts/sec, 19696 bytes/sec
1 minute output rate 202 pkts/sec, 44789 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 65 pkts/sec, 20037 bytes/sec
5 minute output rate 224 pkts/sec, 57000 bytes/sec
5 minute drop rate, 0 pkts/sec
I agree with you about opening the TAC case and our Cisco partner is actually in the process of doing so...I'm supplying them with the requisite configuration information at the moment.
Di you aver get an answer on this?
I have a similar issue with an ASA running v8.0(3).
The ISP can't see the public address of the ASA unless they add a specific route on the upstream router.
The ISP moved us to another block of public addresses (however they were routing the old and new block to us) and since we moved to the new block fully with all our NATs none of the new address block (except) the outside interface were reachable. They had to add specific route which work but they should have to!!!
The ASA is clustered and has not been failed over or reloaded as yet.
Your problem sounds a bit different than the original posters. (Didnt read all of the posted content) In your case it seems that the ASA is not responding to ARP from the ISP.
Have you made sure that the Proxy ARP is NOT disabled on the "outside" inteface of the ASA?
This would cause the ASA to do so that it would only reply to ARP requests when they were directly related to the IP address of its "outside" interface but no additional NATed public IP address would visible in the ARP of the upstream router as the Proxy ARP is disabled.
The default setting is that the Proxy ARP is enabled.
If its disabled then you would see a command
sysopt noproxyarp outside
in your configurations. Confirmable with the command "show run sysopt" or "show run all sysopt"
I went through a falrly long TAC case with Cisco and eventually their development team suggested an upgrade to version 8.2.5 as they suspected that the underlying block changes affiliated with CSCta02170 might be triggering the situation.
Unfortunately, that particular firewall upgrade was postponed indefinitely so I don't know if it would actually have resolved the issue.
However, as Jouni has said your problem does sound a bit different to mine...