Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Step-by-Step Configuration and Troubleshooting Best Practices for the NGFW, NGIPS and AMP Technologies A Visual Guide to the Cisco Firepower Threat Defense (FTD)
New Member

Arp requests not getting through ASA 5510, v8.2(1)

Hi There,

I'd be very interested to hear if anyone has experienced this problem before

Problem Description

  • devices on a network attached to the ASA 5510,v8.2(1) seem to "disappear" off the network. i.e. they are not pingable from outside that network although they are accessable locally from other devices on the same network.
  • looking at the arp cache on the firewall, there is no dynamic arp entry for these devices once they disappear and they cannot be ping'd from the firewall. Static arp is not configured on the firewall. Proxy arp is set on all interfaces as per default configuration.
  • device type ranges from temperature sensors to sms gateways to SAN controllers and ILO cards. The common factor is that these devices do not initiate outgoing connections, they only receive incoming connections such as management logons etc.
  • other servers on the same network that initiate outgoing connection never seem to "disappear" and they always have dynamic arp entries on the firewall.
  • rebooting a "disappeared" device, and in some cases pinging the gateway interface on the firewall brings the device back online and a dynamic arp entry appears in the firewall's arp cache.

Troubleshooting

  • debug arp shows the following output while trying to ping the disappeared device (10.110.34.24) from a remote network.

Nov  7 10:39:58 10.4.50.249 Nov 07 2011 10:39:48: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470862340

Nov  7 10:40:08 10.4.50.249 Nov 07 2011 10:39:58: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470872340

Nov  7 10:40:24 10.4.50.249 Nov 07 2011 10:40:15: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470889340

Nov  7 10:40:28 10.4.50.249 Nov 07 2011 10:40:19: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470893340

Nov  7 10:40:54 10.4.50.249 Nov 07 2011 10:40:44: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470918340

Nov  7 10:41:23 10.4.50.249 Nov 07 2011 10:41:14: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS

Nov  7 10:41:29 10.4.50.249 Nov 07 2011 10:41:20: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14470954340

Nov  7 10:41:56 10.4.50.249 Nov 07 2011 10:41:47: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS

Nov  7 10:42:12 10.4.50.249 Nov 07 2011 10:42:03: %ASA-7-711001: arp-req: request for 10.110.34.24 still  pending

Nov  7 10:42:23 10.4.50.249 Nov 07 2011 10:42:14: %ASA-7-711001: arp-req: generating request for 10.110.34.24 at interface OPS

Nov  7 10:42:40 10.4.50.249 Nov 07 2011 10:42:31: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471025340

Nov  7 10:43:10 10.4.50.249 Nov 07 2011 10:43:01: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471055340

Nov  7 10:43:20 10.4.50.249 Nov 07 2011 10:43:11: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471065340

Nov  7 10:43:56 10.4.50.249 Nov 07 2011 10:43:47: %ASA-7-711001: arp-send: arp request built from 10.110.34.254 d0d0.fd52.92ad for 10.110.34.24 at 14471101340

  • I mirrored the switchport that the firewall OPS interface is plugged into (OPS interface = 10.110.34.254) and no arp traffic reaches the switch from the firewall.
  • It looks like arp requests are not getting out from the firewall.
  • turning off proxy arp does not fix the problem and when it is off I see no output at all from debug arp (filtering for 10.110.34.24)

Has anyone seen anything like this before ?

Thanks,

Marie

7 REPLIES
Cisco Employee

Arp requests not getting through ASA 5510, v8.2(1)

Hi Marie,

It sounds like you've already done a lot of good troubleshooting. I would do 2 additional captures:

1. Setup an ARP capture on the OPS interface and see if the ASA is actually passing the ARP request down to the NIC:

capture arp ethernet-type arp int OPS

show capture arp detail

2. Setup an ASP drop capture to see if the ASA is dropping the ARP requests for any reason:

capture drop type asp-drop all

show capture drop

Does the cable between the OPS interface connect directly to the switchport that you mirrored? Or is there any other device in between like an inline IPS or load balancer?

-Mike

New Member

Re: Arp requests not getting through ASA 5510, v8.2(1)

Hi Mike,

Thanks for your suggestion, I've not used the capture command before.

Capture Setup

  • two "disappeared" devices were selected (IBM tape library = 10.110.34.24 and Foxbox sms gateway = 10.110.34.22)
  • neither device had dynamic arp entries in the firewall's arp cache, neither were ping'able from remote networks
  • a constant ping was set up from Inside/10.4.1.57 to OPS/10.110.34.24 or OPS/10.110.34.22
  • the switchport into which the OPS interface (10.110.34.254) is connected was mirrored and a wireshark trace started
  • the captures were started

Capture Results

  • I attach a file (capture_arp_ASA_5520.txt) which shows the results from the arp and drop captures.
  • it looks like the arp requests are getting to wherever the capture is recording (the OPS nic ?) as seen by the "arp who-has 10.110.34.22 tell 10.110.34.254" and "arp who-has 10.110.34.24 tell 10.110.34.254"
  • I couldn't find any dropped arp requests in the drop capture
  • I also attach two wireshark screenshots showing the arp traffic arriving at the switchport (arp_OPS_port_mirror.png) and at the Foxbox system (arp_Foxbox_mirror.png). Foxbox is 10.110.34.22.
  • I can also confirm that the OPS interface is plugged directly into the switchport with no other devices in between. The cable is brand new and all connections are well seated.

Conclusions

  • Arp traffic for the disappeared devices gets to the OPS interface on the firewall but does not seem to get to either the switch or the disappeared device

Rgds,

Marie

Cisco Employee

Re: Arp requests not getting through ASA 5510, v8.2(1)

Hi Marie,

Interesting. The ASA captures show the ARP requests being put on the wire, but yet they never show up in the switch port mirror capture.

Do you see any input drops on the switch port? How about any output drops on the ASA interface? If this is a Cisco switch I would suggest opening a TAC case to have a switching engineer take a look at this. The captures on the ASA are a reliable indicator of whether or not the packet was sent onto the network (as long as we don't see it dropped in the ASP drop capture).

-Mike

New Member

Re: Arp requests not getting through ASA 5510, v8.2(1)

Hi Maik,

I cleared the counters on the switchport 4 days ago. I see the 219 input errors since then.

GigabitEthernet4/10 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet Port, address is 0012.d9d2.d319 (bia 0012.d9d2.d319)

  Description: Uplink to euedi-fw-gw2-Gi1/2

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000-TX

  input flow-control is off, output flow-control is off

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 9w4d, output never, output hang never

  Last clearing of "show interface" counters 4d00h

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 452000 bits/sec, 213 packets/sec

  5 minute output rate 163000 bits/sec, 57 packets/sec

     493820809 packets input, 620784478408 bytes, 0 no buffer

     Received 0 broadcasts (0 multicast)

     0 runts, 0 giants, 0 throttles

     219 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     89842242 packets output, 12485193314 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

And on the firewall interface (I'm not sure if I cleared counters on this)

Interface GigabitEthernet1/2 "OPS", is up, line protocol is up

  Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec

    Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

    Media-type configured as RJ45 connector

    Description: Interface to UK Operations network

    MAC address d0d0.fd52.92ad, MTU 1500

    IP address 10.110.34.254, subnet mask 255.255.255.0

    89873929 packets input, 12497278308 bytes, 0 no buffer

    Received 54703 broadcasts, 0 runts, 0 giants

    955 input errors, 0 CRC, 0 frame, 955 overrun, 0 ignored, 0 abort

    0 L2 decode drops

    493908286 packets output, 620732665395 bytes, 0 underruns

    0 output errors, 0 collisions, 0 interface resets

    0 late collisions, 0 deferred

    0 input reset drops, 0 output reset drops

    0 rate limit drops

    input queue (blocks free curr/low): hardware (0/0)

    output queue (blocks free curr/low): hardware (0/0)

  Traffic Statistics for "OPS":

    89873139 packets input, 10865741315 bytes

    494057566 packets output, 611909242495 bytes

    210105 packets dropped

      1 minute input rate 51 pkts/sec,  19696 bytes/sec

      1 minute output rate 202 pkts/sec,  44789 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 65 pkts/sec,  20037 bytes/sec

      5 minute output rate 224 pkts/sec,  57000 bytes/sec

      5 minute drop rate, 0 pkts/sec

I agree with you about opening the TAC case and our Cisco partner is actually in the process of doing so...I'm supplying them with the requisite configuration information at the moment.

Rgds.,

Marie

New Member

Arp requests not getting through ASA 5510, v8.2(1)

Hi Marie,

Di you aver get an answer on this?

I have a similar issue with an ASA running v8.0(3).

The ISP can't see the public address of the ASA unless they add a specific route on the upstream router.

The ISP moved us to another block of public addresses (however they were routing the old and new block to us) and since we moved to the new block fully with all our NATs none of the new address block (except) the outside interface were reachable. They had to add specific route which work but they should have to!!!

The ASA is clustered and has not been failed over or reloaded as yet.

Thanks

Fergal

Super Bronze

Arp requests not getting through ASA 5510, v8.2(1)

Hi Fergal,

Your problem sounds a bit different than the original posters. (Didnt read all of the posted content) In your case it seems that the ASA is not responding to ARP from the ISP.

Have you made sure that the Proxy ARP is NOT disabled on the "outside" inteface of the ASA?

This would cause the ASA to do so that it would only reply to ARP requests when they were directly related to the IP address of its "outside" interface but no additional NATed public IP address would visible in the ARP of the upstream router as the Proxy ARP is disabled.

The default setting is that the Proxy ARP is enabled.

If its disabled then you would see a command

sysopt noproxyarp outside

in your configurations. Confirmable with the command "show run sysopt" or "show run all sysopt"

- Jouni

New Member

Arp requests not getting through ASA 5510, v8.2(1)

Hi Fergal,

I went through a falrly long TAC case with Cisco and eventually their development team suggested an upgrade to version 8.2.5 as they suspected that the underlying block changes affiliated with CSCta02170 might be triggering the situation.

Unfortunately, that particular firewall upgrade was postponed indefinitely so I don't know if it would actually have resolved the issue.

However, as Jouni has said your problem does sound a bit different to mine...

Rgds.,

Marie

3439
Views
0
Helpful
7
Replies
CreatePlease to create content