Scenario: Replaced 2 Pix 515e firewalls with 2 ASA 5510 firewalls. Turned old firewalls off. Turned one of the fully-configured new firewalls on. All traffic to and from the IP address configured on the outside interface of the ASA flowed properly. Also, one of the IP address in the middle of our routed block was also able to send/receive behind the firewall. When I did a packet capture on the outside interface of the new ASA, I only saw traffic destined for these two IP addresses. All other traffic was not flowing properly into the outside interface of the ASA properly. The outside interface of the ASA is connected to a port on a Cisco switch provided by the ISP for our fiber run. We restarted that switch, but nothing changed. The capture still only showed traffic destined for 2 of our many public IP addresses. We connected a laptop to the switch and verified that we could in fact browse the internet using the IP addreses that we weren't seeing any traffic coming to while capturing on the ASA. I turned on full debugging, and there were no errors being logged, and no traffic at all was coming into the outside inteface of the ASA for the majority of our public IP addresses (which have static NAT translations configured on the ASA). We also called the ISP who verified the duplex and speed settings on the switch ports. Both sides were operating at 100 Mbps full-duplex. They hadn't made a change to that switch in years.
This was actually the second time we experienced this phenomenon. Both times after many hours of waiting, everything returns to normal. The first time it occurred, we had simply failed over from one Pix firewall to the other. Something we had done numerous times in the past without any trouble. At that time, it was only the IP address configured on the outside interface of the Pix that could traffic. Failing back to the primary firewall did not resolve the problem. We tried everything we could think of, including powering down all of the network switches (WAN and LAN), and then turning them back on. After a few hours of frustration, and at least 30 minutes of not touching anything, everything started working again like magic.
We can't have this problem looming over us, as we do occassionally need to failover to our standby firewall. Any help would be greatly appreciated!
>> After a few hours of frustration, and at least 30 minutes of not touching anything, everything started working again like magic.
if the ISP switch performs only L2 the ARP table is placed on another device that is not under your control, this would explain why power cycle of ISP switch didn't solve the issue.
The ARP table of the ISP L3 device contains ARP entries for all the public IP addresses of your block but the MAC address associated with all entries was likely that of the old device (effect of NAT) that you have replaced with the ASA. The ASA has a different MAC address.
So until the remote device does not ARP again for all the IP addresses you are stucked and probably only the ASA IP address can be reached because it can have sent a gratuituos ARP for its IP address.
the test with the PC directly connected to the ISP switch confirms this.
Hope to help