strange 4 hour timeout issue - replacing Pix 525 with ASA's

waltrich · ‎08-01-2008

Pls refer to the attached diag. We have attempted to do this 3 times now to no avail and I'm hoping someone has seen similar. We put the ASA's into prod and remove the Pixes. All our our external websites and vpn tunnels are accessible after clearing arp on the switches (clear arp-cache). However, we have a monitoring server that does checks against our publicly addressed websites using wget, and it fails every time after the upgrade. In fact, we are unable to access any of our external websites from servers housed within the data center behind the external switches, and not even do a wget to the websites from a Solaris box that is connected directly to the external switches. We can however hit our external websites from anywhere other than machines housed behind our external switches/fw's. And to complicate matters, we are able to access any external websites from the same machines (the ones behind our external switches) such as sun.com, cisco.com other than the websites we host. It's only an issue hitting websites that we host, therefore killing our monitoring capabilities.

We can restore the monitoring server's functionality by failing back to the original Pixes. However, around 3 - 4 hours later, the same exact problem occurs, only to clear itself up after another 4 hours. Then the monitoring server is able to monitor our external websites without the issue manifesting itself. I have heard that the arp timeout on the switches is 4 hours and am trying to make a tie in to this but can't figure out how to do so. Please remember that 3 - 4 hours after putting the pixes back into service, the issue clears up only to remanifest itself, this time with the pixes, but will then clear up on its own around 3 - 4 hours later

Thanks for any assistance!

milan.kulik · ‎08-03-2008

Hi,

4 hours timer is typical for ARP cache problems.

When you replace some router (or PIX in your case) with another device using the same IP addresses on interfaces, you may get into troubles with other machines having the old MAC addresses in ARP caches.

Windows usually update their ARP cache very fast, but UNIX machines and routers might require manual cache cleaning.

4 hours is ARP cache timer used on most systems by default.

So I'd recommend cleaning all ARP caches after the PIX-ASA swap (your monitoring server, routers/L3 switches, even web hosts) in all VLANs directly connected to your ASAs.

If some trouble persists, try to traceroute and see what's the last node replying.

Then check the ARP cache on that machine and on the next-hop machine.

HTH,

Milan

waltrich · ‎08-13-2008

WE replaced the pixes with the new ASA's. We cleared the arp cache on all of the gear under our control. At first the same symptoms happened but they cleared up 8 hours later. That would be 2 arp cycles if that was the issue. It may have been on the ISP's side but only if they are doing something strange with the traffic from our monitoring server that works like a hairpin for lack of a better description.

Thanks for your assistance.