Losing connection after power glitch - keep having to clear arp on PIX

[baskervi]

We have a remote facility with a PIX-525 that lost power yesterday. This site is very simple - just a single subnet on the inside firewall with two switches, but there is a router with one interface on this network, and this router does NATing for various hosts behind the router. The PIX does not know about the subnet behind the router. The problem is that every few minutes, we lose connectivity to some hosts. I'll verify the mac-address is correct on the PIX and shows up on the correct port of the switches. I've disabled proxy arping on the PIX.

When we lose connectivity, all I have to do is to do a clear arp on the PIX, and everything is back up and running. Until I do a clear arp, I cannot ping these hosts from the PIX. As soon as the clear arp is performed, I can immediately ping these hosts and they can be accessed remotely.  Even local hosts cannot ping the hosts that lose connectivity.

Since the arp entries are good on the firewall and the mac-address-table looks good on the switches, I'm left scratching my head. Can someone help me figure out what I'm missing? Thanks

[baskervi]

Some additional information. We don't lose connectivity inside the PIX. The hosts between the switches are all stable, and it appears that the only problem is from the PIX to the inside hosts. These hosts can be on either switch. When the host is down with respect to the PIX, I can't ping them from the PIX, but I do a clear arp and they immediately come back up. The arp address does not change. The mac-address-table on the switches do not change. I have tried to clear the mac-address-table, but this doesn't fix the problem. Only the clear arp fixes the problem.

One other thing... We occasionally lose connection for managing the switches, but we can still get to hosts connected to the switches. This indicates to me there is a L3 problem.

Any ideas? Thanks

[Peter Koltl]

The router probably performs proxy ARP as it is needed for NAT. That is, it might well respond for ARP requests of PIX. You can check the PIX ARP table before clearing.

I suggest that you remove the router and place the NAT'ed segment to a 3rd interface of the PIX. Having two routing devices on a user segment is not a healthy choice.

[baskervi]

Regarding the router, it's not our device - we have another company that wanted to control its own security, so there isn't a way to remote the router.

That having been said, we just found the problem. I unfortunately failed to mention we had two PIXes configured as failover. I'm troubleshooting remotely with another individual who was also troubleshooting remotely, so it would take a moment to get connected to something for additional testing - by then, I think things would straighten out.

The problem was caused by the other PIX - it would be the secondary and appear healthy when looking at the show failover, but then it would apparently go haywire and start arping for the primary PIX. I just didn't look at the failover when it was broken. I'm not sure if it's the PIX or switch that's malfuctioning. We'd look at the arp tables of the PIX and servers, and everything looked OK, but just an hour ago, when I looked at the arp entry of a server, I noticed the gateway entry had the incorrect mac address. The mac-address-table showed it was the inside interface of the secondary PIX. I disabled the switch interfaces for the secondary PIX, and everything has been stable since.

Thanks for responding.