PIX Failover - Other firewall reports this firewall failed.

[vcolla]

Every once in a while, I receive an error “Other firewall reports this firewall failed.” on PIX firewall. I am running two PIX 520s in fail over mode. The firewalls are stable and working for months, but once is a while I receive this error at which point none of the firewalls are active. When I access them via the console port, they both say that the other one is primary. After I reboot one of them and perform “failover active” everything is ok.

Any ideas?

May 24 13:01:15 fqppix03 May 24 2001 11:08:22: %PIX-1-103004: (Primary) Other firewall reports this firewall failed.

May 24 13:01:15 fqppix03 May 24 2001 11:08:22: %PIX-1-104002: (Primary) Switching to STNDBY - switch to failed state

May 24 13:01:15 fqppix03 May 24 2001 11:15:00: %PIX-1-105003: (Secondary) Monitoring on interface 0 waiting

May 24 13:01:15 fqppix03 May 24 2001 11:15:00: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting

May 24 13:01:15 fqppix03 May 24 2001 11:08:22: %PIX-1-103004: (Primary) Other firewall reports this firewall failed.

May 24 13:01:15 fqppix03 May 24 2001 11:08:22: %PIX-1-104002: (Primary) Switching to STNDBY - switch to failed state

May 24 13:01:15 fqppix03 May 24 2001 11:15:00: %PIX-1-105003: (Secondary) Monitoring on interface 0 waiting

May 24 13:01:15 fqppix03 May 24 2001 11:15:00: %PIX-1-105003: (Secondary) Monitoring on interface 1 waiting

May 24 13:01:42 fqppix03 May 24 2001 11:16:39: %PIX-1-104002: (Secondary) Switching to STNDBY - ifc check, mate is healthier

May 24 13:01:42 fqppix03 May 24 2001 11:16:39: %PIX-1-104002: (Secondary) Switching to STNDBY - ifc check, mate is healthier

May 24 13:15:38 fqppix03 May 24 2001 11:30:35: %PIX-1-105004: (Secondary) Monitoring on interface 0 normal

May 24 13:15:38 fqppix03 May 24 2001 11:30:35: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal

May 24 13:15:38 fqppix03 May 24 2001 11:30:35: %PIX-1-105004: (Secondary) Monitoring on interface 0 normal

May 24 13:15:38 fqppix03 May 24 2001 11:30:35: %PIX-1-105004: (Secondary) Monitoring on interface 1 normal

May 24 13:17:41 fqppix03 May 24 2001 11:32:38: %PIX-1-104002: (Secondary) Switching to STNDBY - the otherside want me standby

May 24 13:17:41 fqppix03 May 24 2001 11:32:38: %PIX-1-104002: (Secondary) Switching to STNDBY - the otherside want me standby

May 24 13:17:58 fqppix03 May 24 2001 11:26:20: %PIX-1-709003: (Primary) Beginning configuration replication: Send to mate.

May 24 13:17:58 fqppix03 May 24 2001 11:26:20: %PIX-1-709003: (Primary) Beginning configuration replication: Send to mate.

May 24 13:18:06 fqppix03 May 24 2001 11:26:29: %PIX-1-709004: (Primary) End Configuration Replication (ACT)

May 24 13:18:06 fqppix03 May 24 2001 11:26:29: %PIX-1-709004: (Primary) End Configuration Replication (ACT)

May 24 13:18:21 fqppix03 May 24 2001 11:26:44: %PIX-1-105004: (Primary) Monitoring on interface 0 normal

May 24 13:18:21 fqppix03 May 24 2001 11:26:44: %PIX-1-105004: (Primary) Monitoring on interface 1 normal

May 24 13:18:21 fqppix03 May 24 2001 11:26:44: %PIX-1-105004: (Primary) Monitoring on interface 0 normal

May 24 13:18:21 fqppix03 May 24 2001 11:26:44: %PIX-1-105004: (Primary) Monitoring on interface 1 normal

[s.jankowski]

Well, I would say it’s either a bug in the code, mis-configuration, or a physical layer issue (bad cable, switch auto-negotiation, port/nic). What (exact) version of PIX code are you running? Can you post your failover lines and interface lines from your config? If not, you’ll have to talk to Cisco’s TAC.

[vcolla]

Here is some info. As I mentioned before, everything works fine, even failover, but every once in a while it goes crazy.

Cisco Secure PIX Firewall Version 5.1(2)

Compiled on Tue 16-May-00 16:09 by bhochuli

nameif ethernet0 outside security0

nameif ethernet1 inside security100

nameif ethernet2 unused1 security10

nameif ethernet3 unused2 security15

nameif ethernet4 failover1 security20

nameif ethernet5 dmz security1

interface ethernet0 auto

interface ethernet1 auto

interface ethernet2 auto shutdown

interface ethernet3 auto shutdown

interface ethernet4 auto shutdown

interface ethernet5 auto

ip address outside 1.1.1.244 255.255.255.248

ip address inside 10.90.3.5 255.255.255.0

ip address unused1 10.99.3.1 255.255.255.0

ip address unused2 10.99.2.1 255.255.255.0

ip address failover1 10.99.1.1 255.255.255.0

ip address dmz 2.2.2.5 255.255.255.0

failover

failover timeout 0:00:00

failover ip address outside 1.1.1.245

failover ip address inside 10.90.3.6

failover ip address unused1 0.0.0.0

failover ip address unused2 0.0.0.0

failover ip address failover1 0.0.0.0

failover ip address dmz 2.2.2.6

failover link inside

*******************************************************************

sho failover

Failover On

Cable status: Normal

Reconnect timeout 0:00:00

This host: Primary - Active

Active time: 3207750 (sec)

Interface dmz (2.2.2.5): Normal

Interface failover1 (10.99.1.1): Link Down (Waiting)

Interface unused2 (10.99.2.1): Link Down (Waiting)

Interface unused1 (10.99.3.1): Link Down (Waiting)

Interface outside (1.1.1.244): Normal

Interface inside (10.90.3.5): Normal

Other host: Secondary - Standby

Active time: 1149285 (sec)

Interface dmz (2.2.2.6): Normal

Interface failover1 (0.0.0.0): Link Down (Waiting)

Interface unused2 (0.0.0.0): Link Down (Waiting)

Interface unused1 (0.0.0.0): Link Down (Waiting)

Interface outside (1.1.1.245): Normal

Interface inside (10.90.3.6): Normal

The only thing I am thinking is that I have several interfaces "shutdown". Perhaps there is some kind of huge timeout in the code that "screws" things up if they are down for a while. If that's the case, I can

just do cross-over and bring them up...

Thank you,

Vladimir

[yv]

vladimir,

1. did you check for physical identity of both boxes.

Based on cisco docs, they say for both pixes to have identical IOS ver, and to be physically identical in number of ports/cards.

2. also, make sure that unused ports on both pixes are connected via x-over cable.

3. Look at syslog just before it failed to see what is triggering a failover

4. I don't know if you pix is old or not but lately, failover pix is shipped with an extra card (4 port FE). Which you can use on active box for stateful failover.

Hope this helps :)

yury

[s.jankowski]

Try locking down your interface speeds. Auto-detect is never recommended. Also, I realize your unused interfaces are shut down but I would still cross-connect them and bring them up for failover. The shutdown command was added somewhere around this version (for this purpose) and may be a little buggy. Finally, upgrade your code to something more current. All of 5.1 code is still ED (Early Deployment) so keep as current as possible until a General Deployment version is released. 5.1(4) should be a good one to go to but locking down your interface speeds may be all you need here. Good luck.

[thomas.waddell]

since you said "Auto" is never recommended, please

correct me if I'm wrong (I just might be), but I thought the only way to get the PIX interface to run in Full duplex was to select Auto. I'm pretty sure I saw this in Cisco documentation.

Is this not the case?

As for the issue at hand, another possibility may be network saturation. If one of the segments that the fail-over PIX's are connected to gets flooded to the point where the PIX Firewall fails to ARP for itself every 15 seconds (the default), then it will fail-over. You can adjust the frequency of the failover ARPing with the "failover poll" command. If possible check the traffic levels and packet loss on all of the network segments the PIX's are connected to.

Regards,

-Thomas

[wraights]

the pix interface will run in full duplex if you say for it to:

interface ethernet2 100full