strange failover message on PIX 8.0.4

[whattaneguiconsul]

Hello,

since a few days ago I'm receiving strange failover messages on my syslog server.

The topology is :

2 pixes 525  running version 8.0.4, routed mode, single context

They're are connected using the typical serial failover cable and an ethernet interface for state failover ( through single-switch).

Well, the message received is :

PIX1		2010-10-07	12:09:11		%PIX-1-105009: (Primary) Testing on interface users Passed	local7	alert
PIX1		2010-10-07	12:09:11		%PIX-1-105008: (Primary) Testing Interface users	local7	alert
PIX1		2010-10-07	12:09:11		%PIX-1-105005: (Primary) Lost Failover communications with mate on interface users

But when you look at the show failover, it seems OK.

PIX525# sh failover    
Failover On
Cable status: Normal
Failover unit Primary
Failover LAN Interface: N/A - Serial-based failover enabled
Unit Poll frequency 5 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 6 of 250 maximum
failover replication http
Version: Ours 8.0(4), Mate 8.0(4)
Last Failover at: 12:54:58 CEDT Jun 26 2010
        This host: Primary - Active
                Active time: 8899905 (sec)
                  Interface outside (ip): Normal
                  Interface uno (ip): Normal
                  Interface otro (ip): Normal (Not-Monitored)
                  Interface users (ip): Normal
                  Interface tres (ip): Normal
                  Interface Voice (ip): Normal
                  Interface Voice2 (ip): Normal (Not-Monitored)
                  Interface ADSL (ip): Normal (Not-Monitored)
                  Interface partner (ip): Normal (Not-Monitored)
                  Interface intf4 (0.0.0.0): Link Down (Waiting)
        Other host: Secondary - Standby Ready
                Active time: 0 (sec)
                  Interface outside (ip): Normal
                  Interface uno (ip): Normal
                  Interface otro (ip): Normal (Not-Monitored)
                  Interface users (ip): Normal
                  Interface tres (ip): Normal
                  Interface Voice (ip): Normal
                  Interface Voice2 (ip): Normal (Not-Monitored)
                  Interface ADSL (ip): Normal (Not-Monitored)
                  Interface partnerr (ip): Normal (Not-Monitored)
                  Interface intf4 (0.0.0.0): Link Down (Waiting)

Stateful Failover Logical Update Statistics
        Link : failover Ethernet2 (up)
        Stateful Obj    xmit       xerr       rcv        rerr     
        General         1407547377 0          1186667    0        
        sys cmd         1186667    0          1186667    0        
        up time         0          0          0          0        
        RPC services    0          0          0          0        
        TCP conn        594965233  0          0          0        
        UDP conn        728168378  0          0          0        
        ARP tbl         27684218   0          0          0        
        Xlate_Timeout   0          0          0          0        
        VPN IKE upd     942        0          0          0        
        VPN IPSEC upd   19274      0          0          0        
        VPN CTCP upd    0          0          0          0        
        VPN SDI upd     0          0          0          0        
        VPN DHCP upd    0          0          0          0        
        SIP Session     55522665   0          0          0        

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       17      2966693
        Xmit Q:         0       286     1458155018

Today is 7/october/2010 ( last failover was in june )

PIX525# sh failover history
==========================================================================
From State                 To State                   Reason
==========================================================================
12:54:55 CEDT Jun 26 2010
Not Detected               Negotiation                No Error

12:54:58 CEDT Jun 26 2010
Negotiation                Just Active                No Active unit found

12:54:58 CEDT Jun 26 2010
Just Active                Active Drain               No Active unit found

12:54:58 CEDT Jun 26 2010
Active Drain               Active Applying Config     No Active unit found

12:54:58 CEDT Jun 26 2010
Active Applying Config     Active Config Applied      No Active unit found

12:54:58 CEDT Jun 26 2010
Active Config Applied      Active                     No Active unit found

==========================================================================

And the interface users ( because syslog message ) :

there are some input errors + overruns :-( but they're growing low ( 2 input errors / 5 minutes )

Interface Ethernet3 "users", is up, line protocol is up
  Hardware is i82559, BW 100 Mbps, DLY 100 usec
        Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
        MAC address 000d.88ff.c731, MTU 1500
        IP address x.x.x.x, subnet mask 255.255.0.0
        12406166512 packets input, 4636800403321 bytes, 0 no buffer
        Received 39731160 broadcasts, 0 runts, 0 giants
        65000 input errors, 0 CRC, 0 frame, 65000 overrun, 0 ignored, 0 abort
        0 L2 decode drops
        13193150792 packets output, 9812135046036 bytes, 266801 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 babbles, 0 late collisions, 0 deferred
        0 lost carrier, 0 no carrier
        input queue (curr/max packets): hardware (0/1) software (21/195)
        output queue (curr/max packets): hardware (20/128) software (0/1105)
  Traffic Statistics for "Usuaris":
        12401605053 packets input, 4441626843654 bytes
        13195317098 packets output, 9613938432561 bytes
        142169291 packets dropped
      1 minute input rate 11246 pkts/sec,  5688809 bytes/sec
      1 minute output rate 3989 pkts/sec,  2271670 bytes/sec
      1 minute drop rate, 14 pkts/sec
      5 minute input rate 12175 pkts/sec,  5443086 bytes/sec
      5 minute output rate 6135 pkts/sec,  5241796 bytes/sec
      5 minute drop rate, 16 pkts/sec
  Control Point Interface States:
        Interface number is 5
        Interface config status is active
        Interface state is active

Do you think these input errors are the cause ?

Maybe a traffic burst  ?

Interface is 100 Mbps but today's max througput is 60 Mbps.

Thank you

Regards.

[praprama · Accepted solution]

Hi,

Based on the syslog, it does look like the primary PIX lost 3 consecutive hello packets on the interface "users". Yes it could be related to the overruns/input errors on the interface which generally come up due to large amounts of traffic received on that interface in a bursty manner.

Again, the reason for this could be a mismatched speed/duplex setting on the directly connected device on the "users" interface. Hope this helps!!

Thanks and Regards,

Prapanch

[whattaneguiconsul]

Hello Prapanch,

Thanks for your answer.

There's no duplex mismatch, both sides ( PIX and switch ) are configured in 100 Full, then a bursty traffic is the most probable explanation.

I think that my hello packet polling is very aggressive, 5 seconds.

I see in http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080094ea7.shtml that the default polling interval is 15 seconds.

Are you agree ?

Unit Poll frequency 5 seconds, holdtime 15 seconds
Interface Poll frequency 5  seconds, holdtime 25 seconds

PIX525# sh run failover
failover
failover lan unit primary
failover polltime unit 5 holdtime 15
failover key *****
failover replication http
failover link failover Ethernet2
failover interface ip failover 192.168.35.1 255.255.255.252 standby 192.168.35.2

Both interfaces, in pix and switch are in access mode, portfast.

It seems that in the switch, there are only input errors, but in the pix there are also overruns.

I think that bursty traffic confirms that.

switch#sh int fa0/12
FastEthernet0/12 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 0019.30a0.fd0c (bia 0019.30a0.fd0c)
  Description: PIX INTERFACE USERS
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 173/255, rxload 94/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters 3d02h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 37177000 bits/sec, 8075 packets/sec
  5 minute output rate 67927000 bits/sec, 11132 packets/sec
     788717621 packets input, 508529752535 bytes, 0 no buffer
     Received 265282 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
    2293 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1051577182 packets output, 471075236472 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

PIX side :

5 minut input rate = 2462598 bytes/sec -> 25 Mbps ( approximately )

5 minut output rate = 4433133 bytes/sec -> 44 Mbps ( approximately )

Interface Ethernet3 "users", is up, line protocol is up
  Hardware is i82559, BW 100 Mbps, DLY 100 usec
        Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
        MAC address 000d.88ff.c731, MTU 1500
        IP address x.x.x.x, subnet mask 255.255.0.0
        12755545570 packets input, 4796324154847 bytes, 0 no buffer
        Received 40282759 broadcasts, 0 runts, 0 giants
        67727 input errors, 0 CRC, 0 frame, 67727 overrun, 0 ignored, 0 abort
        0 L2 decode drops
        13408480858 packets output, 9938045524101 bytes, 267867 underruns
        0 output errors, 0 collisions, 0 interface resets
        0 babbles, 0 late collisions, 0 deferred
        0 lost carrier, 0 no carrier
        input queue (curr/max packets): hardware (0/1) software (0/195)
        output queue (curr/max packets): hardware (0/128) software (0/1105)
  Traffic Statistics for "Usuaris":
        12751061782 packets input, 4595954622893 bytes
        13410659826 packets output, 9736655785499 bytes
        142643083 packets dropped
      1 minute input rate 6386 pkts/sec,  2221782 bytes/sec
      1 minute output rate 5291 pkts/sec,  3857845 bytes/sec
      1 minute drop rate, 11 pkts/sec
      5 minute input rate 7403 pkts/sec,  2462598 bytes/sec
      5 minute output rate 6521 pkts/sec,  4433133 bytes/sec
      5 minute drop rate, 12 pkts/sec

Thanks again.

Regards

[praprama]

Hi,

By default the interface polling hello times is 5 seconds and hold time is 5 times that (25 seconds). Please look at the below link:

http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/ef.html#wp1928586

In my opinion, this is not an aggressive value and should not cause problems. But you can always alter the vlaues if needed.

Thanks and Regards,

Prapanch