Hi Guys,
This morning I upgraded a failover-set of ASA-5520's from 8.2(1.11) to 8.2(2), by loading the new software onto flash on both devices, setting the boot parameter and reloading the standby-unit to 8.2(2).
The secondary unit boots 8.2(2), and I can verify by 'show failover' that it goes into 'standby ready'. So far, so good.
But when I tried to switch the active unit to the one running 8.2(2) by issuing 'no failover active' on the primary unit, something weird happens;
%ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Standby Ready,peer=Standby Ready.
%ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Standby Ready.
%ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Standby Ready, peer state Standby Ready.
%ASA-1-104001: (Secondary) Switching to ACTIVE - Other unit wants me Active. Primary unit switch reason: Set by the config command. .
%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=200,op=10,my=Just Active,peer=Standby Ready.
%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_fast.
%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_fast.
%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_FAST, my state Just Active, peer state Standby Ready.
%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=201,op=10,my=Active Drain,peer=Standby Ready.
%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_drain.
%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_drain.
%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_DRAIN, my state Active Drain, peer state Standby Ready.
%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=202,op=10,my=Active Applying Config,peer=Standby Ready.
%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_pre_config.
%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_pre_config.
%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_PRECONFIG, my state Active Applying Config, peer state Standby Ready.
%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=203,op=10,my=Active Config Applied,peer=Standby Ready.
%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_post_config.
%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_post_config.
%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_POSTCONFIG, my state Active Config Applied, peer state Standby Ready.
%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=204,op=10,my=Active,peer=Standby Ready.
%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active.
%ASA-6-720039: (VPN-Secondary) VPN failover client is transitioning to active state
%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active.
%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE, my state Active, peer state Standby Ready.
%ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=405,op=130,my=Active,peer=Standby Ready.
%ASA-6-720027: (VPN-Secondary) HA status callback: My state Active.
%ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_MY_STATE, my state Active, peer state Standby Ready.
%ASA-6-721003: (WebVPN-Primary) HA progression change: event HA_PROG_STANDBY_READY, my state Standby Ready, peer state Active.
%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=405,op=80,my=Standby Ready,peer=Active.
%ASA-6-720027: (VPN-Primary) HA status callback: My state Standby Ready.
%ASA-1-105003: (Primary) Monitoring on interface outside waiting
%ASA-1-105003: (Primary) Monitoring on interface inside waiting
%ASA-1-105003: (Primary) Monitoring on interface DMZ_Management waiting
%ASA-1-105003: (Primary) Monitoring on interface ISA_Outside waiting
%ASA-1-105003: (Primary) Monitoring on interface DMZ waiting
%ASA-1-103001: (Primary) No response from other firewall (reason code = 1). (aprox 10 seconds after the 'no failover active' is issued)
- The secondary unit (8.2.2) becomes active, but almost immediatly reloads (either due to crash or because the primary unit makes it?), making the primary unit go from standby again. This doesn't happen before i make the 8.2(2) unit active.
- Vieving the 'show failover history' on the primary unit (8.2(1.11)) telles me 'HELLO not heard from mate', indicating that the failover communication stops when the 8.2(2) unit goes active, perhaps because the secondary unit crashes when becoming active.
I've noticed some weird failover counters, when running 8.2(1.11) and 8.2(2):
Output from primary (8.2(1.11)) :
Stateful Failover Logical Update Statistics
Link : statefull GigabitEthernet0/3.702 (up)
Stateful Obj xmit xerr rcv rerr
General 4948752 0 7681 11
sys cmd 1269 0 1269 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 3625569 0 4289 5
UDP conn 1319661 0 2007 6
ARP tbl 1362 0 8 0
Xlate_Timeout 0 0 0 0
VPN IKE upd 317 0 0 0
VPN IPSEC upd 316 0 108 0
VPN CTCP upd 258 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Output from Secondary (8.2(2)) :
KBN-ASA01/pri/act# fail exec mate show fail | beg Stateful Failover Logical Up$
Stateful Failover Logical Update Statistics
Link : statefull GigabitEthernet0/3.702 (up)
Stateful Obj xmit xerr rcv rerr
General 8 0 14604 2434
sys cmd 8 0 8 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 9902 1894
UDP conn 0 0 4476 540
ARP tbl 0 0 68 0
Xlate_Timeout 0 0 0 0
VPN IKE upd 0 0 52 0
VPN IPSEC upd 0 0 52 0
VPN CTCP upd 0 0 46 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
In the above, I'm seing way more rerr's on the secondary unit that on the primary, and they keep climbing pretty fast. When both units run 8.2(1.11), I see little or no rerr's at all.
Anyone got a clue on what I'm looking at here, or if I'm missing something? - The releasenotes for 8.2 sais nothing about known open failover bugs in 8.2(2), but maybe I've found one? I've done noumerous upgrades this way before without problems.
Other info on my setup:
KBN-ASA01/pri/act# sh ver
Cisco Adaptive Security Appliance Software Version 8.2(1)11
Device Manager Version 6.2(5)
Compiled on Mon 21-Sep-09 17:47 by builders
System image file is "disk0:/asa821-11-k8.bin"
Config file at boot was "startup-config"
KBN-ASA01 up 30 days 1 hour
failover cluster up 30 days 1 hour
Hardware: ASA5520, 512 MB RAM, CPU Pentium 4 Celeron 2000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB
Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
Boot microcode : CN1000-MC-BOOT-2.00
SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
IPSec microcode : CNlite-MC-IPSECm-MAIN-2.04
0: Ext: GigabitEthernet0/0 : address is 001f.9e8b.d060, irq 9
1: Ext: GigabitEthernet0/1 : address is 001f.9e8b.d061, irq 9
2: Ext: GigabitEthernet0/2 : address is 001f.9e8b.d062, irq 9
3: Ext: GigabitEthernet0/3 : address is 001f.9e8b.d063, irq 9
4: Ext: Management0/0 : address is 001f.9e8b.d05f, irq 11
5: Int: Internal-Data0/0 : address is 0000.0001.0002, irq 11
6: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 5
Licensed features for this platform:
Maximum Physical Interfaces : Unlimited
Maximum VLANs : 150
Inside Hosts : Unlimited
Failover : Active/Active
VPN-DES : Enabled
VPN-3DES-AES : Enabled
Security Contexts : 2
GTP/GPRS : Disabled
SSL VPN Peers : 2
Total VPN Peers : 750
Shared License : Disabled
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials : Enabled
Advanced Endpoint Assessment : Disabled
UC Phone Proxy Sessions : 24
Total UC Proxy Sessions : 24
Botnet Traffic Filter : Disabled
This platform has an ASA 5520 VPN Plus license.
show failover (from before 8.2.2):
KBN-ASA01/pri/act# sh fail
Failover On
Failover unit Primary
Failover LAN Interface: LANBasedFailover GigabitEthernet0/3.701 (up)
Unit Poll frequency 1 seconds, holdtime 3 seconds
Interface Poll frequency 3 seconds, holdtime 15 seconds
Interface Policy 1
Monitored Interfaces 6 of 160 maximum
failover replication http
Version: Ours 8.2(1)11, Mate 8.2(1)11
Last Failover at: 10:30:07 dk Feb 15 2010
This host: Primary - Active
Active time: 2438414 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.2(1)11) status (Up Sys)
Interface outside (XXXXXXX): Normal
Interface inside (XXXXXXXXX): Normal
Interface DMZ_Management (XXXXXXX): Normal
Interface ISA_Outside (XXXXXXX): Normal
Interface GuestNet (XXXXXXX): Normal (Not-Monitored)
Interface DMZ (XXXXXXX): Normal
slot 1: ASA-SSM-20 hw/sw rev (1.0/5.1(4)S257.0) status (Up/Up)
IPS, 5.1(4)S257.0, Up
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5520 hw/sw rev (2.0/8.2(1)11) status (Up Sys)
Interface outside (XXXXXXXX): Normal
Interface inside (XXXXXXXX): Normal
Interface DMZ_Management (XXXXXXXX): Normal
Interface ISA_Outside (XXXXXXX): Normal
Interface GuestNet (XXXXXXXX): Normal (Not-Monitored)
Interface DMZ (XXXXXXXXX): Normal
slot 1: ASA-SSM-20 hw/sw rev (1.0/5.0(2)S152.0) status (Up/Up)
IPS, 5.0(2)S152.0, Up
Stateful Failover Logical Update Statistics
Link : statefull GigabitEthernet0/3.702 (up)
Stateful Obj xmit xerr rcv rerr
General 5101608 0 7723 11
sys cmd 1311 0 1311 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 3729295 0 4289 5
UDP conn 1368683 0 2007 6
ARP tbl 1420 0 8 0
Xlate_Timeout 0 0 0 0
VPN IKE upd 320 0 0 0
VPN IPSEC upd 319 0 108 0
VPN CTCP upd 260 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 14 19044
Xmit Q: 0 999 5452309
regards,
Lasse
