hi,
i had a strange issue yesterday where i upgraded an ASA 5555-x active-standby pair to 9.8.4.20 to a patch a recent cisco vulnerability.
both FW got stuck on HA state as 'active' and encountered an 'HA state progression failed'. i did the secondary FW upgrade first which went fine but primary failed and had to do a hard reboot.
i also noticed the new code 9.8.4.20 auto generated the 'no failover wait-disable' command and was wondering if this caused the failure for the FW pair upgrade. i don't see this command documented in cisco website.
i didn't have this issue when i upgraded pair of ASA 5525-X using the same code.
was wondering if anyone had this issue? could this be a bug?
would like to have an advise while waiting for TAC's reply. TIA!
/sec/act# failover reload-standby <<< HANGED/OUTAGE ON BOTH FW; sec/act BECAME sec/stby
------------------ show logging buffered ------------------
Jun 25 2020 14:12:41 FW01 : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_CLIENT_NEGOTIATED_VERSION, my state Active, peer state Failed.
Jun 25 2020 14:12:42 FW01 : %ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=130,my=Active,peer=Active.
Jun 25 2020 14:12:42 FW01 : %ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Active.
Jun 25 2020 14:12:42 FW01 : %ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Active.
Jun 25 2020 14:12:49 FW01 : %ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=100,op=2,my=Cold Standby,peer=Active.
Jun 25 2020 14:12:49 FW01 : %ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_STANDBY_COLD, my state Cold Standby, peer state Active.
Jun 25 2020 14:14:55 FW01 : %ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=100,op=16,my=Cold Standby,peer=Active.
-----
------------------ show failover history ------------------
==========================================================================
From State To State Reason
==========================================================================
14:48:37 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:48:39 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
14:50:42 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
14:50:44 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:50:46 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
14:52:49 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
14:52:50 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:52:52 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
14:54:55 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
14:54:57 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:54:59 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
14:57:02 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
14:57:03 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:57:05 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
14:59:08 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
14:59:10 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
14:59:12 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
15:01:15 UTC Jun 25 2020
Sync Config Negotiation HA state progression failed
15:01:16 UTC Jun 25 2020
Negotiation Cold Standby Detected an Active mate
15:01:18 UTC Jun 25 2020
Cold Standby Sync Config Detected an Active mate
-----
------------------ show failover ------------------
Failover On
Failover unit Secondary
Failover LAN Interface: FAILOVER GigabitEthernet0/7 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 15 of 516 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.8(4)20, Mate 9.8(4)20
Serial Number: Ours FCH191zzz Mate FCH193yyy
Last Failover at: 14:09:35 UTC Jun 25 2020
This host: Secondary - Sync Config
Active time: 192 (sec)
slot 0: ASA5555 hw/sw rev (1.0/9.8(4)20) status (Up Sys)
<SNIP>
Other host: Primary - Active
Active time: 2937 (sec)
slot 0: ASA5555 hw/sw rev (1.0/9.8(4)20) status (Up Sys)
<SNIP>
-----
SECONDARY FW CONSOLE:
/sec/stby#
Unable to sync configuration from Active
.
Detected an Active mate
sec/stby# conf t <<< FROM sec/act BECAME sec/stby
**** WARNING ****
Configuration Replication is NOT performed from Standby unit to Active unit.
Configurations are no longer synchronized.
sec/stby(config)# failover active
This unit is in syncing state. 'failover' command will not be effective at this time
sec/stby(config)# no failover
This unit is in syncing state. 'failover' command will not be effective at this time
-----
PRIMARY FW CONSOLE:
Username: TACACS-LOGIN
Password: *********
Configuration replication is in progress. Please try
authentication again when replication completes.
Username: LOCAL-USER
Password: ***********
Configuration replication is in progress. Please try
authentication again when replication completes.
-----
AFTER PRI FW REBOOT
pri/act# sh run failover
failover
failover lan unit primary
failover lan interface FAILOVER GigabitEthernet0/7
failover key *****
failover replication http
failover interface ip FAILOVER 192.7.1.253 255.255.255.252 standby 192.7.1.254
no failover wait-disable
pri/act# ping 192.7.1.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.27.1.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
pri/act# conf t
pri/act(config)# no failover ?
configure mode commands/options:
group Configure/Enable failover group
health-check Failover unit health check enable/disable
interface Configure the IP address to be used for failover and/or
stateful update information
interface-policy Set the policy for failover due to interface failures
ipsec Configure the use of IPSec tunnel for failover
key Configure the failover shared secret or key
lan Specify the unit as primary or secondary or configure the
interface and vlan to be used for failover communication
link Configure the interface and vlan to be used as a link for
stateful update information
mac Specify the virtual mac address for a physical interface
mac-notification Configure failover MAC address movement notification
settings
polltime Configure failover poll interval
replication Enable HTTP (port 80) connection replication
standby Execute command in standby
timeout Specify the failover reconnect timeout value for
asymmetrically routed sessions
wait-disable Disable switchover waiting for peer state
<cr>
exec mode commands/options:
active Make this system to be the active unit of the failover pair
