Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover problem

Hi Guys,

This morning I upgraded a failover-set of ASA-5520's from 8.2(1.11) to 8.2(2), by loading the new software onto flash on both devices, setting the boot parameter and reloading the standby-unit to 8.2(2).

The secondary unit boots 8.2(2), and I can verify by 'show failover' that it goes into 'standby ready'. So far, so good.

But when I tried to switch the active unit to the one running 8.2(2) by issuing 'no failover active' on the primary unit, something weird happens;

%ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Standby Ready,peer=Standby Ready.

%ASA-6-720028: (VPN-Secondary) HA status callback: Peer state Standby Ready.

%ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_PEER_STATE, my state Standby Ready, peer state Standby Ready.

%ASA-1-104001: (Secondary) Switching to ACTIVE - Other unit wants me Active. Primary unit switch reason: Set by the config command. .

%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=200,op=10,my=Just Active,peer=Standby Ready.

%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_fast.

%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_fast.

%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_FAST, my state Just Active, peer state Standby Ready.

%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=201,op=10,my=Active Drain,peer=Standby Ready.

%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_drain.

%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_drain.

%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_DRAIN, my state Active Drain, peer state Standby Ready.

%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=202,op=10,my=Active Applying Config,peer=Standby Ready.

%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_pre_config.

%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_pre_config.

%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_PRECONFIG, my state Active Applying Config, peer state Standby Ready.

%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=203,op=10,my=Active Config Applied,peer=Standby Ready.

%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active_post_config.

%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active_post_config.

%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE_POSTCONFIG, my state Active Config Applied, peer state Standby Ready.

%ASA-6-720037: (VPN-Secondary) HA progression callback: id=3,seq=200,grp=0,event=204,op=10,my=Active,peer=Standby Ready.

%ASA-7-720048: (VPN-Secondary) FSM action trace begin: state=, last event=, func=vpnfo_fsm_active.

%ASA-6-720039: (VPN-Secondary) VPN failover client is transitioning to active state

%ASA-7-720049: (VPN-Secondary) FSM action trace end: state=, last event=, return=0, func=vpnfo_fsm_active.

%ASA-6-721003: (WebVPN-Secondary) HA progression change: event HA_PROG_ACTIVE, my state Active, peer state Standby Ready.

%ASA-6-720032: (VPN-Secondary) HA status callback: id=3,seq=200,grp=0,event=405,op=130,my=Active,peer=Standby Ready.

%ASA-6-720027: (VPN-Secondary) HA status callback: My state Active.

%ASA-6-721002: (WebVPN-Secondary) HA status change: event HA_STATUS_MY_STATE, my state Active, peer state Standby Ready.

%ASA-6-721003: (WebVPN-Primary) HA progression change: event HA_PROG_STANDBY_READY, my state Standby Ready, peer state Active.

%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=405,op=80,my=Standby Ready,peer=Active.

%ASA-6-720027: (VPN-Primary) HA status callback: My state Standby Ready.

%ASA-1-105003: (Primary) Monitoring on interface outside waiting

%ASA-1-105003: (Primary) Monitoring on interface inside waiting

%ASA-1-105003: (Primary) Monitoring on interface DMZ_Management waiting

%ASA-1-105003: (Primary) Monitoring on interface ISA_Outside waiting

%ASA-1-105003: (Primary) Monitoring on interface DMZ waiting

%ASA-1-103001: (Primary) No response from other firewall (reason code = 1). (aprox 10 seconds after the 'no failover active' is issued)

- The secondary unit (8.2.2) becomes active, but almost immediatly reloads (either due to crash or because the primary unit makes it?), making the primary unit go from standby again. This doesn't happen before i make the 8.2(2) unit active.

- Vieving the 'show failover history' on the primary unit (8.2(1.11)) telles me 'HELLO not heard from mate', indicating that the failover communication stops when the 8.2(2) unit goes active, perhaps because the secondary unit crashes when becoming active.

I've noticed some weird failover counters, when running 8.2(1.11) and 8.2(2):

Output from primary (8.2(1.11)) :

Stateful Failover Logical Update Statistics

        Link : statefull GigabitEthernet0/3.702 (up)

        Stateful Obj    xmit       xerr       rcv        rerr

        General         4948752    0          7681       11

        sys cmd         1269       0          1269       0

        up time         0          0          0          0

        RPC services    0          0          0          0

        TCP conn        3625569    0          4289       5

        UDP conn        1319661    0          2007       6

        ARP tbl         1362       0          8          0

        Xlate_Timeout   0          0          0          0

        VPN IKE upd     317        0          0          0

        VPN IPSEC upd   316        0          108        0

        VPN CTCP upd    258        0          0          0

        VPN SDI upd     0          0          0          0

        VPN DHCP upd    0          0          0          0

        SIP Session     0          0          0          0

Output from Secondary (8.2(2)) :

KBN-ASA01/pri/act# fail exec mate show fail | beg Stateful Failover Logical Up$

Stateful Failover Logical Update Statistics

        Link : statefull GigabitEthernet0/3.702 (up)

        Stateful Obj    xmit       xerr       rcv        rerr

        General         8          0          14604     2434

        sys cmd         8          0          8          0

        up time         0          0          0          0

        RPC services    0          0          0          0

        TCP conn        0          0          9902       1894

        UDP conn        0          0          4476       540

        ARP tbl         0          0          68         0

        Xlate_Timeout   0          0          0          0

        VPN IKE upd     0          0          52         0

        VPN IPSEC upd   0          0          52         0

        VPN CTCP upd    0          0          46         0

        VPN SDI upd     0          0          0          0

        VPN DHCP upd    0          0          0          0

        SIP Session     0          0          0          0

In the above, I'm seing way more rerr's on the secondary unit that on the primary, and they keep climbing pretty fast. When both units run 8.2(1.11), I see little or no rerr's at all.
Anyone got a clue on what I'm looking at here, or if I'm missing something? - The releasenotes for 8.2 sais nothing about known open failover bugs in 8.2(2), but maybe I've found one? I've done noumerous upgrades this way before without problems.
Other info on my setup:
KBN-ASA01/pri/act# sh ver
Cisco Adaptive Security Appliance Software Version 8.2(1)11
Device Manager Version 6.2(5)
Compiled on Mon 21-Sep-09 17:47 by builders
System image file is "disk0:/asa821-11-k8.bin"
Config file at boot was "startup-config"
KBN-ASA01 up 30 days 1 hour
failover cluster up 30 days 1 hour
Hardware:   ASA5520, 512 MB RAM, CPU Pentium 4 Celeron 2000 MHz
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB
Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
                             Boot microcode   : CN1000-MC-BOOT-2.00
                             SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
                             IPSec microcode  : CNlite-MC-IPSECm-MAIN-2.04
0: Ext: GigabitEthernet0/0  : address is 001f.9e8b.d060, irq 9
1: Ext: GigabitEthernet0/1  : address is 001f.9e8b.d061, irq 9
2: Ext: GigabitEthernet0/2  : address is 001f.9e8b.d062, irq 9
3: Ext: GigabitEthernet0/3  : address is 001f.9e8b.d063, irq 9
4: Ext: Management0/0       : address is 001f.9e8b.d05f, irq 11
5: Int: Internal-Data0/0    : address is 0000.0001.0002, irq 11
6: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 5
Licensed features for this platform:
Maximum Physical Interfaces    : Unlimited
Maximum VLANs                  : 150
Inside Hosts                   : Unlimited
Failover                       : Active/Active
VPN-DES                        : Enabled
VPN-3DES-AES                   : Enabled
Security Contexts              : 2
GTP/GPRS                       : Disabled
SSL VPN Peers                  : 2
Total VPN Peers                : 750
Shared License                 : Disabled
AnyConnect for Mobile          : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials          : Enabled
Advanced Endpoint Assessment   : Disabled
UC Phone Proxy Sessions        : 24
Total UC Proxy Sessions        : 24
Botnet Traffic Filter          : Disabled
This platform has an ASA 5520 VPN Plus license.
show failover (from before 8.2.2):
KBN-ASA01/pri/act# sh fail
Failover On
Failover unit Primary
Failover LAN Interface: LANBasedFailover GigabitEthernet0/3.701 (up)
Unit Poll frequency 1 seconds, holdtime 3 seconds
Interface Poll frequency 3 seconds, holdtime 15 seconds
Interface Policy 1
Monitored Interfaces 6 of 160 maximum
failover replication http
Version: Ours 8.2(1)11, Mate 8.2(1)11
Last Failover at: 10:30:07 dk Feb 15 2010
        This host: Primary - Active
                Active time: 2438414 (sec)
                slot 0: ASA5520 hw/sw rev (2.0/8.2(1)11) status (Up Sys)
                  Interface outside (XXXXXXX): Normal
                  Interface inside (XXXXXXXXX): Normal
                  Interface DMZ_Management (XXXXXXX): Normal
                  Interface ISA_Outside (XXXXXXX): Normal
                  Interface GuestNet (XXXXXXX): Normal (Not-Monitored)
                  Interface DMZ (XXXXXXX): Normal
                slot 1: ASA-SSM-20 hw/sw rev (1.0/5.1(4)S257.0) status (Up/Up)
                  IPS, 5.1(4)S257.0, Up
        Other host: Secondary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5520 hw/sw rev (2.0/8.2(1)11) status (Up Sys)
                  Interface outside (XXXXXXXX): Normal
                  Interface inside (XXXXXXXX): Normal
                  Interface DMZ_Management (XXXXXXXX): Normal
                  Interface ISA_Outside (XXXXXXX): Normal
                  Interface GuestNet (XXXXXXXX): Normal (Not-Monitored)
                  Interface DMZ (XXXXXXXXX): Normal
                slot 1: ASA-SSM-20 hw/sw rev (1.0/5.0(2)S152.0) status (Up/Up)
                  IPS, 5.0(2)S152.0, Up
Stateful Failover Logical Update Statistics
        Link : statefull GigabitEthernet0/3.702 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         5101608    0          7723       11
        sys cmd         1311       0          1311       0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        3729295    0          4289       5
        UDP conn        1368683    0          2007       6
        ARP tbl         1420       0          8          0
        Xlate_Timeout   0          0          0          0
        VPN IKE upd     320        0          0          0
        VPN IPSEC upd   319        0          108        0
        VPN CTCP upd    260        0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       14      19044
        Xmit Q:         0       999     5452309
regards,
Lasse

Everyone's tags (3)
1 ACCEPTED SOLUTION

Accepted Solutions

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Hello,

DUde I think that i Have found the issue.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtb27147&from=summary

Bug:CSCtb27147

Try to disable SNMP and try again.

Let me know

8 REPLIES

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Could you send me the current show failover as well the sho failover history.

Can you also ping or check connectivity between the devices through the failover link.

New Member

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Hi,

I've attatched output from the the requested cmd's.

I can confirm that I have connectivity between the 2 ASA's on the failover (and failover-state) interfaces via ping (both ways).

Thanks for your time.

Regards

Lasse

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

This issue might be happening due of a bug but I would like to check the Show

Crash info from both units. I will try to find a bug related, for the meantime get that info.

New Member

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Thanks - I've attatched the crashinfo from the secondary unit.

Regards,

--

Lasse

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Hello,

DUde I think that i Have found the issue.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtb27147&from=summary

Bug:CSCtb27147

Try to disable SNMP and try again.

Let me know

New Member

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

I think you're right on!

After issuing the 'no snmp-server enable', I was able to  do a failover, without the 8.2(2) unit crashing on me. aprox 10 seconds after I tried to re-enable SNMP, the ASA crashed (probably when some of the allowed servers SNMP polled the ASA.

So I'm thinking that it's very likely that the bug you're mentioning is what i'm seing - Good spotted!

What do you recommend in terms of an upgrade path? (Kind of need 8.2(2) due to the new no-licensing of UC-mobility)

Regards

--

Lasse

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Send me the related SNMP configuration on your ASA. But I'm sure that the

issue is with SNMP

New Member

Re: ASA: Upgrading from 8.2(1.11) to 8.2(2) presents failover pr

Below is the SNMP-config:

----------------------------------------

KBN-ASA01/pri/act# sh run | inc snmp

snmp-server host inside 10.100.10.112 community *****

snmp-server host inside 10.100.10.119 community *****

snmp-server host inside 10.100.10.99 community *****

snmp-server host inside 10.100.2.140 community *****

snmp-server host inside 10.100.254.1 community *****

snmp-server host inside 10.100.254.3 poll community ***** version 2c

snmp-server host inside 10.100.3.14 community *****

snmp-server host inside 10.100.3.160 community *****

snmp-server host inside 10.100.3.44 trap community ***** version 2c

snmp-server host inside 10.100.3.50 trap community ***** version 2c

snmp-server location KBN-ASA01

snmp-server contact IT-Support

snmp-server community *****

snmp-server enable traps snmp authentication linkup linkdown coldstart
snmp-server enable traps syslog
snmp-server enable traps ipsec start stop
snmp-server enable traps entity config-change fru-insert fru-remove
snmp-server enable traps remote-access session-threshold-exceeded
I'll do some tests, with various parts of the SNMP-cfg disabled. Thanks for taking the time so far dude.
regards,
Lasse

3836
Views
6
Helpful
8
Replies
CreatePlease to create content