Config HA 5508 Release 7.4.121 Secondary in Maintenace Mode

[Mario Andres Proano Navas]

Hello Everyone

I have a problem, last weekend I performed HA in 5508, using a spare of WLC, in secondary I installed a new license for supporting 50 APs, Main WLC supports 25 APs, however when I enabled SSO in secondary, this one went to Maintenance Mode, pls your help because I performed in secondary "config redundancy unit primary" and "config redundancy unit secondary" as suggested in other forum, but it didn´t work.  Pls follow the config, take in mind that subnet has mask  /23:

CONFIGURATION

WLC1
management                       1    100      169.6.201.235   Static  Yes    No
redundancy-management            1    100      169.6.200.10    Static  No     No
redundancy-port                  -    untagged 169.254.200.10  Static  No     No
service-port                     N/A  N/A      3.3.3.3         Static  No     No
virtual                          N/A  N/A      1.1.1.1         Static  No     No

Commands for WLC1
config interface address redundancy-management 169.6.200.10 peer-redundancy-management 169.6.200.22
config redundancy unit primary
config redundancy mode sso

WLC2
management                       1    100      169.6.200.15    Static  Yes    No
redundancy-management            1    100      169.6.200.22    Static  No     No
redundancy-port                  -    untagged 169.254.200.22  Static  No     No
service-port                     N/A  N/A      3.3.3.3         Static  No     No
virtual                          N/A  N/A      1.1.1.1         Static  No     No

Commands for WLC2
config interface address redundancy-management 169.6.200.22 peer-redundancy-management 169.6.200.10
config redundancy unit secondary
config redundancy mode sso

Then of enable SSO:

WLC1

(Cisco Controller) >show redundancy summary
 Redundancy Mode = SSO ENABLED
     Local State = ACTIVE
      Peer State = UNKNOWN - Communication Down
            Unit = Primary
         Unit ID = 50:57:A8:C7:2B:00
Redundancy State = Non Redundant
    Mobility MAC = 50:57:A8:C7:2B:00

Redundancy Management IP Address................. 169.6.200.10
Peer Redundancy Management IP Address............ 169.6.200.22
Redundancy Port IP Address....................... 169.254.200.10
Peer Redundancy Port IP Address.................. 169.254.200.22

WLC2

(Cisco Controller) >show redundancy summary
 Redundancy Mode = SSO ENABLED
     Local State = MAINTENANCE
      Peer State = UNKNOWN - Communication Down
            Unit = Secondary - HA SKU
         Unit ID = 3C:08:F6:CC:F4:A0
Redundancy State = Non Redundant
    Mobility MAC = 3C:08:F6:CC:F4:A0

Maintenance Mode = Enabled
Maintenance cause= Negotiation Timeout

Redundancy Management IP Address................. 169.6.200.22
Peer Redundancy Management IP Address............ 169.6.200.10
Redundancy Port IP Address....................... 169.254.200.22
Peer Redundancy Port IP Address.................. 169.254.200.10

Additional Notes when rebooted WLC2 after of SSO enable

***********************************************************************************
Starting Redundancy: Starting Peer Search Timer of 120 seconds
Cannot open the file /mnt/application/ha/InheritedApCount.txt
Cannot open the file /mnt/application/ha/InheritedApCount.txt

Found the Peer. Starting Role Determination...

Error:Unable to add Licenses on secondary Controller
Standby started downloading configurations from Active...

***********************************************************************************

Starting Redundancy: Starting Peer Search Timer of 120 seconds
Management Gateway and Peer Redundancy Management interface are not reachable.
Entering maintenance mode...

ok

Standby comparing its own configurations with the configurations downloaded from Active...

[mohanak]

CSCuo96281

Description

Symptom:
5508 WLC pair running 7.4.121 fails over frequently with no crash file present.

Conditions:
5508 WLC pair running in HA SSO mode.

[Scott Fella]

Just for kicks, is the cable between the two RP ports connected and have you tried a new cable.  The directions you followed is correct, but if the secondary doesn't come up, it seems like the connection has failed.

Scott

[Scott Fella]

Here is a link to reasons why the WLC can go into maintenance mode also.  Reboot of the WLC is the way to get out of maintenance mode.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/7-5/High_Availability_DG.html#pgfId-44024

Scott

[Mario Andres Proano Navas]

Scott,

I tried with 3 diferents cables ST and 2 cables crossover with the same result.

Regards MN

[jworth123]

Hi,

I have the same exact problem. If I reboot the standby, will that affect the primary at all? Just wondering if I can do this during business hours. I will probably end up doing it after hours. 

[jordanburnett]

HA sync display Error:Unable to add Licenses on secondary Controller

CSCug85798

There is a bug description with the issue you are experiencing. 

However, the SSO status clearly states that communication is down. Try looking into that first. Per the configuration guide:

"After the WLCs are configured with Redundancy Management and Peer Redundancy Management 
IP Addresses and Redundant Units are configured, it is time to enable SSO. It is important to make 
sure that physical connections are up between both the controllers (that is, both the WLCs are 
connected back to back via the Redundant Port using an Ethernet cable) and the uplink is also 
connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO 
is enabled.

Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as 
per the configuration via Redundant Port. If the WLCs cannot reach each other via Redundant Port 
or via the Redundant Management Interface, the WLC configured as Secondary may go in to 
Maintenance Mode. Maintenance Mode is discussed later in this document."

Make sure that the redundant port connection and cable is fine--and not going through another switch as that is not supported in this version I believe.

However, based on the last message in your document, make sure that the WLCs can ping each others redundancy management interfaces and default gateways BEFORE converting to SSO mode (or currently, if possible). There may be an issue with the upstream distribution switch connection (trunking parameters, LAG, etc.).  

Also, make sure that if you enabled LAG you reboot before attempting to enable SSO. If you find there is a communication issue with the upstream network, you should be able to fix the issue and then reboot the secondary WLC which will take it out of maintenance mode and cause it to renegotiate with the primary (assuming they can now communicate with one another). 

I have seen in the past where the SSO fails because engineers are unable to ping the redundancy management interfaces of the other WLC--which something SSO peers check for on initial configuration and periodically to make sure there isn't a split brain scenario. 

[Mario Andres Proano Navas]

Jordan,

Before enabling SSO, I had ping to redundancy management, peer redundancy management, redundancy port, peer redundancy port and gateway and then after enabling SSO just I have ping to redundancy port and peer redundancy port.

MN

[jordanburnett]

If you're asking what the WLCs do themselves, then that is not exactly correct.

After enabling SSO, the standby WLC will monitor the peer through BOTH the redundancy interface (accessed through the back-to-back redundancy port) AND the redundancy management interface (accessed via the network infrastructure). 

The two separate keepalives are used to ensure there isn't a split brain situation or unnecessary switchover. 

Think, for example, if the redundancy port cable was somehow cut, but both WLCs were still up other than the back-to-back redundancy port connection. 

In this scenario, the standby WLC would check whether or not the active WLC was actually down by using its distribution system ports (connect to the wired infrastructure). 

Per the SSO configuration guide:

"Redundancy Management Interface:

This interface will check the health of the Active WLC via network infrastructure once the Active WLC does not respond to Keepalive messages on the Redundant Port. This provides an additional health check of the network and Active WLC, and confirms if switchover should or should not be executed. Also, the Standby WLC uses this interface in order to source ICMP ping packets to check gateway reachability."

[Mario Andres Proano Navas]

Dear Jordan

This is the problem, I don´t know why I cannot ping by management and redundancy management after enabling SSO

Regards

[jordanburnett]

Oh, gotcha. Well at least you know where to begin troubleshooting. Which WLC can you not ping the management/redundancy management interfaces on? Active or Standby?

If the second switch is still in maintenance mode, try rebooting it and see if it allows you to ping then. 

You didn't happen to enable LAG or anything similar, did you? And are you positive the upstream switchport configuration is correct? 

 

[Saurav Lodh]

This could be bug