controller connectivity fluctuates

[suthomas1]

Hi,

One of our subsidaries have a anchor controller setup. They have main controllers and second controller which anchors to the main controller to tunnel the traffic.

They are seeing, frequent disconnections on the second controller. logs as below are seen.

Controller '192.160.51.5'. An anchor of WLAN 'nolwl' is up.

11:01:50 2013 Data path to mobility member 192.160.101.11 is down.

11:01:50 2013 Data path to mobility member 192.160.101.12 is down.

11:01:50 2013 Data path to mobility member 192.160.51.5 is down.

11:01:18 2013 RX Multicast Queue Full 

11:03:02 2013 Data path to mobility member 192.160.101.12 is up.

everytime down messages come immediately after queue message. controller is not having any configuration for multicast.

appreciate all help.

[Sahil Mengi]

Ok so whats the code you are running on the Main wlc that all traffic is tunelled to?

It may be hitting a known issue/bug.

Thanks

Sahil

[suthomas1]

main controller is running 7.0.240.

thanks.

[Scott Fella]

Are all the WLC's running the same code?  The RX Multicast Queue Full is a bug in older versions, but what I would make sure is that there is no firewall or acl that might be droppign the mobility traffic between the wlc's.  What model wlcs do you have and what code is on the anchors?

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[suthomas1]

main(anchor) controller is 7.0.240.0 - Wireless module on 6500 switch

remote controller ( anchor client ) is 7.3.101

appreciate all help.

[Scott Fella]

Why not just upgrade your anchor wlc to the same code as your WiSM?  Is the anchor WLC a 4400?  Somethimes its good to delete the mobility group information and add it back in.  I have had to do that a few times to get the mobility groups up and stable.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[suthomas1]

anchor controller is a wism, whereas the other remote controller is a 4400.

[Scott Fella]

Well.... I too have some customer with a 4400 as an anchor with no issues.  That is a good code for the 4400's, but maybe look at going with v7.4.110.0 which is MR1 for the WiSM2.  We have seem weird issue on v7.3 and also on early versions of v7.4.  So far many of our v7.4 MR1 upgrades or installs have been pretty smooth.  Just keep that in mind.  If you want to stick with v7.3, then look at v7.3.112.0.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[George Stefanick]

I have a ticket open for my anchors right now .. Doing the same on 7.0.240.0. Althought I think my issue is VPC related.

Do you notice it when there is a lot of traffic on the link.

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[George Stefanick]

Here is the latest from TAC on my issue:

Thank you very much for all the provided information.

From the logs we can see that only the control path goes down, that means UDP port 16666. Data path IP 97 is always up.

The controller sends keepalives, epings (for IP 97) and mpings (UDP 16666) to the different mobility members. By default and this is the way your WLCs are configured, the Mobility Keepalive interval is set to 10 seconds , this is the amount of time between each ping request, then the Mobility Keepalive count, which is the number of times a ping request is sent to a mobility list member before it is considered unreachable,  which is 3.

So, if a mobility member cannot be reached 3 times on 10 seconds periods then it is considered to be down. We can see from the logs that it takes over 30 second for a mobility member ip to get the control path to be reported as up and then down.

We need to confirm if this a problem in the path (firewall, ACL, etc) or a problem with the controller software.

Please get CLI access to a foreign and the anchor as well, make sure that these are controllers that are constantly reported as down and start doing “mping ” between them, if at some point you get no response then we know for sure that there is problem somewhere in the path by a device dropping the packets.

You can also enable and capture “debug mobility keep-alive enable ”, please forward the output from both controllers to me. The IP address must be the address of the mobility peer.

I will be looking forward for your reply.

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[suthomas1]

George,

Did you manage to resolve your mobility problem with TAC?

Appreciate if you can share on what was/has been done.

Thank You.

[Abhishek Abhishek]

Possible cause  Solution/Debug steps 

Not enough power.

Note Intermittent power availability, low AP uptime and AP resets are all indicators of inadequate power availability.

1) Verify that the PoE port on the switch or router is disabled by checking the PoE LED. If the LED is on continuously or fluttering, disable PoE on the port.

Note Cisco mesh access points (1500 Series) are not compliant with 802.3af, which allows an access point to take power from a switch or router. PoE must always be disabled on the switch or router associated with the AP.

2) Check voltage level.

3) Verify that streetlight "bank switching" is not in use where the access point is installed.

4) Check for variations in power during the day and at night.

5) Connect the detachable LED indicator to the access point to verify that power is present (LED on = power)

6) Verify that the Ethernet port LED is active for the port that connects to the access point.

7) If the unit you are troubleshooting is an AP1520, then check the power failure trap.

[sbhadrav@cisco.com]

Configure the controller to detect failed anchor controllers within a mobility group as follows:
Choose Controller > Mobility Management > Mobility Anchor Config to open the Mobility Anchor Config page.
In the Keep Alive Count text box, enter the number of times a ping request is sent to an anchor controller before the anchor is considered to be unreachable. The valid range is 3 to 20, and the default value is 3.
In the Keep Alive Interval text box, enter the amount of time (in seconds) between each ping request that is sent to an anchor controller. The valid range is 1 to 30 seconds, and the default value is 10 seconds.
In the DSCP Value text box, enter the DSCP value. The default is 0.
Note       While configuring the Mobility DSCP value, the mobility control socket (i.e control messages exchanged between mobility peers only and not the data) is also updated. The configured value must reflect in the IPV4 header TOS field. This is a global configuration on the controller that is used to communicate among configured mobility peers only.
Click Apply to commit your changes.
Step 2      Choose WLANs to open the WLANs page.
Step 3      Click the blue drop-down arrow for the desired WLAN or wired guest LAN and choose Mobility Anchors. The Mobility Anchors page appears.
This page lists the controllers that have already been configured as mobility anchors and shows the current state of their data and control paths. Controllers within a mobility group communicate among themselves over a well-known UDP port and exchange data traffic through an Ethernet-over-IP (EoIP) tunnel. They send mpings, which test mobility control packet reachability over the management interface over mobility UDP port 16666 and they send epings, which test the mobility data traffic over the management interface over EoIP port 97. The Control Path text box shows whether mpings have passed (up) or failed (down), and the Data Path text box shows whether epings have passed (up) or failed (down). If the Data or Control Path text box shows “down,” the mobility anchor cannot be reached and is considered failed.

Step 4      Select the IPv4/IPv6 address of the controller to be designated a mobility anchor in the Switch IP Address (Anchor) drop-down list.
Step 5      Click Mobility Anchor Create. The selected controller becomes an anchor for this WLAN or wired guest LAN.
Note       To delete a mobility anchor for a WLAN or wired guest LAN, hover your cursor over the blue drop-down arrow for the anchor and choose Remove.
Step 6      Click Save Configuration.
Step 7      Repeat Step 4 and Step 6 to set any other controllers as mobility anchors for this WLAN or wired guest LAN.
Step 8      Configure the same set of mobility anchors on every controller in the mobility group.