Control path to mobility member flaps

[cisconell]

Here is the problem  I have 2 controllers as below one with 6.0 version and other with 5.2 version

AIR-WLC4404-100-K9  6.0.202.0 
AIR-WLC4402-12-K9   5.2.178.0  GUEST

Control path between controller flaps

All mobility anchors on wlan index 7 are down.
All mobility anchors on wlan index 3 are down.
Control path to mobility member x.x.x.x is down.

No problem observed when i have software 5.2.178.0 on both the controler

Did some one experience this problem , or any thoughts on this please

[Stephen Rodriguez]

can you post the show mobility summary and show interface detailed management from the two WLC?  Sometimes the mac address you see under the managmeent interface, is not what gets used in the mobility group.  Though that should cause the mobility tunnel to not come up.

Also, which way are you allowing the communication in the FW?  As Wesley hinted at, there is the possiblity that the keepalive is only allowed to initiate one way.  I like to allow this bi-direcitonally, as you can't tell which WLC will initiate the keepalive by default.  IIRC the keepalive is initiated by the WLC with the lower mac address and this is per WLC pair.

Steve

[Scott Fella]

Like Steve mentioned this doesn't help... What I wanted to check is the Mac address on each of the wlc mobility group info.

Thanks,

Scott Fella

Sent from my iPhone

[cisconell]

Thanks  Scott and Stephen Rodriguez

Here is the whole set up

In the current scenario

3 internal controller
WLC1 (5.2)
WLC2 (5.2)
WLC3 (6.0)

1 guest controller (5.2) WLC GUEST

the control path between

WLC 1 and WLC guest - good and stable
WLC 2 and WLC guest - good and stable
WLC 3 and WLC guest - unstable (get stable for about a while , last time it got stable for 3 hrs and when down for 30 minutes and then came up )

Now with reference to Stephen Rodriguez .

The mac to be entered in the mobility group should be off the management  ? the one which see in show interface detailed management ?

Now whats the mac I see when give the command show sysinfo ?

For

WLC 1 , WLC 2 , Guest I see the mac shown in sysinfo and show interface detailed management is same and thats what in mobility group .

But here is the catch

For wlc 3

The mac seen in show sysinfo and show interface detailed management is different . the one in mobility group is of management.

But if you say in the mobilty group it should be off management mac

how the links are stable between WLC1 and WLC3 and WLC2 and WLC 3 ? could be that because they are on 5.2 and work with both macs in sys info and mgmt

and does the higher version like 6.0 will work only on the mgmt mac ?

and not sure how come some time it get stable for about 3hrs .

Your thoughts , hope now we are in right direction

[Scott Fella]

You should use the Mac address off the wlc mobility group list. The top entry is for that wlc and that should be entered on the other WLC's.

Thanks,

Scott Fella

Sent from my iPhone

[cisconell]

Correction on my last post

For

WLC 1 , WLC 2 , Guest I see the mac shown in sysinfo and show interface detailed management is same and thats what in mobility group .

But here is the catch

For wlc 3

The mac seen in show sysinfo and show interface detailed management is different . the one in mobility group is of

*show sys info*

Checking on Scott last point

[cisconell]

Scott now its confusing .

The first mac which i see in the WLC 3 is used in all other wlc as member.

But this mac is different from the mac which Stephen Rodriguez asked to use "show interface detailed management" output

so which mac i should go for .

Why only wlc 3 shows different mac on 2 commands and all other show same mac ?

Any idea

[Stephen Rodriguez]

when you configre the mobility group, you should always use the mac address that shows in the mobility config for the WLC

So on WLC 1, you go to the mobility config, and whichever mac is listed there, is what you use on WLC 2/3(same thing on WLC 2 and 3).  This way you always have the correct mac address.

Steve

[Scott Fella]

See below.

[weterry]

This would be easier to confirm/deny if we could see the complete "show mobility summary". What was provided earlier was only the half of the output and only from two WLCs.   Your example of aa:bb:cc:dd:ee:ff doesn't allow anyone to help verify.

With that said, if you dont want to provide the real mac addresses and IP's used, then I suppose we have to assume you have it configured the same way everywhere.  Like has been mentioned, nevermind whatever mac address you see listed for the interfaces, whatever WLC1 has in its mobility list for itself is that MAC you use on all controllers. whatever WLC2 has in its mobility list for itself is the MAC you use on all controllers. same for WLC3 and Anchor.

I want to know the real mac addresses because we could learn alot if your WLC3 has a mac address that is lower/higher than the anchor (but the anchor is higher/lower than WLC1/WLC2).  Who starts a keepalive to who is based on mac address.... and if your WLC3 is the only WLC with a problem, then its mac address could give us something to go off of (or confirm what I'm looking for is irrelevent).

By the way, you have multiple 5.2 controllers in the group.  So is your problem ONLY between WLC3 and the anchor?

Or is mobility flapping between WLC3 and WLC1/WLC2?

Also, is your "5.2 to 5.2 works, but 6.0 to 5.2 and 6.0 to 6.0 does not" based on WLC3 being on 6.0 and your 5.2's just reference WLC1 and WLC2?       Because if WLC3 is not both your 5.2 and 6.0 candidate in those failures (upgrade/downgrade) that is misleading the problem description.

[cisconell]

weterry Thanks m  here is the info you are looking for

whatever WLC1 has in its mobility list for itself is that MAC you use on all controllers. whatever WLC2 has in its mobility list for itself is the MAC you use on all

controllers. same for WLC3 and Anchor. -----------Right this is how its configured

is your problem ONLY between WLC3 and the anchor----------Correct 

Or is mobility flapping between WLC3 and WLC1/WLC2? ---no flap , its stable

Also, is your "5.2 to 5.2 works--- YES if all on 5.2 no problem

Also tried all 4 WLC on 6.0  - This had issue with all peers each other .

So its says something to do with the 6.0 version ?

These are the macs shown under mobility

00:23:04:7E:E6:A0 WLC1
00:22:55:90:9F:40 WLC2
00:22:55:90:A4:40 WLC3
00:23:33:B2:BF:80 GUEST

[weterry]

So the mac addresses at least verifies and disproves one my concerns.WLC1/2/3 are all "lower" than the GUEST WLC so mobility keepalives are always flowing in the same direction regarding the GUEST WLC. No concerns there.

Now I'm confused with what you said.

WLC3 on 6.0, has no issues with WLC1/2 (5.2)  but the Mobility Flaps with the GUEST WLC?

But WLC3 on 6.0, has problems with WLC1/2 (6.0)???

Anyway you look at it, if your problem only happens on 6.0, and it happens between everything on 6.0, then there is no reason to even discuss 5.2.  Its clearly something with 6.0 in your environment, but it still doesn't make any sense.

Proper diagnosis of this is still to run "debug mobility keepalive enable" between two controllers with the issue so you can at least characterize who isn't hearing who.....  but a wired packet capture is really the only way to prove the packets are correct.

[cisconell]

Reply to you query

Now I'm confused with what you said.

WLC3 on 6.0, has no issues with WLC1/2 (5.2) but the Mobility Flaps with the GUEST WLC? -- Correct

But WLC3 on 6.0, has problems with WLC1/2 (6.0)???-- Correct .

The debug did not help much.

The strange thing is it even got stable for about 3 hrs and  then got resetted

its not a frequent flap . Some time  stable for 30 minutes and then get reset.

[cisconell]

Do some one know why the mac output of these commands show sysinfo and show interace detials mgmt same for  WLC1 , WLC2 and guest and in case of wlc3 its different.

[weterry]

is LAG enabled on 1-2-guest but not enabled on 3? (or vice versa)?

Or, if 6.0 is the only one on wlc3 right now, it could be something was changed in 6.0 to clear up which addresses are used with or without lag....

If you have 4 wlcs on 6.0, all with LAG enabled, and wlc3 was still different,  I'd likely be concerned....

[weterry]

What did the debugs tell you when this occured?

A debug mobility keepalive shows you the keepalives, which are the mobility up/down.  So if mobility went down during your debug, you'd be able to see which WLC says it didn't hear which WLC (and therefor characterize which direction the teardown is occuring).

As far as 5.2 vs 6.0, makes no sense why you'd be seeing this on everything in 6.0....

The fact that you say you have a problem between all controllers when on 6.0 though, is all the more reason to track down through the mobility debug and a wired packet capture what is really going on with the packets....

"Intermittent" mobility flapping is either going to be someone is sending the packet wrong occaisionally (see it in a wired capture) or the packet is not physically getting from 1 wlc to the other (see it in a wired capture).

Sounds like you have an interesting problem here though, perhaps working through TAC would help make more progress?