Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Control path down - between Intranet <> DMZ

We are about the deploy 2 WLCs 5508 in the DMZ to ensure traffic on mobile devices will be going thru Internet directly.

 

Between the Intranet and the DMZ we have a firewall, we did a lot of tests and the following is occuring :

 

normal pings between Intranet <> DMZ are fine

 

epings between Intranet <> DMZ are fine

 

mpings are not working between Intranet <> DMZ.

 

We checked carefully that the firewall is having IP Protocol 97 and UDP 16666 open but even with that in place mpings are still failing.

 

We opened completely the firewall during a couple of minutes same problem.

 

We checked the logs and we had the impression that IP Protocol 97 was clearly visible on the logs but not UDP.

 

I verified that WLCs in Intranet and / or DMZ are not using ACLs and it is not the case.

 

A debug mobility keep-alive show the same, IP Protocol 97 is ok but not UDP 16666...

 

So my anchor stays at Control path Down on both ends.

 

We don't have also ACLs on the switches in between...

 

We are running out of ideas here and I would be very glad to have more information how to process further on that...

 

Thanks for your help.                   

 

 

Update 14.03.2014 : I finally found the solution, it seems that when the network route is defined to widely this could affect the management interfaces during the tunnel mounting. This is not normal and not properly documented, usually network should serve only service port as the default gateway option is not configurable. Cisco will try to add this in the TAC knowledge bade for helping other people facing the same to spare some time in troubleshooting :)

in my case i simply changed the route to be really specific and the tunnel mounted immediately.

but thanks all for the suggestions and support provided.

Everyone's tags (5)
28 REPLIES
Hall of Fame Super Silver

Control path down - between Intranet <> DMZ

Delete the mobility group from both WLC's and add it back on.  Sometimes that is the issue especially if the FW is wide open.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***
New Member

Control path down - between Intranet <> DMZ

Hi Scott,

Thanks for your fast reply,

We agree that the Default Mobility Domain Name should be different in Intranet WLCs and in DMZ WLCs for security reasons ?

It is just important to create the anchor by specifying the Domain name configured on the destination controller ?

Thanks

Hall of Fame Super Silver

Control path down - between Intranet <> DMZ

The mobility domain name should be different... best practice and you should specify the mobility domain name when creating the anchor.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***
New Member

Control path down - between Intranet <> DMZ

Ok I removed the anchor on both and recreated it, but it is the same :-(

Control path down

I try also to reboot the WLC's in the DMZ but same problem,

Would be something else to check in order to mount up the EoIP tunnel properly?

Thanks

Hall of Fame Super Silver

Control path down - between Intranet <> DMZ

The only way to test if the FW is dropping the traffic is to mount the WLC in the inside for testing and create your mobility anchor.  If this works, then something is dropping UDP 16666/16667.

Post your show mobility summary from both

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***
New Member

Control path down - between Intranet <> DMZ

Ok this was my thought maybe

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor is instead of up / up is Data Path Down,

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

Here the output :

DMZ WLC 1 = 172.21.2.6 :

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... DMZ_SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xe8e1
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
70:81:05:1f:e4:40  10.136.10.36     SG_MOBILITY                       0.0.0.0          Control Path Down
78:da:6e:8a:ee:20  172.21.2.5       DMZ_SG_MOBILITY                   0.0.0.0          Data Path Down
78:da:6e:8b:14:60  172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Up

Intranet WLC 1 = 10.136.10.36  :

(Cisco Controller) >show mobility summary

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xd9f7
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 5
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status

70:81:05:1f:e4:40  10.136.10.36     SG_MOBILITY                       0.0.0.0          Up
78:da:6e:8b:14:60  172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Control Path Down
cc:ef:48:0c:85:80  10.136.10.32     SG_MOBILITY                       0.0.0.0          Up

New Member

Dear Scott,

Dear Scott, I mounted one of the DMZ WLCs into the Intranet, changed the management IP as well. Normal pings were going thru, epings as well but again mpings were not able to pass thru, So we can conclude that the firewall was not in cause, should I upgrade the software ? I have no clues why this WLC is doing that, Someone has an idea to go further on that? This becomes urgent for us and I need to go further :-) Thanks
VIP Purple

Control path down - between Intranet <> DMZ

mping verify the control path between two WLC. It use the UDP 16666 & in your case it is not working & hence control path is down.

eping verify the data path between two WLCs & uses EoIP. Since it is working for you no issue with EoIP.

Pls post the output of "show mobility summary" & "show sysinfo" of both WLCs to see any config issues.

HTH

Rasika

**** Pls rate all useful responses ****

New Member

Control path down - between Intranet <> DMZ

Hello,

show mobility summary is posted above, here the show sysinfo :

Intranet WLC 1 10.136.10.36 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.100.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS

System Name...................................... xxxxx

System Location.................................. xxxx

System Contact................................... xxxxx
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.136.10.36
Last Reset....................................... Software reset
System Up Time................................... 369 days 10 hrs 47 mins 44 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries:AU,BE,CH,CN,DE,FR,GB,HK,IT,J2,MX,NL,RU,SG,TH,TR,US,ZA

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +20 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 5
Number of Active Clients......................... 128

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 70:81:05:1F:E4:40
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500

DMZ WLC 1 172.21.2.6 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.110.0
Bootloader Version............................... 1.0.18
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... xxxx

System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.21.2.6
Last Reset....................................... Software reset
System Up Time................................... 0 days 1 hrs 54 mins 0 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... CH  - Switzerland
Operating Environment............................ Commercial (0 to 40 C)

--More-- or (q)uit
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +45 C
External Temperature............................. +33 C
Fan Status....................................... OK

State of 802.11b Network......................... Disabled
State of 802.11a Network......................... Disabled
Number of WLANs.................................. 0
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 78:DA:6E:8B:14:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 12

Thanks for your support :-)

VIP Purple

Re: Control path down - between Intranet <> DMZ

Thanks for the inputs, configs looks ok to me.

Does all these WLCs having same virtual interface IP ?

Though it may not related, I can see lots of conflicting regulatory domain country codes configured on your DMZ controller ? Does this something you purposely configured ?

On a side note the software version (7.4.100.0) you are running on intranet WLCs are too buggy & recommend you to upgrade them to 7.4.121.0.  Upgrade FUS to 1.9.0.0 as well.

HTH

Rasika

New Member

Control path down - between Intranet <> DMZ

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains :-)

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Thank you

New Member

Re: Control path down - between Intranet <> DMZ

Hi Holzhirt1,

PFB link for FUS upgrade.

Expect a downtime of 30-40 mins.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.pdf

Thanks,

Ashish.

VIP Purple

Re: Control path down - between Intranet <> DMZ

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Is there any difference made by that ? You should have same virtual IP address in order to mobility to work properly. So make sure it is same everywhere.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains :-)

It is not ideal configuring conflicting regulatory domain country codes in single WLC (could impact channels, power levels for certain APs, sometime certain AP radio band won't come up). Best would be having unique controller to serve same regulatory domain country code APs.

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Here is the link to FUS 1.9.0.0 upgrade. Keep note that this will take 30-40min of downtime to your wireless & get a sufficient outage window to do this.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.html

In the below thread I have posted CLI commands for this upgrade with respect to 2504. You can follow that with required image downloads for 5508

https://supportforums.cisco.com/thread/2270290

HTH

Rasika

**** Pls rate all useful responses ****

New Member

Control path down - between Intranet <> DMZ

Now I have same virtual IPs on all but it is the same,

mpings are not going thru....

I will try again to remove mobility groups and re-create it...

thanks for the links Rasika

Hall of Fame Super Silver

Control path down - between Intranet <> DMZ

If that still doesn't work, move the one to the inside and reconfigure the management for testing.  Make sure that it works..

You also have two DMZ SLC's, so put them in the same mobility group to test and see if both the control and data comes up.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***

Re: Control path down - between Intranet <> DMZ

By having a different mobility name saves on mobility exchanges meaning less packets .. Imagine all your controllers have the same name .. Your anchor would get mobility messages from all your anchors which isn't needed.

I would sniff the fw interface and the interface outside the WLC interface .. Do you see the control packet ..

Sent from Cisco Technical Support iPhone App

__________________________________________________________________________________________ "Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin ___________________________________________________________
New Member

Dear George,We did a sniff in

Dear George,

We did a sniff in the beginning and UDP 16666 was not visible I believe, we have seen some malformed packets, but I will give it a try again today and see how it looking...

 

Cisco Employee

Re: Control path down - between Intranet <> DMZ

did you checked the logs on the firewall , why it's dropping the mpings ?

New Member

Control path down - between Intranet <> DMZ

First of all thanks all for your tips and contributions, really appreciated.

Dear Ali odd enough only Ethernet IP 97 are coming into the FW, no UDP 16666 visible.

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor between DMZ WLCs is instead of up / up is Data Path Down,

Epings are not passing thru....

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

As we will have at least 2 Vlans we need to have trunk on the switches, I have no native Vlan configured and pings between DMZ WLCs are working...

Thanks

Cisco Employee

Control path down - between Intranet <> DMZ

one thing to check on the switchport , the portchannel load-balance should be src-dst-ip,

New Member

I changed the management IPs

I changed the management IPs on the DMZ WLCs and reacreated the anchor strangely I was able to mount the tunnel immediately, before it was not passing thru for epings (Data Path Down), now it is up / up. Even if in my opinion it was not passing thru a FW.

However the link with the Intranet is still showing the same issue "Control Path Down".

I will have a look again with my colleague about the Firewall and if we can see at least UDP 16666 packets coming from one end or the other.

If this is not leading to something, I will follow the recommendation to mount it internall and see how the tunnel is reacting...

I keep you posted,

Thanks

New Member

I followed Rasika's advice

I followed Rasika's advice and upgraded both DMZ WLCs to latest FUS 1.9.0.0 as well as 7.4.121.0 firmware.

Problem is still the same...Control path down...

Anyone has an idea how to go further on that?

I'm out of ideas here :(

Thanks

New Member

After more time in this issue

After more time in this issue without results,I opened a case to TAC, the engineer spent more than 2 hours with me in a webex without finding something else, the debugs are under investigations.

Let's cross the fingers we can find out what's wrong here...

New Member

Hi,I really don't understand

Hi,

I really don't understand the problem resolution. Do you mean that, in order to implement Guest-Anchor solution we have to define specific routes to Anchors from the foreign management?

I'm asking because we are experiencing problems with anchor in dmz but we still not have a solution. (TAC doesn't help).

 

Thanks for your support

Francesco

New Member

Dear Sir,Under controller >

Dear Sir,

Under controller > network routes, you can define routes when the WLC's are in DMZ and you are using the service-port to reach them.

If the routes entered are too widely defined like a whole B or A Class this could affect the behavior of the mobility tunnels.

It is better then to precisely define the routes to match only the management stations.

I hope this explanation is clear enough, otherwise feel free to ask me again :)

New Member

Hi,thanks for your answer. In

Hi,

thanks for your answer. In our case we don't use service port to reach DMZ. Maybe this workaround is also needed in Guest tunneled via Management interface?

 

Thanks

 

Francesco

Hall of Fame Super Silver

Guest is always tunneled

Guest is always tunneled using the management ip address... that is defined in the mobility group configuration which has the mac address and the ip of the WLC.  You don't define a static route for mobility to work, you just need the ports opened to allow mobility to come up.

-Scott
*** Please rate helpful posts ***
New Member

Thanks holzhirt1! I had the

Thanks holzhirt1! I had the same issue and removing the broad network route resolved it.

1630
Views
25
Helpful
28
Replies
CreatePlease login to create content