WLC 2112 - APs disassociating

Unanswered Question
Mar 22nd, 2012

I've found some discussion on this, but nothing that's helped resolve so I'm hoping someone can help. We recently purchased a 2112 WLC, running version 6.0.199.4. I have everything running through a single port on the controller, which is connected to a trunk port on our 3750 stack. Our management VLAN is vlan 40, and AP is plugged into another port on the same switch which is an access port in vlan 40. APs appear in the controller ok, receive an IP address, but we aren't able to connect to any of the WLANs and periodically the APs will disassociate with the following messages:

0Thu Mar 22 09:47:51 2012AP 'AP003a.99eb.702d', MAC: b8:62:1f:45:95:b0 disassociated previously due to AP Reset. Uptime: 0 days, 00 h 35 m 52 s . Last reset reason: operator changed 11g mode
1Thu Mar 22 09:47:39 2012AP Disassociated. Base Radio MAC:b8:62:1f:45:95:b0
2Thu Mar 22 09:47:39 2012AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:b8:62:1f:45:95:b0 Cause=New Discovery

I'm not sure if that's related to why we can't connect to the WLANs or not...

Another wrinkle is that if we plug in the AP directly to one of the PoE ports on the controller, it all works perfectly. Any ideas? I'm guessing its something switch-related since the only difference is that we're not going through the switch when it works.

Thanks in advance,

Matt

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 5 (1 ratings)
Scott Fella Thu, 03/22/2012 - 06:54

I would make sure you have enough power available on the switches and also check the logs on the switch.

Thanks,

Scott Fella

Sent from my iPhone

matt.millington Thu, 03/22/2012 - 07:36

Should be plenty of power. The AP is the only PoE device powered in to the switch, and its a 3750G PoE switch. Let me ask this: Could it be an issue that the AP connects at 1Gbps and the WLC at 100Mbps?

I don't see any errors or issues on the switch side. Could it be network slowness? I don't see any of that, but just trying to figure out what could be causing it. They are plugged into the same switch. Its part of a stack, but there are only 3 devices plugged in to this particular switch.

Scott Fella Thu, 03/22/2012 - 07:39

That shouldn't make a difference.  You might want to check to verify that the AP is drawing 15.4 and that it is negotiating at 1000 full.  Do you see anythning on the switch logs?  Is the ap's on the same switch or different?  Have you tried switching the patch cable and moving the ap to a different port?

matt.millington Thu, 03/22/2012 - 07:56

I have tried a different port, same result. I have also swapped out network cables, same result.

It is drawing 15.4W on both of the ports I tried, and always negotiates at 1000 full. AP and WLC are on the same 3750 in the stack.

I also see this in the WLC log in the web interface:

AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:b8:62:1f:45:95:b0 Cause=Heartbeat Timeout

Anything in particular that I should be looking at in the switch logs?

kmiller1634 Thu, 03/22/2012 - 07:59

Typically a heartbeat timeout means it's losing the connection to the controller (ie: capwap tunnel goes down); but that doesn't make any sense if they are on the same switch..

How long does the AP stay on-line before rebooting or is it random?

Scott Fella Thu, 03/22/2012 - 08:00

Weird that the WLC and AP are in the same switch stack and you are getting these errors.  Just look at the logs to see if the port shut down, etc.  Do the AP's in the same switch as the WLC fail also or just ap's that are on a different switch on the same stack?

matt.millington Thu, 03/22/2012 - 08:08

Both APs in the same switch and in different switches do the same thing. They don't reboot, just disassociate. Here is the status on one of them:

UP time: 2 hours+

Controller Associated Time: 6 minutes

Controller Association Latency: 14 seconds


They associate as soon as they come up, but then disassociate and reassociate. They have this message:

AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:b8:62:1f:45:95:b0 Cause=New Discovery

in addition to sometimes showing a Heartbeat Timeout cause for the disassociation. In addition, sometimes it shows the cause being Max Retransmissions. I thought it was just the AP, but it happens with the second AP as well. Here are my switch port configs:

For AP:

interface GigabitEthernet7/0/14

description Test AP

switchport access vlan 40

switchport mode access

spanning-tree portfast

For WLC:

interface GigabitEthernet7/0/2

description WLC Management Port

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 40,55,82,94

switchport mode trunk

spanning-tree portfast trunk

It is detecting Rogue APs (these are the existing APs that we want to replace with this system). Could that be causing issues? I don't think so since it works just fine when plugged directly in to the WLC.

Could it be WLC software version?

matt.millington Thu, 03/22/2012 - 08:12

In addition, I get this error:

AP 'AP003a.99eb.7167', MAC: b8:62:1f:45:bf:e0 disassociated previously due to AP Reset. Uptime: 0 days, 02 h 17 m 33 s . Last reset reason: operator changed 11g mode

Not sure what that means that the operator changed 11g mode...

Scott Fella Thu, 03/22/2012 - 08:56

Can you post the output from the console on one of the AP's?  If this is a new install? you don't have any AP's on this right now correct?  YOu should upgrad that WLC to 7.0.230.0.... musch beteer code if you ask me.

Stephen Rodriguez Thu, 03/22/2012 - 08:59

Take a look at the counters on a port, and check if you are seeing large amounts of broadcast traffic.  The other thing to check is spanning-tree.  make sure that VLAN 40 is not reconverging often.

Steve

matt.millington Thu, 03/22/2012 - 09:00

We are in the process of getting our contract approved on cisco.com so we can have access to download the new code. Hopefully, that will do it. Here is what I get at the console of the AP:

*Mar 22 15:57:48.038: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Mar 22 15:57:48.038: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Mar 22 15:58:01.060: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Mar 22 15:58:01.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.83.0.67 peer_port: 5246

*Mar 22 15:58:04.000: %CAPWAP-5-CHANGED: CAPWAP changed state to

*Mar 22 15:58:41.000: DTLS_CLIENT_ERROR: ../dtls/dtls_connection_db.c:2015 Max retransmission count reached!

*Mar 22 15:58:41.000: %DTLS-3-HANDSHAKE_RETRANSMIT: Max retransmit count for 10.83.0.67 is reached.

*Mar 22 15:59:00.999: %DTLS-5-SEND_ALERT: Send WARNING : Close notify Alert to 10.83.0.67:5246

*Mar 22 15:59:04.037: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Mar 22 15:59:04.037: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Mar 22 15:59:23.058: %CAPWAP-3-ERRORLOG: Go join a capwap controller

matt.millington Thu, 03/22/2012 - 09:04

Also these:

*Mar 22 16:02:06.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.83.0.67 peer_port: 5246

*Mar 22 16:02:08.001: %CAPWAP-5-CHANGED: CAPWAP changed state to

*Mar 22 16:02:11.018: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.83.0.67 peer_port: 5246

*Mar 22 16:02:11.019: %CAPWAP-5-SENDJOIN: sending Join Request to 10.83.0.67

*Mar 22 16:02:11.019: %CAPWAP-5-CHANGED: CAPWAP changed state to JOIN

*Mar 22 16:02:11.100: %CAPWAP-5-CHANGED: CAPWAP changed state to CFG

*Mar 22 16:02:11.384: %CAPWAP-5-CHANGED: CAPWAP changed state to UP

*Mar 22 16:02:11.406: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down

*Mar 22 16:02:12.406: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down

*Mar 22 16:02:14.385: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller Cisco_Annex1

*Mar 22 16:02:14.424: %LWAPP-3-CLIENTEVENTLOG: SSID vcannex added to the slot[0]

*Mar 22 16:02:14.426: %LWAPP-3-CLIENTEVENTLOG: SSID vlan55 added to the slot[0]

*Mar 22 16:02:14.427: %LWAPP-3-CLIENTEVENTLOG: SSID vcannex added to the slot[1]

*Mar 22 16:02:14.429: %LWAPP-3-CLIENTEVENTLOG: SSID vlan55 added to the slot[1]

*Mar 22 16:02:14.439: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

Scott Fella Thu, 03/22/2012 - 09:04

Okay... is your management and ap mangers on the same vlan?  YOur AP's are on vlan 40, what about your management and ap-manager?

matt.millington Thu, 03/22/2012 - 09:07

Yes, they are all in VLAN 40, 10.83.0.64/26 is the network.

10.83.0.65 - gateway

10.83.0.66 - management

10.83.0.67 - ap-manager

.68-126 - DHCP pool for APs

Scott Fella Thu, 03/22/2012 - 09:12

What model AP's by chance... if they are the 3500's, you need to run 7.0

Scott Fella Thu, 03/22/2012 - 09:18

Okay... Well look at upgrading to 7.0.230.0. Any who... You certain you don't have another device using the same address as the ap-manager? If you disconnect the wlc from the network, can you make sure the ip address you used are not pingable?

Thanks,

Scott Fella

Sent from my iPhone

matt.millington Thu, 03/22/2012 - 09:23

Its not pingable when the WLC is disconnected. Neither is the management IP. If I ping those IPs from another network, I have great connectivity. No apparent issues at all. Only with the APs.

I will say this, sometimes when just navigating the WLC's web interface (or via CLI), it can get really slow almost unresponsive. I'm not sure if that's because of the constant rediscoveries, or issues talking to the APs or not. I'm going to debug some on the WLC and see what I see.

Scott Fella Thu, 03/22/2012 - 09:26

So the port the WLC is connected to negotiates fine at 100 full?  On the WLC for the management, ap-manager and wlans, you have port 1 as primary and the backup set to '0'?

matt.millington Thu, 03/22/2012 - 09:30

Yes, it negotiates to 100Full. There are some runts on the switchport, but I don't know how old they are. They haven't incremented at all today. Otherwise, switchport looks good.

I have everything (managment, ap-manager, and WLANs) on port 1 of the WLC. I don't have a backup set at all. Not sure how to do that...

Scott Fella Thu, 03/22/2012 - 09:41

Don't worry about the backups until you can get these aps to join.  Are these ap's new or converted?

matt.millington Thu, 03/22/2012 - 09:45

OK. These are brand new APs, not converted. They join, then they "unjoin" and have to rejoin, but even when they show as joined, we can't connect to any of the WLANs.

Could it be DHCP option 43-related? I was a little questionable on that setup, but I figured that since the APs found the WLC and joined, it must have been set up correctly. I followed the Cisco instructions exactly...

Scott Fella Thu, 03/22/2012 - 09:48

You don't need option 43 nor DNS since the AP's and WLC are on the same subnet. Upgrade to either 6.0.x MD or 7.0.230.0

Thanks,

Scott Fella

Sent from my iPhone

matt.millington Thu, 04/12/2012 - 08:33

So, after calling TAC, the fix ended up being to go into the Advanced tab on the wireless section of the controller and check the box for Pre-standard 802.3af switches as the source of power. By default, both that box and the Power Injector box were unchecked, and the switches are Cisco 3750s, so I'm not sure exactly why that worked. Do those switches not run standard 802.3af power?

kmiller1634 Thu, 03/22/2012 - 10:38

Matt,

   This maybe a long shot but do you have multiple controllers?  If so are any of them setup as a master controller??  I have seen something similar happen when 2 WLC's were both set as master.

If all else fails try setting your only controller as a master and see if the behavior changes..

Hope this help.

matt.millington Thu, 03/22/2012 - 18:31

Thanks Kayle. I'll try that tomorrow.

Scott, thank you so much for your time and efforts. I upgraded to 7.0.230.0 and have the same symptoms, though the error messages in the WLC logs are different now. They now say that the radio is disabled, but the APs still keep disassociating for some reason, but they work perfectly when the AP is plugged into the WLC directly.

I'm definitely lost. I opened a TAC case tonight.

-Matt

Scott Fella Fri, 03/23/2012 - 04:52

Well keep us posted so others can know what the fix was.

Thanks,

Scott Fella

Sent from my iPhone

Actions

Login or Register to take actions

This Discussion

Posted March 22, 2012 at 6:52 AM
Stats:
Replies:27 Avg. Rating:5
Views:1409 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard