Re: Issues with client looses connection with 6.0.196.0

martin.wigertz · ‎05-11-2010

Hi,

We recently upgraded our Wisms to version 6.0.196.0 and after the upgrade we have seen the clients sometimes looses connectivity and has problems to connect. We use an network with 802.1x and PeapMschapv2 with dynamic WEP.

We have disabled aggresive load-balancing but still see the issues.

Any ideas anyone?

Regards,

Martin

CFayNTAdmin83 · ‎05-13-2010

I have a very similar network configuration except I have a 4402 standalone controller. I noticed client disconnection problems on version 6.0.196.0. In fact, I spoke to TAC and they gave me an engineering release. I had major intermittent client disconnection problems and failure to pass traffic / broadcast traffic problems using version 6.0.196.0 and H-REAP. It worked much better with normal mode / central switching enabled. We tried H-REAP on the 6.0.196 AS code and it still had the problem where the client would have to reauthenticate or would not grab an IP address. Once I put the 1252 AP's back into access port mode on the switch and normal mode on the controller, the problems are almost non-existent. This is the 3rd day after the config was reverted. I have to schedule a time / date to get the H-REAP config back in place and perform packet captures with TAC to get to the bottom of the problem. I did a quick test with the normal mode and keeping the AP's in trunk native mode. This resulted in intermittent client issues. I reverted back to access port mode and the problems were resolved. Right now we're working really well on the AS release, but HREAP still has a problem. I just thought I'd share my battle damage stories with this version to see if anyone else comments on what they've experienced. We have a fairly ancient IOS release installed on our core switches, so I'm thinking that the trunk port / Dot1Q tagging was the issue with HREAP. The packet trace will hopefully tell us for sure. Just to be thorough, I noticed this problem on a 40bit wep network and also on a PEAP (MS-CHAP v2) / WPA2-E / AES network. If you're having issues on this code, I'd contact TAC and see if the problems you're having can be cured by the AS release. The 6.0.196.0 release caused some of my radios to reset and a few AP's to crash. The AS release fixed these problems for me. Apparently this code release is going to be available in June as a non AS release. Anyone else care to comment on version 6 code problems?

Kayle Miller · ‎05-13-2010

I ran into a scenario that had TAC chasing a ghost for a long time and it turned out to be something really stupid.. Make sure that the vlan interface for the SSID on the controller is not the same as the interface you are terminating locally to, if they can see each other because it creates a spanning-tree loop. it was wierd and took time to track down.

manandhar · ‎11-15-2010

Hi,

Could you please calrify a bit? I did not understand "the interface you are terminating locally to". Which interface is this one and "locally" to which eqpt ?

Thanks,

jrodriguez · ‎06-01-2010

I Have the same problem. We have a mixed enviroment with some local SSID and the production one in HREAP mode.

The people connected to the production SSID (HREAP) are disconnected 10 times per day or event more.

This not happens in all of our wireless so I`m suspecting too in the trunk link or a bug in the WiSM firmware. Thats not happens with the 5.2.178, I´m thinking seriously in a downgrade.

In the wireless we can see a fisical disconnect due to a change of capwap state

*Jun 1 08:23:25.314: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN
*Jun 1 08:23:25.336: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to a
dministratively down
*Jun 1 08:23:26.474: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio
0, changed state to down
*Jun 1 08:23:26.885: %CAPWAP-5-CHANGED: CAPWAP changed state to UP
*Jun 1 08:23:27.018: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to r
eset
*Jun 1 08:23:27.513: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up
*Jun 1 08:23:28.513: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio
0, changed state to up

Some one has found a different solution that configure the local wlan?

Thanks

trent.g.allie · ‎06-15-2010

I have had issues on 6.0.196.0 with all clients associated with the same AP closing connectivity all at once, and found that the issue was due to the AP randomly stopping sending beacons. This was related to an issue with multicast being enabled on the controller. Once multicast was disabled, beacons were no longer missed.

you can check this by telnetting to AP and doing a show cont d1 (or d0 depending upon radio) | beg TXSM and look for Max Time w/o Beacons.

show cont d1 | beg TXSM
TXSM Jammer information:
State: 0
Beacons seen: 0
Time since: 0
Jammer Radio resets: 0
TXSM Beacon information:
Monitoring State: 1
Flags: 00000000
Beacons seen: 1851048
Time since: 1
Max Time w/o beacons: 2
Beacon stopped count: 0
Counts > 120 0
Counts > 90 0
Counts > 60 0
Counts > 30 0
Counts > 15 0
Counts > 10 0
Counts > 5 0

Not sure if this applies to your issue, but maybe it will help someone

jrodriguez · ‎06-22-2010

Hi Trend,

I´m afraid mine is not the same problem as yours. I already had multicast disabled.

I have a fisical disconnect of the radio interface after the CAPWAP protocol negotiate.

*Jun 15 13:36:22.838: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN
*Jun 15 13:36:22.896: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Jun 15 13:36:23.076: %CAPWAP-5-CHANGED: CAPWAP changed state to UP
*Jun 15 13:36:23.132: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Jun 15 13:36:23.423: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up

I have the ap in HREAP mode and some WLAN in H-REAP local switchig mode and others in Local Mode.

I have realized that when I configure all WLANs as Local (not H-REAP mode) then I don´t have any lose of connection and in the ap the interface remains up, even when the ap is in H-REAP mode.

At the beggining i tried the ap in local mode and all the WLANs in local mode too, and that works, but the problem was the time that the ap last changing from H-REAp to Local. Then I try with the APs in H-REAP and the WLAN in local mode and that works

But this workaround is not useful for the small offices APs where the throughput is a concern. So at last I decided for a mixed enviroment. In the offices where the throughput is high we use H-REAP aps with local Wlan and the WiSM 6.0.196.0 version.

For the small remote offices I downgrade de WiSM to the 5.2.178.0 version. (The prized 5.1.51.0 version with the same configuration makes the controller reset)

We´ll have to wait for a new version.

Leo Laohoo · ‎06-22-2010

We´ll have to wait for a new version.

7.X is now out.

trent.g.allie · ‎06-23-2010

7.0.98.0 is out, but I am playing it safe and waiting for MR1 of 7.0 before migrating the enterprise. IMO the safer option is waiting a few weeks untill 6.0MR3, but it all depends on your organization, and how easy it will be to upgrade again if/when major bugs are found in 7.0.

jrodriguez · ‎06-25-2010

I've found this note from the VLC Config guide 6.0:

"When you enable hybrid-REAP local switching, the Learn Client IP Address check box is enabled by default. However, if the client is configured with Fortress Layer 2 encryption, the controller cannot learn the client IP address, and the controller periodically drops the client. Disable this option so that the controller maintains the client connection without waiting to learn the client IP address. The ability to disable this option is supported only with hybrid-REAP local switching; it is not supported with hybrid-REAP central switching."

But I did it and the problem persist.

I will try with the version 7

jlouhelainen · ‎11-12-2010

Hi,

Any news for these problems? Did the version 7.x fix these CAPWAP UP/DOWN problems with HREAP APs? It seems that I'm having these same problems now. Local APs seems to be fine, but HREAP APs goes UP/DOWN with versions 6.0.196.0 and 6.0.199.4.

There are three 4404 WLCs, where are different sites' office and warehouse APs connected to. The environment is configured as N + N + 1

like following:

WLC1 is for officeAPs. APs are 1131 and they're configured as H-REAP

WLC2 is for warehouseAPs. APs are 1242 and they're configured as Local

WLC3 is for backup and I used this for testing.

After I updated all the WLCs at summer to version 6.0.196.0, there have been issues where the clients looses randomly WLAN-connections. But the complaints just came from the two largest offices. And these didn't happend immeaditly after the update, but after few months. It can be that the summer holidays was reason for it. After some investigations, I updated the WLC3 for version 6.0.199.4, which just got released at the time, and moved these two largest offices' APs to there. Complaints stopped but after a month the second largest office informed that the clients' disconnections started to raise again and the WLAN-network has become unstable again. First there was just couple of issues but it have been raised as the time goes by.

When I made some more investigation to this site, I noticed that the APs' CAPWAP goes DOWN and UP randomly for a second:
*Nov 12 07:39:29.537: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN
*Nov 12 07:39:29.590: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Nov 12 07:39:29.590: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Nov 12 07:39:29.759: %CAPWAP-5-CHANGED: CAPWAP changed state to UP
*Nov 12 07:39:29.811: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov 12 07:39:29.812: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov 12 07:39:29.849: %LINK-3-UPDOWN: Interface Dot11Radio1, changed state to up
*Nov 12 07:39:29.850: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov 12 07:39:29.889: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up

Then I investigated some more and noticed that this happens on all of the H-REAP APs and almost in the same time:
*Nov 12 07:39:29.537: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN - WLC3, second largest site's AP (6.0.199.4)
*Nov 12 07:40:17.439: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN - WLC3, second largest site's other AP (6.0.199.4)
*Nov 12 07:41:20.297: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN - WLC3, largest site's AP (6.0.199.4)
*Nov 12 07:42:31.928: %CAPWAP-5-CHANGED: CAPWAP changed state to DOWN - WLC1, random site's AP (6.0.196.0)

When I checked the warehouseAPs, which are configured as local in the WLC2 (6.0.196.0), there was nothing like this in the logs. Logs are clean.

But when H-REAP is configured for these APs, the clients should not be disconnected when the CAPWAP goes down? And this whole "process" seems to last just for a second. Not sure if this is the real reason for these client disconnections, but at least this is not normal. I haven't heard complaints from smaller sites, even when APs at them also does the same.

I don't feel great to update the WLCs for the version 7.x, because it just seems to be new feature release, but if it fixes these, then I should think about it. Has someone, who had these same problems, got this fixed with version 7? Or has TAC provide some other patches for this?

Thanks,
Joni

jlouhelainen · ‎11-12-2010

Hi,

I investigated some more and finally noticed that these CAPWAP DOWNs and UPs are actually coming when I enable/disable SSH for the AP from WLC. So actually these has nothing to do with the original question and for the unstable WLANs. Interesting here is that the SSH enable/disable didn't seem to have effect for the Local APs in WLC2. Of course the SSH worked normally, but at the logs were clean from these CAPWAP downs and ups.

It seems that the update from 6.0.196.0 to 6.0.199.4 fixed the problem with the largest site's unstable WLAN, but there are some other issues at the second largest site.

Joni