AP541N - Degrading Service

Unanswered Question
Feb 9th, 2010

We have an AP541N that has been deployed to replace a Cisco 1200 AP (B/G radio).  The 1200 functioned perfectly in our environment.  The new AP541N on the other hand seems to work fine right after a reboot but immediately starts to degrade service.  Over a short period of time, the devices bandwidth degrades to the point were the wireless network is not usable.  This happens with just one device connected.  Eventually, the device stops accepting client connections.  We are unable to get any relevant logging out of the device to help diagnose the problem.

What should be our course of action?

Lucas Budman

Cognilytics

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
jestowe Tue, 02/09/2010 - 16:13

Thank you for contacting Cisco Community.

Normally we would ask you to perform a factory reset on the device, flash the device with the latest firmware (downloadable from cisco.com) and then Factory default again.

Check to make sure you have a stable power source (no fluctuations).

Change your wireless settings.. i.e. remove security (for testing)

If security was a factor, try using WPA with a 10 character passphrase, remove any spaces or CAPS from the SSID, and use only a single SSID.

If all else fails, contact our Small Business Support Center and continue troubleshooting toward replacement.

Thanks for asking.

Jeff.

garrettct Fri, 02/26/2010 - 07:27

I was having the same problem and recently upgraded to AP541N-K9-1.8(0).  So far this has fixed the problem.

garrettct Tue, 03/02/2010 - 06:08

Disregard my first message, the firmware update did not resolve this problem.  I have sent these units back.

CHRIS YEO Mon, 03/01/2010 - 22:01

Hi Guys,

I have recently experienced this as well.

I have it setup as this:

1 ssid / wpa2 / psk (8 character, all lowercase or numbers), running on a/n with both 1.72 and 1.8.

I have also experienced this while plugged into an ESW-520-8P and into an 871W.

What I have noticed is that over time, the 'ping' times to the router goes up to 300 - 3000 ms and clients are unable to connect.  Rebooting fixes the problem.

I have just put in a new (different) temporary one at our customer testing this, to ensure it is not related to the device.

This was my concern with not putting in a standard Cisco AP.

When I was testing prior to installation, we were using an ESW-540-8P, and didn't seem to have that problem.

We have put in a different ap to ensure it is not a problem with this particular unit.

Thanks!

Chris

CHRIS YEO Wed, 03/03/2010 - 19:46

Ok,

So in an effort to elminate 'environment' as the issue, I have put in a AP1242AG, this is not a 'N' access point, but it will eliminate switch, router, and local airspace.

So... assuming that I have no issues with this, is my solution going to be the same as garrettct, which is just return them.

Thanks!

jwalker@kingsco... Wed, 03/10/2010 - 11:29

I just puchased one of these access point for the office to replace the linksys thats currently hosting our wireless.  I thought this would be heads above what I was using.

I setup the unit with basic settings and had three clients attached to it.  After a few hours we started noticing issues as well.

I upgraded the firmware to 1.8 and reset to factory default.  In mixed mode it doesnt take long for it to degrade out with 3 clients attached.  I changed it to 5 GHz 802.11n only, and just myself attached.  I have been fine for a few hours. Id hate to not be able to use the AP in mixed mode though.

Has anyone else made any progress?

Lichtstein12 Wed, 03/10/2010 - 12:12

We have the same problem. Have three AP541n's, been using a/n and works for about 1 hour then hangs up. The three units are clustered. Have lock the channel on one unit to see if it makes a difference. Has anyone seen a difference with the firmware update??


Elliott Lichtstein lichtstein@westwoodcardio.com

lucas4 Wed, 03/10/2010 - 13:01

I have tried a bunch of different setups.  The access point first had a bunch of VLANs on it.  I took them off and ran a native vlan.  Tried fixed channels and other configurations.  Nothing helps.  My dhcp server in this setup is a cisco 3670 switch.  We are currently running on our old 1200 AP with no problems (same environment and location).

Lucas

sbaharaeen Mon, 03/15/2010 - 09:54

We have had the same issue here.  The problem we are experiencing is that once the AP stops working, we are able to see the AP and connect to it, but we are unable to get an IP address at that point.  Initially, clients were obtaining IP addresses through the network DHCP server.  We changed the setup and connected the AP directly to our Sonicwall firewall and set up the firewall to run DHCP for this connection.  We are still having the issue.

I reported the issue to Cisco and the initial technician I talked to was able to duplictae the problem in their lab.  He has sent the case to the higher level support.  The last email I got from the higher level support (about a week ago) said the case has been qeued for further testins and that I will be contacted when that is done.  It also said that they are very busy and it might take a while for them to get back to me.

We purchased this unit to replace a Lincsys device we were having issues with.  We were told by our consultant that the Cisco AP is a much better unit that would resolve all our issues.  My clients are a lot more frustrated than they were with the Lincsys unit.  Is there another Cisco AP (or any other brand for that matter) that works without any issues in a network environment that costs about the same?

jwalker@kingsco... Tue, 03/16/2010 - 08:11

Thank you for the update sbaharaeen, please keep us posted if Cisco responds to your support request.

I really like the features of the unit and the price point we purchased it.  I really hope they are able to rectify the issues.

Justin

psmith@metafore.ca Thu, 03/18/2010 - 12:02

I have just deployed one of these to a remote office, and I am recieving word of this same issue.

This was going to be the first of probably a dozen or more of these devices for me, but until this is resolved, I obviously can not deploy any more.

I really hope this get's resolved, because I was really liking the product.

jwalker@kingsco... Thu, 03/18/2010 - 12:08

I've been running ours on strictly 5GHz N mode only for the past week and it seems to stay stable.  In mixed mode it continued to have issues.

Ideally I wanted this device to be used for both G and N. Until a fix is available, the change has at least let some users access through the AP.

jdemmer Mon, 03/22/2010 - 09:31

Hi,

in order to ensure that this is solved, would the people who are affected by this please sent me the case number as a private message?

This would help a lot as this would provide insides into FW version, problem description etc.

I use a AP541 at home myself and I don't see this behavior, I use it for VoIP using 2 x 525G phones in Wireless mode, 2 cam's and 3 PC's / laptops.

We take this reported issue very serious, so I'm looking forward to see some details.

Thanks!

Best Regards

Jens

sbaharaeen Mon, 03/22/2010 - 15:38

Jens,

I was having a problem sending you a private message, but I figured it out and sent it to you.

opresterud Mon, 04/05/2010 - 14:30

Hi!

We was hit severly by this today, but have also seen it sporadicaly during testing. The system consists of 4 AP541 accesspoints with firmware 1.8.0 running in a cluster with 802.11bgn support, where 3 is connected directly to a SA540 gateway with cable, and one acts as repeater with WDS connection to one of the cabled AP's. All AP's broadcasts with the same SSID and on channel 9.

We have applied the recommended key refresh fix and removed WPA2 support. The AP's are now configures with WPA-PSK encryption with both TKIP and AES available. Todays usage scenario was about 20 laptop users roaming around for ~3 hours, and potentially moving from one AP to another. The internet connection is a shared 10Mbit access, where we typically measures a bandwith of 5-7Mbit/sec with speed tests at our location through a wireless AP that is connected with cable to the gateway. The speed degrade did occur on all AP's today, but some was hit more often than others (might seem to be related to number of clients connected). When the degrade happens, we could first sence it that the signal strength from the particular AP started to fluctuate more than normal and also drop for a periode of 1-2 seconds before reappearing (monitored using 'inSSIDer' WI-FI scanner), and if we then connected to the particular AP and ran a speed test through it, it had dropped down in the range 300-100Kbps, and after a short while it would hit ~60Kbps and stay there.

DHCP issues have not been seen here (DHCP assigned by the SA540), connecting to a degraded AP was always possible it seemed, but general all access through an AP that had the degrade state was extremely slow. To be able to access the web interface of such an AP, we typically had to ensure we had a connection to a different AP and then enter the web interface for the degraded AP through the normal functioning one.

To get it properly working for our users, we had to constantly monitor the four AP's, and do a reboot at first sign of degrade. Thankfully, since the AP's overlap each other, most of the connected clients was transferred to a different AP when we hit reboot, but without constantly personally monitoring this, the WLAN had practically gone down.

For us, this is a very serious issue, and really hope for a fix in a short time periode. We would gladly provide any kind of information requested about the issue, and also test other recommended fixes or workarounds for this if available.

- Ola

ActiveInfoTech Tue, 04/06/2010 - 07:52

Is ther an update or time frame when this will be fixed.  Just deployed the AP yesterday to replace a Linksys and the Cisco AP541N is showing 1M connection where the Linksys is showing 11M connection.

Steven DiStefano Wed, 04/07/2010 - 13:31

Tier 2 Support and the BU have communicated to us that the Level 1 SBSC agents were informed of this AP541N issue and another DHCP issue as being solved in the next FW release for this product.  When customers call SBSC, the support the agents will follow proper protocol and due diligence to determine the problem, and in the event it is considered to be resolved in the coming release, the case will be escalated to 2nd level support.  At that time and with the Partners agreement, access to the beta firmware 1.9.0 in test right now can be given, once a beta agreement has been accepted.

Any Partner experiencing either DHCP issues or degrading service, please contact the SBSC for further resolution.

I do not have an ETA for the formal release of 1.9.0.

Steve DiStefano

Systems Engineer,

Partner Sales, U.S.

farrshepherd Mon, 04/19/2010 - 12:24

Per your post below, here is our case # on this issue. We are a Cisco Select partner and have these Ap's at many locations are experiencing the problem at most of the locations.

Case # for Cisco Wireless Issue
614127237

lucas4 Tue, 03/23/2010 - 13:02

Cisco has replaced my original AP.  I have just deployed the new AP.  Unfortunately, I believe that it is experiencing the same degrading problem.  It is running firmware 1.8 (same as the original AP).  One interesting thing I did notice though is the following:

From the linux console the AP is reporting a lot of packet errors on the wireless link (this is with one client attached and uptime of one day):

wlan0     Link encap:Ethernet  HWaddr 00:21:29:00:7D:50 

          UP BROADCAST RUNNING  MTU:1500  Metric:1

          RX packets:162000 errors:650285 dropped:0 overruns:0 frame:15361294

          TX packets:421509 errors:3336 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:42518247 (40.5 MiB)  TX bytes:291051214 (277.5 MiB)

          Interrupt:44

Here are the radio interface stats for my Cisco 1200 in the same environment (Uptime on this AP is 4 weeks and has several clients connected):

Dot11Radio0 is up, line protocol is up

  Hardware is 802.11G Radio, address is 0012.01e5.2e90 (bia 0012.01e5.2e90)

  MTU 1500 bytes, BW 54000 Kbit, DLY 1000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/523/0 (size/max/drops/flushes); Total output drops: 130206

  Queueing strategy: fifo

  Output queue: 0/30 (size/max)

  5 minute input rate 5000 bits/sec, 9 packets/sec

  5 minute output rate 4000 bits/sec, 4 packets/sec

     29633972 packets input, 2302875179 bytes, 4 no buffer

     Received 235828 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     26072809 packets output, 2208254017 bytes, 0 underruns

     21421 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

Lucas

Steven DiStefano Wed, 03/24/2010 - 14:11

OK,  spoke to development team.

They suggested we may understand the 'degrading' issue and have a possible workaround, which may be related to "Key refresh".

Try to just disable 'Key Refresh' in each VAP.   Set the Broadcast Key Refresh Rate to 0 for each vap that does WPA(personal or enterprise).

Degrading_work_around.GIF

sbaharaeen Wed, 03/24/2010 - 20:08

I just heard back from the second tier support and they recommended disabling the Key Refresh as well.  I will try this tomorrow and let everyone know how it goes.  Is everyone who is experiencing this issue using WPA?

jwalker@kingsco... Thu, 03/25/2010 - 07:28

I was having issues when I first bought the unit.  I switched over to N mode only and I'm only using WPA2 with AES and things have been running fine.

I will enable WPA and TKIP and attempt this workaround today.

Justin

dmoore@lancerco... Fri, 03/26/2010 - 10:44

Has changing the Refresh Rate to 0 worked for anyone? I changed mine this morning but have not seen any difference. Pings to our core router that were 1ms on the LinkSys the AP541N replaced are still all over the place and are often in the 1000's on the 541.

Steven DiStefano Fri, 03/26/2010 - 11:12

Dennis and all,

These workarounds are suggested until we can be sure of the root cause and fix it.

Based on what we think is going on, there are code changes under test right now.

But what would really be helpful, is if everyone can post their logs (attach to your discussion threads) so we can see clearly what is happening in your environments?

Steve

sbaharaeen Mon, 03/29/2010 - 09:12

Came in this morning and the AP is not responding at all, even to the Windows 7 client that was working fine last week.  We are taking it offline and switching to our old setup.  I will try to run it in a test environment.  Attached are the logs from this morning.

Attachment: 
Steven DiStefano Mon, 03/29/2010 - 14:54

Thank you for the logs.   I passed them to development team.

Will advise whan I hear back.

psmith@metafore.ca Tue, 03/30/2010 - 09:52

My only user using this AP reported losing it 5 times yesterday.

He said it disappears, and then comes back about 5 minutes later.

I can't test it because it is on the other side of Canada from me.

But here is my log file, since the last reboot.

I will ask them not to reboot it again so I can capture more logs.

Attachment: 
Steven DiStefano Tue, 03/30/2010 - 11:30

Thanks for this.  Feeling pretty confident, based on your description of whats happening, that what we are fixing will address this, but more logs would be welcomed as well in the mean time since these logs are just showing disassociate deauth from an unknown reason.

Here are the reasons codes our team found in the log file.

1     /* Unspecified reason */

3     /* Deauthenticated because sending station is leaving (or has left) IBSS or ESS

4     /* Disassociated due to inactivity */

8     /* Disassociated because sending station is leaving (or has left) BSS

.

We will keep you posted and thank you for your help...

UMS-Orsay Tue, 03/30/2010 - 23:59

Hi Steve,

When I had this issue, I have set up a syslogd thinking that It should be more verbose than the AP embedded log page...

but even after several days of monitoring, I've never caught any other error than the ones shown in the log you've seen !

Regards

Vincent

sbaharaeen Fri, 03/26/2010 - 15:25

I changed the refresh rate to 0 on ours and we were still having issues yesterday.  I am attaching the log file from earlier today, but it was quiet around here so I will monitor next week and post the logs.

One interesting thing I noticed though is that one of my notebooks running Windows 7 has no issues whatsoever connecting and maintaining the connection to the AP.  Is anyone else seeing this as well?

Attachment: 
lucas4 Wed, 03/24/2010 - 16:40

I made the change as described above and it seems like I am still getting a relatively large amount of receive errors.

wlan0     Link encap:Ethernet  HWaddr 00:21:29:00:7D:50 

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:82612 errors:84412 dropped:0 overruns:0 frame:774824

          TX packets:125679 errors:4 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:56553958 (53.9 MiB)  TX bytes:135659862 (129.3 MiB)

          Interrupt:44

Lucas

jwalker@kingsco... Wed, 04/07/2010 - 13:39

Steve, this is good news.  Thank you for posting an update.

Has anyone got their hands on this update yet?  I know Steve just posted, but I also know a few of you have been working with level 2 tech support for a while now.  Has the beta firmware corrected any of your issues?

mindmaster Wed, 04/14/2010 - 04:42

Hi,

i have 8 AP541N on an UC560 with 31 SPA525G.

None of the Phones can make calls all over the day without horrible noise during the calls. It's like talking with a mobile phone in the 1980th.

It is equal if the Phone is in the same room as the AP or in the next room. The Signal on the Phones is jumping from green to red and back without doing anything.

The PCs on the AP with external N USB Adapaters won't get more than 10-30 Mbit.

I took for testing an MSI AP200+ Powernet to WLAN Adapter for 3 Phones and this works. The signal goes 2-4 times wider then the AP541.

Nothing fixes the Problem. The latest new thing is, that the cluster doesn't change the channels. Every AP is on Channel 6.

If i can't fix the Problem this week, i think i will loose my customer.

Did the 1.90 fix this Problem, can i get the Beta Version?

Any Workarround?

Config:

UC560

GIG0/1/0 -> SRW2008P -> 4x AP541N              -> x Phones and x PCs

                                    -> 1x AP200+Powernet -> 3 Phones  (working fine)

GIG0/1/1 -> SRW2008P -> 4x AP541N              -> x Phones and x PCs

                                    -> 1x SRW208             -> 4 Phones and 4 PCs (working fine, PCs get 99,3 Mbit from SBS2003)

GiG0/1/2 -> SA520  -> 48 Port 100/1000 Switch -> SBS2003, NAS, ...

                              -> WAN Port VDSL50000 over speedport 920

                              -> Opt. WAN Port VDSL50000 over speedport 920

GIG0/1/0+GIG0/1/1 are configured as APs

GIG0/1/2 configured as Desktop Desktops

All phones are configured for cisco-Voice(vlan100) with WPA2 and TKIP (long key and non ASCI Characters)

APs are configured in Cluster like SBSC2.0 2 SSIDs, Qos, Cluster, ...

Steven DiStefano Wed, 04/14/2010 - 05:56

Please open a case to get the beta code to save this deal.  I am not saying the new code will fix you, but we have heard problems with degrading signal i9n the FCS release....

mindmaster Wed, 04/14/2010 - 06:48

Ok, I'll try it.

Does any one use SPA525G phones wireless?

Is it working?

Thanks

Steven DiStefano Thu, 04/15/2010 - 10:23

Clue number 1 is: "Nothing fixes the Problem. The latest new thing is, that the cluster doesn't change the channels. Every AP is on Channel 6."

If Channel Management is enabled when clustering, make sure to also statically assigns a channel on the radio page, otherwise the channels won't change.  Auto Channel Selection must be disabled to allow Channel Management to change channels automatically in a cluster.

Per the help page, "Note: The proposed channel assignment will not take effect if the Channel field on the Wireless Radio page is set to auto. The channel must be set to a static channel."

mindmaster Thu, 04/15/2010 - 10:46

after a complete power down of all APs the channelchangings works now...

thanks

hss

psmith@metafore.ca Mon, 04/19/2010 - 08:51

How do you get access to the 1.9.0 beta version?

I am new to the Small Business Pro stuff, but we are a Gold level partner.

My AP is just over 90 days old, so I don't think support is going to help me.

opresterud Thu, 04/15/2010 - 13:24

We have tried 1.9.0 beta on our 4 AP541Ns, and during yesterday evening when a number of clients was connected (this is on a football/soccer stadium), the APs seemed to stay stable. The reason I say "seems", is that we had a major problem with our internet provider so the network did not have high traffic during all the 4 hours with use. But I am quite sure some positive things have happened. During testing the evening before, we also found the throughput on a WDS bridge to have increased too. The next scheduled happening with many connected clients will happen in ~2 weeks, so hopefully we will get a real test then. Conclusion so far is good.

jgawf@sajeps.com Tue, 04/20/2010 - 22:07

I have access to the beta 1.9.  The support engineer said to load it and take tho AP back to factory default and re-configure.  I'm managing these two boxes remotely and that isn't an easy proposition.  Does the AP need to be returned to factory defaults after 1.9 is loaded?

F1recloud Tue, 04/20/2010 - 22:40

So far it has worked for us.. upgraded to 1.9 and reset to factory settings and reconfigured.. I'll update tomorrow when I find out a little more on this.

Steven DiStefano Thu, 04/22/2010 - 07:39

If upgrading from v1.8.0 the upgrade process will probably go smoothly without the need for a factory default.  If  upgrading from v1.7.2 that's another story as the default AP parameters changed between this version and v1.8.0 so in this scenario that will definitely require a factory default.

Steve

jgawf@sajeps.com Fri, 04/23/2010 - 16:40

Steve,

Thanks for the information.  There were no notifications I could find about resetting to  factory defaults when going from 1.7 to 1.8.

The Small Business Support Center engineer said that "Per our policy and past experience, it is highly recommended to reset the device to factory defaults after a firmware upgrade.  If you do not reset the AP(s) and they develop problems or become unstable, you will be asked to reset them to factory defaults as the next troubleshooting step."

This is a non starter for me as it really impeeds me from selling these units if the product development team has this "policy".

Please pass along to the product development team that this can't continue for all upgrades.  Even though this product is GPL development from the Linksys group, they can't play fast and loose with the product when it is in sitting in a Small Business.  I can't charge a small business customer my service time to factory reset their products when upgrading software to fix a bug in their small business systems including these APs.

Steven DiStefano Fri, 04/23/2010 - 16:45

I actually got the advice I posted to you from our business unit Product manager.

I am sorry for this, and I understand what your saying. I will pass it along...

Steve

Actions

Login or Register to take actions

This Discussion

Posted February 9, 2010 at 2:49 PM
Stats:
Replies:84 Avg. Rating:
Views:19767 Votes:0
Shares:0

Related Content