Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Wireless APs going down randomly

Hi All

We have a situation where wireless access points are just going down randomly on our network. The situation is as follows:

We have 2 Cisco 5508 WLCs (licensed for 250 APs each, with more licenses en-route), both running software version 7.2.103.0. Then we have a large-ish number (±300) of Cisco AIR-CAP3502I-E-K9 APs connecting to these two controllers. The APs are either connected to Cisco WS-C2960S switches, or in "older" areas of the network they are connected to WS-C2950 switches. We do not have PoE switches installed yet, so all APs are powered by power injectors (AIR-PWRINJ4=).

Anyhat, all the access points are in FlexConnect (previously HREAP) mode, and they all connect via trunk ports to the switches in order to get the VLAN per SSID working. Both controllers are in the same mobility group, and both are in the DNS as "cisco-capwap-controller" as they are load balancing. The APs are installed about 25m apart in most areas. There is one installation where two APs are about 10m apart, but this is due to the shape of the building in which they are installed. We have no autonomous APs deployed at all...

The problem is that some access points work for about a day, and then they just disassociate with whichever controller they were associated to, and then they just fall off the network completely. Both WLCs, the NCS, and the network monitoring system (WhatsUp Gold) show the affected APs as physically unreachable. Looking at the trunk port on the switch, it shows that the port is up and the line protocol is up, input rate is 0 bits/sec, and output rate is up to 70,000bits/sec. There are no input or output errors on the switch port. Here is the output from one such switch port:

FastEthernet0/19 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is dc7b.9439.da13 (bia dc7b.9439.da13)

  Description: Link to NC_K22-AP1

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:50:20, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 71000 bits/sec, 91 packets/sec

     977358 packets input, 80661413 bytes, 0 no buffer

     Received 29925 broadcasts (16038 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 16038 multicast, 0 pause input

     0 input packets with dribble condition detected

     37567889 packets output, 3624164243 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

 

Connecting a console cable to an affected AP is fruitless, as the AP appears to be unresponsive while it is "down". Rebooting the AP, then connecting a console cable works, but then you've lost your syslogs. Looking at the Ethernet interface on the AP, it appears as if the interface is picking up a lot of errors, especially CRC errors, which would indicate a physical cable problem.

I got our cabling contractor out to do a cable test again, and it passes every time. We have them test both the Systimax and the Molex cable installations of affected APs, and all is OK. They use a Fluke MicroScanner² cable tester which certifies the Cat 6 cable from patch lead to flylead. No problems there, and I trust that the cable is 100% as tested.

So I swap the power injector of an affected AP (let's call it AP A) with an AP that is not affected by this (let's call this one AP B), and AP A goes down again after a day, or sometimes only a couple of hours. AP B remains operational as usual. If I swap AP A out with a new AP (AP C), then AP C starts showing the same symptoms after about a day, and then it also just goes down. When "suspect" AP A is installed somewhere else, then it works and stays operational for, well, 58 days now. Changing the switch/switch port also makes no difference.

This happens to one of the APs that are part of the "10m apart deployment", but it also happens to another AP that is completely on its own in a building far away from any other wireless source, so I cannot think that this could be the problem. I happens to an AP that is less than 100m away from its closest WLC, and it happens to an AP that is more than 350km away from its closest controller. All the other APs deployed in our remote campus (350km fibre link) are 100% OK. It happens to APs connected to the older WS-C2950 switches as well as the brand new WS-C2960S switches.

I am at wits end with this one. Where could the problem be? I trust the "Big Green" company that does our cable installations, and I trust the equipment they use. I also trust both Systimax and Molex to be very good products. And I do trust Cisco, as this is the only network equipment we deploy.

Where to from here?

Regards

Connie

Everyone's tags (4)
1 ACCEPTED SOLUTION

Accepted Solutions
Hall of Fame Super Silver

Re: Wireless APs going down randomly

That's called the white LED of death:) console into the ap and look at the output when you reboot the ap.

Thanks,

Scott Fella

Sent from my iPhone

-Scott
*** Please rate helpful posts ***
9 REPLIES
Hall of Fame Super Gold

Wireless APs going down randomly

I got our cabling contractor out to do a cable test again, and it passes every time. We have them test both the Systimax and the Molex cable installations of affected APs, and all is OK. They use a Fluke MicroScanner² cable tester which certifies the Cat 6 cable from patch lead to flylead. No problems there, and I trust that the cable is 100% as tested.

I hope some of your problematic WAPs are connected to a 2960S because if you do, then I recommend you run a TDR.

New Member

Wireless APs going down randomly

Hi leolaohoo

Thanks for that! That was something I have not tried up to now, so I did it. Here is the output:

TDR test last run on: May 15 12:44:57

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi0/8     1000M Pair A     61   +/- 0  meters Pair A      Normal
                Pair B     62   +/- 0  meters Pair B      Normal
                Pair C     62   +/- 0  meters Pair C      Normal
                Pair D     61   +/- 0  meters Pair D      Normal

I did find the two bugs relating to this issue (CSCtw90465 and CSCtu24972). Both should have been resolved in releases 7.0.230.1 and 12.4(23c)JA5. That being said, I did notice that our 3502i APs are on boot version 12.4.23.0, which does not appear to be the same as the 12.4(23)JA5 mentioned in the bug ID...

Cheers

Connie

Hall of Fame Super Gold

Wireless APs going down randomly

Hmmmm ... TDR looks good.

I remembered reading something like this in the forum.  I believe Wes or "weterry" (from Cisco) has a solution but I can't find the post relating to this.

Hall of Fame Super Silver

Wireless APs going down randomly

I have three 3502i acting the same way out of 650+.  I would bounce the port (PoW) and it would come back up.  The funny thing is that if I do a show mac address-table interface giX/X/X, I would see no vlan information at all.  I'm going to see if they keep on failing and I will just RMA those three.

-Scott
*** Please rate helpful posts ***
New Member

Wireless APs going down randomly

what do the lights look like when the AP is locked up? are they white?

Hall of Fame Super Silver

Re: Wireless APs going down randomly

That's called the white LED of death:) console into the ap and look at the output when you reboot the ap.

Thanks,

Scott Fella

Sent from my iPhone

-Scott
*** Please rate helpful posts ***
New Member

Re: Wireless APs going down randomly

Yup, indeed, it looks like the white LED of death... We have 7 out of 160 so far...

New Member

Re: Wireless APs going down randomly

White LED of death it was. After a quick WebEx with a Cisco Engineer (Ishant, you were ace!), we were showed what to do to have the problem resolved using console access with a live network connection and a script that updates the required code with interim software.

Apparently the bug will be fixed in the next major update of the WLC software...

New Member

Wireless APs going down randomly

This issue has now been resolved. Details can be found here: http://www.cisco.com/en/US/partner/ts/fn/635/fn63537.html.

Here is the short version:

Problem Description:
Calibration between DRAM and processor on AIR-CAP350x and AIR-LAP126x results in Access Points (APs) failing to boot properly. One symptom includes a white LED and no console output.

Background:

The root cause is the DDR memory calibration process that was performed on the Access Points (APs) at the time of manufacture. The fix recalibrates the existing memory hardware, so it will operate more reliably.

Workaround/Solution:
The fix for this issue is now publicly available. Go to www.cisco.com to obtain Access Point (AP) IOS code version: 15.2(2)JA or later and/or WLC software version: 7.0.235.3 (or later 7.0 release), 7.2.111.3 (or later 7.2 release) or 7.3.101.0 (or later 7.3 release).

As of June 2012, new AP3500 and AP1260 products with version ID (VID) V02 are free of this issue.

1506
Views
4
Helpful
9
Replies
CreatePlease login to create content