AP reboot issue

Unanswered Question
Feb 21st, 2012

Hi All,

I have a deployment in Customer site, with 3502i AP and 5508 WLC, we are 3750X series switches for the AP connectivity. My issue is I see the AP's reboot randomly. There are 25 + stores in the building and at times the AP will reboot in the basement, next it might reboot in the 20 floor, next in 5th floor. And that also the AP will reboot many times, I was just looking at the WCS events and could see 1 AP was rebooting . Below is the log from the WCS

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 08:05:02           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 08:04:59           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 08:04:59           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 08:04:12           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 08:04:12           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:29:39           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:29:39           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:29:36           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:29:36           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:28:58           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 05:28:58           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:55:47           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:55:47           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:55:44           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:55:44           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:54:53           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 01:54:53           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:19:58           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:19:58           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:19:55           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:19:55           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:18:48           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           22/02/12 00:18:48           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:51           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:51           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:48           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:48           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:10           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 17:26:10           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:01:18           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:01:18           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:01:15           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:01:15           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:00:29           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 09:00:29           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:15:05           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:15:05           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:15:02           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:15:02           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:14:07           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 08:14:07           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:55:44           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:55:44           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:55:41           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:55:41           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

[Clear]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:54:51           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Critical]           AP AP-NAME, Interface 802.11b/g/n           21/02/12 07:54:51           802.11b/g/n interface of AP AP-NAME is down: Controller 172.19.12.25

[Clear]           AP AP-NAME, Interface 802.11b/g/n           20/02/12 21:33:42           '802.11b/g/n' interface of AP 'AP-NAME' associated to controller '...

Let me know if there is anything that i can change to make sure the AP wont reboot

Thanks

NikhiL

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 0 (0 ratings)
nikhilcherian Tue, 02/21/2012 - 22:19

WLC is having 7.0.220.0 and

C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE1, RELEASE SOFTWARE (fc3) and switch

WS-C3750X

Leo Laohoo Wed, 02/22/2012 - 13:30

Hmmm ... Good info you've provided.

I've never heard of anything regarding both firmwares that would cause random WAP to reboot randomly ... unless you have a cable issue.

There's one way to determine this ... Try the 3750E/X's feature of TDR.  (Link attached)

chopra.harish1 Wed, 02/22/2012 - 19:15

Just a guess,

How about checking the DHCP lease. May be the AP looses the IP address due to lease expires and hunts to get the IP and gets into complete WLC hunting process again.

Anyways, what the min duration of those APs to stay ON ?

nikhilcherian Thu, 02/23/2012 - 01:20

I dont think it s DHCP issue, as there are no DHCP conflcts coming up.

This issue looks to be something related with the power negotiation with the AP and the switch

I did a tdr testing and the AP came up and again after a while it went down.. and the tdr testing was showing as incompleted

ATTW.IDFL15#show cable-diagnostics tdr interface gi 3/0/44

TDR test last run on: February 23 11:44:39

Interface Speed Local pair Pair length        Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Gi3/0/44  100M  Pair A     N/A                N/A         Not Completed

                Pair B     N/A                N/A         Not Completed

                Pair C     N/A                N/A         Not Completed

                Pair D     N/A                N/A         Not Completed

But I can see the Speed negiotiation done to 100M, but actually it should be 1000M.

The AP fell back to this state after TDR testing

Gi3/0/44  auto   on         15.4    Ieee PD             3     30.0

This AP was not joining the WLC and i had to console, it joined the WLC once when I plugged out the cable and plugged in. But lost the connectivity soon and I had to boot with the rcv image for the AP join the WLC

Thanks

NikhiL

Leo Laohoo Thu, 02/23/2012 - 13:34

Thanks for the output.  It's telling alot of stuff.

Firstly, the output states that, yes, it's negotiated to 100 Mbps.  This is already raising red flags.  Next, the Pair status is showing "Not Completed".  What IOS are you running?

Next, there's no result.  That's bad.

Based on your "sh power inline" output, this is telling me the port can support up to 30.0 w of PoE but it's stuck in the "IEEE PD" status.

So based on the two output, here's my analysis:

1.  You've got a cable issue.  Because of the "Not Completed" output I can't determine WHERE the fault is.  (You may need to do a "shut"/"no shut" and run the TDR again.)

2.  Try configuring "speed 100".  This will force the port NOT to use the 4th pair.  If your WAP goes up, then it proves that you've got a cable issue with the 4th pair of pins.

nikhilcherian Thu, 02/23/2012 - 22:03

will this have any effect if i try to do the test within veryshort span, i tried the test in short spans and i got the following output once

ATTW.IDFL15#sh cable-diagnostics tdr interface gi 3/0/44
TDR test last run on: February 23 14:18:10

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi3/0/44  auto  Pair A     2    +/- 10 meters N/A         Open
                 Pair B     N/A                N/A         Normal
                 Pair C     2    +/- 10 meters N/A         Short/Crosstalk
                 Pair D     N/A                N/A         Normal

The AP in test wasnt joining the WLC, but I consoled it and booted with the RCV image and it joined, but that is surprising for me

Thanks

NikhiL

Leo Laohoo Thu, 02/23/2012 - 22:40

Thanks Nik.

.... Ummmm ... I have no idea where to begin.  Let me end it this way, you've got one LOUSY cable.  I've never seen a cable result that BAD before.  Open circuit?  Short?  These two values are pretty bad.

Please change the cable closest to your switch.

nikhilcherian Thu, 02/23/2012 - 23:02

Leo, thanks for the analysis, i m in a really bad state, this issue is happening for me through out the network. I have 140+ AP in my network and it s randomly happening..I have raised a TAC, let me see, how the TAC will resolve the issue. But it s Jordan TAC, so I have my fingers crossed

Thanks

NikhiL

airframes Thu, 02/23/2012 - 23:15

Leo,

Could Nik be looking at an entire cable plant issue? I'm thinking maybe the building is cabled with pre-Cat5e, pre-1995 "Cat5." Gigabit is intermittent on that old stuff, not to mention PoE in the mix adding EMF.

Good call on the TDR diag.

Justin

Sent from Cisco Technical Support iPhone App

nikhilcherian Fri, 02/24/2012 - 00:28

Justin,

Do you mean to day with pre-Cat 5e I will not get 1 Gig speed, which means it will not suppor the AP with 1 Gig speed, I have asked for a cable report, Let me see if that will help

Thanks

NikhiL

airframes Fri, 02/24/2012 - 01:24

Nik,

You can get 1Gb to run on Cat5, but only at shorter distances. Specific distances vary by manufacturer and product because, if I recall, Cat5 wasn't really an industry wide standard. Rather, it was a loose spec that meant a wide range of product quality hitting the market. It wasn't until 1995 and Cat5e that minimum specs for twists/m, frequency and other parameters were standardized.

I could be wrong about the specific history there, but to put it simply, you can't trust 1Gb on Cat5 and you can't trust what exactly "Cat5" means.

Check the jacketing on your cable. Pre-standard wiring will usually have "CAT5," cryptic/unfamiliar printing, or nothing at all stenciled on it.

Justin

Sent from Cisco Technical Support iPhone App

Leo Laohoo Fri, 02/24/2012 - 01:26

Look at your latest TDR result.

Pair "A" is "OPEN"???  This pair will determine if your WAP should connect or not. 

The cable directly connected to the switch is faulty.  Sorry to say this (I'm not being rude or arrogant here), but I don't care if the cable came out from a pack.  If your TDR result is saying it's faulty, it's faulty.

You can RMA all the WAP you can lay your hands on, if the cable is faulty ...

Leo Laohoo Fri, 02/24/2012 - 01:27
Could Nik be looking at an entire cable plant issue?

Heck no Justin.  Look at the cable "length".  It says it's under 10 metres.  So I'm suspecting the fault is as close as possible to the switch.

nikhilcherian Sat, 02/25/2012 - 06:37

Thanks for the replies, I have asked for the cable report for the entire building, hope that will help me out

Thanks

NikhiL

Leo Laohoo Sat, 02/25/2012 - 14:16

Cable report?  What method?

The official report I'll only accept is from a Fluke DTX-1800.  Anything else is subject to heated debate. 

nikhilcherian Sat, 02/25/2012 - 22:35

Leo,

That was awesome :

i havent received the report yet, hope that will a Fluke report

NikhiL

nikhilcherian Sun, 02/26/2012 - 10:43

Leo,

I got the fluke report, but this looks quite odd for me, I can see all are Cat 6 cables and well under 100 meters.All the wires also have passed the testing.

I have attached one print screen from the report, can you please help me with what I should  check from this one

Thanks

NikhiL

Leo Laohoo Sun, 02/26/2012 - 12:44
 I can see all are Cat 6 cables

TIA Cat6 Perm Link is an adaptor with a PLA002 "tip".  IT doesn't measure cable-to-cable.  This only measure patch panel-to-patch-panel.

Good enough for me.  I'm still wondering as to why your Cisco TDR is coming up with the result. 

Hmmmm ... May I request something from the cable testers?  Can I request if they test from cable-to-cable test?  I want to eliminate the patch cord (switch end) to the WAP end patch cord using the same method.

I do NOT mean they take down both patch cord and test it.  I mean leave the patch cords connected to the patch panel and run the test using the DTX-CHA001 module at both ends.

I've just reviewed your cable report.  Why did the cablers do that????  Look at the right-hand corner.   Look what is heading under the "Main Adapter" and "Remote Adapter".  Experience have taught me NEVER to mix the PLA002 with the CHA001.  If one was to test using the PLA002 then the remote should also be PLA002.  Weird!

Scott Fella Sun, 02/26/2012 - 11:09

Well that is a good report, what I would check next is the patch cables. If you take that ap and connect it directly to the same switchport using the same cat5/6 cable, does the ap negotiate at 1000mbps? I not, swap out the patch and test again. I had an install about half a year ago that tested clean using a fluke. When we started to bring up the AP's, only 5% of the AP's came up. After hours of testing, we swapped out the patch with a different batch of patch cables we bought and presto... Everything came up. So it was a bad batch of cat6 even though for some reason we could also test those clean with a fluke. One trick I do now when I run into situations like this is to have a can of air and blow out any dust from the ap Ethernet port, the jack, patch panel and switchport. The only reason is usually I noticed that hen mounting on the ceiling, the ports usually gets some ceiling tile particles in it before they plug the able into the jack and the ap. On the wiring closet side, not so bad unless they are putting up the ceiling or something at the same time.

Sent from Cisco Technical Support iPhone App

nikhilcherian Mon, 02/27/2012 - 08:06

Leo/Scott

Thanks for the mails,

I couldnt do much more testing on the cable part, as this was done by a separate vendor. I re-routed the TAC call to RTP and I got a sudden repsonse CSCtu24972

Wonder why the Jordan TAC couldnt inform this

I have been requested to RMA, all the AP which rebooted with the reason Watchdog timer reset, Hope this will resolve my issue,

Thanks a lot for all the help

Regards

NikhiL

Leo Laohoo Mon, 02/27/2012 - 13:44

I hope Cisco TAC is right on this one as I don't agree that it's a hardware fault on the WAP.

Scott Fella Mon, 02/27/2012 - 19:34

Well I must agree with Leo. Hopefully

It is an issue with the AP, but just keep us inform if it is:)

The one thing is that many of the cable testing is from patch panel to jack (on the ap side). They only need this to verify their installation for warranty. I don't know any vendors that I have worked with that will test with the patch cable since that might affect their testing and qualifying their installation.

Sent from Cisco Technical Support iPhone App

Actions

Login or Register to take actions

This Discussion

Posted February 21, 2012 at 9:50 PM
Stats:
Replies:23 Avg. Rating:
Views:1748 Votes:0
Shares:0
Tags: No tags.

Discussions Leaderboard