WLC Heartbeat/timeout errors

Unanswered Question
Mar 29th, 2012
User Badges:

Had various issues with WLC recently.  Still not sure if it's WLC or LAP that's causing issue.


Here are the logs:


5Thu Mar 29 16:20:10 2012AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac
6Thu Mar 29 16:20:10 2012AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Max Retransmission Status:NA
7Thu Mar 29 16:20:10 2012AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Max Retransmission Status:NA




0 Thu Mar 29 05:16:31 2012 AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac

1 Thu Mar 29 05:16:31 2012 AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

2 Thu Mar 29 05:16:31 2012 AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

3 Thu Mar 29 05:09:12 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

4 Thu Mar 29 05:06:45 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

5 Thu Mar 29 05:05:09 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

6 Thu Mar 29 05:02:49 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

7 Thu Mar 29 05:01:01 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

8 Thu Mar 29 04:59:24 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

9 Thu Mar 29 04:56:46 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

10 Thu Mar 29 04:55:11 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

11 Thu Mar 29 04:53:59 2012 Coverage hole pre alarm for client[1] 70:de:e2:79:b1:16 on 802.11b/g interface of AP 00:3a:98:98:fb:b0 (AP5475.d09b.baca). Hist: 0 1 7 19 24 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

12 Thu Mar 29 04:53:36 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

13 Thu Mar 29 04:51:26 2012 Client Excluded: MACAddress:00:20:00:9a:32:7d Base Radio MAC :00:3a:98:98:fb:b0 Slot: 0 User Name: unknown Ip Address: unknown Reason:802.1x Authentication failed 3 times. ReasonCode: 4 

14 Thu Mar 29 04:48:36 2012 Rogue AP : 00:24:b2:80:a8:aa detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -58 and SNR: 34 and Classification: unclassified

15 Thu Mar 29 04:48:36 2012 Rogue AP : 00:18:39:d8:91:77 detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -82 and SNR: 10 and Classification: unclassified

16 Thu Mar 29 04:48:36 2012 Rogue AP : 00:20:a6:a5:18:b5 detected on Base Radio MAC : 00:3a:98:98:f9:c0 Interface no:0(802.11b/g) with RSSI: -82 and SNR: 13 and Classification: unclassified


Log System Time Trap

3 Thu Mar 29 05:31:03 2012 AP Disassociated. Base Radio MAC:c4:71:fe:42:15:ac

4 Thu Mar 29 05:31:03 2012 AP's Interface:1(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout

5 Thu Mar 29 05:31:03 2012 AP's Interface:0(unknown type) Operation State Down: Base Radio MAC:c4:71:fe:42:15:ac Cause=Heartbeat Timeout






Issue occurs across two different APs.  One I think may actually be bad?  the one with MAC 15:AC.  No lights come on this device when plugged in.  Keep getting these errors.


The other is working on the LAN at work but not working at remote location.  Gonna head back out there with it and see if it works.


15:AC is a brand new out of the box LAP.


Any ideas of where to start would be appreciated!

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
George Stefanick Thu, 03/29/2012 - 11:02
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, October 2015

Hi Kyle,



Have you looked at the switch port status this ap is connected to, if not lets take a peek and see if there are any issues on the port. Also do a show log on the switch and see if there is any activity on the ap ports as well.


How many WLCs and APs do you have and all the other aps are fine?

Kyle Gatlin Thu, 03/29/2012 - 11:15
User Badges:

The issue has followed the AP across multiple interfaces at location.  AP also generates same error message on WLC at main office as well.


The WLC and two APs are currently working at remote location.  Two APs were sent back.  The non functioning AP and the one that was to replace it but didn't work.  I got one to work at the office here, but the other still generates those errors.  The 15:AC device.


Gonna take the other one out Monday and see if I can get it to work.  But when in remote location, AP would not work on any interface.  Just joins the WLC for a few seconds and then disappears. 

George Stefanick Thu, 03/29/2012 - 11:27
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, October 2015

What is the connection between the remote locations and the location the WLC is at..?


The AP creates a CAPWAP tunnel between the AP and the WLC. Inside this tunnel there are 2 paths. A control path and a data path. The control path is what keeps your ap connected to the WLC. There are speciifc latecy requirements for this control path to stay up. If you see her up and down. I suspect the latecny could be your issue. What do these WAN links look like ?


Also if they are remote could might consider HREAP local switching perhaps to keep that traffic local.

Kyle Gatlin Thu, 03/29/2012 - 11:31
User Badges:

There's no connection.  They're both networks that I manage that happen to have a similar WLC setup.  Both are independent from each other.


At the remote location it's three 3750s fiber trunked together.  Vlan 130 is the Vlan for wireless.  This vlan is the vlan for the WLC and AP interfaces.


The 3750 that the WLC is connected to is the switch I grabbed some of those logs from.  The AP I was testing with was also hanging off this same switch (15:AC). 


The APs that were working were hanging off of the trunked switches. 

George Stefanick Thu, 03/29/2012 - 11:39
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, October 2015

Sorry I like pictures and crayons sometimes.. Helps me visualize ... So your remote location(s) has a WLC and acces points. There is no WAN invloved. Understood...



Now knowing this a new round of questions ...


What WLC model and code are on

What is your AP model

How is your switch port configured at your WLC

How is your switch port configured at your access points

Kyle Gatlin Thu, 03/29/2012 - 11:47
User Badges:

NP.  Thanks for your help so far!  No WAN.  Remote has a cable modem that hooks up to a Cisco router-on-a-stick with sub interfaces and all that.  I'd also like to note that everythign was working just fine until last week.


Here's the interface pointing to WLC.


interface FastEthernet0/2

description Trunk wlc

switchport trunk allowed vlan 130

switchport mode trunk

end


Here's a switchport for an AP.


interface FastEthernet0/7

switchport access vlan 130

end


I am not at location so I can't get exact WLC info at the moment.  Here's what I got via cdp neigh detail


Device ID: Cisco_d6:a2:e0

Entry address(es):

  IP address: 192.168.130.5

Platform: AIR-WLC2106-K9,  Capabilities: Host

Interface: FastEthernet0/2,  Port ID (outgoing port): Fastethernet0/0/1

Holdtime : 134 sec



Version :

Manufacturer's Name: Cisco Systems Inc.  Product Name: Cisco Controller  Product Version: 6.0.196.0  RTOS Version: 6.0.196.0  Bootloader Version: 4.0.191.0  Build Type: DATA + WPS



advertisement version: 1

Management address(es):

George Stefanick Thu, 03/29/2012 - 12:09
User Badges:
  • Purple, 4500 points or more
  • Community Spotlight Award,

    Best Publication, October 2015

I might recommend upgrading the code on the WLC. The 6 train is very buggy.. I posted this on my blog, that code had some bugs labeled as  "catastrophic", not my words, rather what cisco labeled the bug severity.


http://www.my80211.com/cisco-field-alerts/2010/6/24/bugs-csctf34858-severity-1-catastrophic-wlc-code-levels-6018.html



So this was working fine and then started up all of a sudden ... Any chnages to the network at all ?

Kyle Gatlin Thu, 03/29/2012 - 12:16
User Badges:

Thank you very much!  I'll look up on how to upgrade the WLC to that code base!  Hopefully that does the trick!

Kyle Gatlin Wed, 04/04/2012 - 07:27
User Badges:

Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Apr 02 12:34:01.451

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 12:38:34.546



(Cisco Controller) show>ap join stats summary c4:71:fe:42:15:ac



Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Not applicable

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 11:33:12.444



(Cisco Controller) show>ap join stats summary 54:75:d0:9b:bf:f2



Is the AP currently connected to controller................ No

Time at which the AP joined this controller last time...... Not applicable

Type of error that occurred last........................... AP got or has been disconnected

Reason for error that occurred last........................ Timed out while waiting for ECHO repsonse from the AP

Time at which the last join error occurred................. Apr 02 11:58:09.217

Actions

This Discussion

Related Content

 

 

Trending Topics: Other Wireless Mobility

client could not be authenticated
Network Analysis Module (NAM) Products
Cisco 6500 nam
reason 440 driver failure
Cisco password cracker
Cisco Wireless mode