Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

Error: Wireless Controller 5508 whit 68 ap

Hello friends

I have a WC whit 68 APs, they work ok, but suddenly occurs following :

   - All AP are detached from WC.

   - WC not respond to Ping

The log is:

*Jan 17 02:59:46.666: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:98:d6:8d:00:  DTLS connection not found forAP  10:11:48:74 (21030), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:46.559: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:46.555: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:46.508: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:99:98:f9:f0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:46.507: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:99:98:f9:f0:  DTLS connection not found forAP  10:11:48:41 (55863), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:46.356: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.109 from the operating system. ioctl operation failed

*Jan 17 02:59:46.304: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:99:85:9a:d0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:46.263: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:98:ea:e4:10:  DTLS connection not found forAP  10:11:48:97 (56609), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:46.050: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:99:7c:bc:00 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:46.049: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:99:7c:bc:00:  DTLS connection not found forAP  10:11:48:35 (36884), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:45.959: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:45.954: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:45.691: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:26:99:71:35:f0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:45.645: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.93 from the operating system. ioctl operation failed

*Jan 17 02:59:45.645: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:98:50:14:50 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:45.620: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.81 from the operating system. ioctl operation failed

*Jan 17 02:59:45.403: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:45.392: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:45.378: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:45.316: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:99:85:ce:00:  DTLS connection not found forAP  10:11:48:105 (21107), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:07.959: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:2872 Max EAP identity request retries (3) exceeded for client 00:0c:e7:0a:a6:7a

*Jan 17 02:59:04.747: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:2872 Max EAP identity request retries (3) exceeded for client 7c:6d:62:73:3c:c2

*Jan 17 02:59:02.957: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:2872 Max EAP identity request retries (3) exceeded for client 00:0c:e7:0a:a6:7a

*Jan 17 03:00:07.431: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.96 from the operating system. ioctl operation failed

*Jan 17 03:00:06.840: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.84 from the operating system. ioctl operation failed

*Jan 17 03:00:03.608: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:98:cf:ae:40 entry in the database, could not process delete WLAN ALL MNs message

--More-- or (q)uit

*Jan 17 03:00:03.242: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.107 from the operating system. ioctl operation failed

*Jan 17 03:00:03.242: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:99:72:33:80 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 03:00:00.345: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.33 from the operating system. ioctl operation failed

*Jan 17 02:59:59.281: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:1f:9e:42:0c:d0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:59.195: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:3a:98:45:33:80 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:59.176: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.51 from the operating system. ioctl operation failed

*Jan 17 02:59:59.165: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:58.717: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:1f:9e:41:fb:10 entry in the database, could not process delete WLAN ALL MNs messag

*Jan 17 02:59:58.716: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:1f:9e:41:fb:10:  DTLS connection not found forAP  10:11:48:52 (41529), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:57.380: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.73 from the operating system. ioctl operation failed

*Jan 17 02:59:57.304: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:1f:9e:42:0c:b0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:56.361: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:55.824: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:1f:9e:80:e1:b0 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:55.755: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:53.043: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:3a:99:72:14:c0:  DTLS connection not found forAP  10:11:48:30 (40909), Controller: 10:11:48:27 (5246) send packet

*Jan 17 02:59:51.356: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.43 from the operating system. ioctl operation failed

*Jan 17 02:59:51.354: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

Previous message occurred 2 tim

*Jan 17 02:59:51.167: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:50.888: %LWAPP-3-AP_DB_ERR1: spam_apf.c:323 Unable to find AP 00:24:97:0e:1f:90 entry in the database, could not process delete WLAN ALL MNs message

*Jan 17 02:59:50.556: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:50.555: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5697 Invalid AP event (7) and state (4) combination

*Jan 17 02:59:49.960: %DTL-3-OSARP_DEL_FAILED: dtl_arp.c:1380 Unable to delete an ARP entry for 10.11.48.39 from the operating system. ioctl operation failed

--More-- or (q)uit

1 ACCEPTED SOLUTION

Accepted Solutions
New Member

Re: Error: Wireless Controller 5508 whit 68 ap

If connected as a LAG interface, from te switch the wlc is connected to shut and no shut the port channel interface.

If seen his in the past and this has occasionally fixed the issue. Otherwise reboot the WLC as per Scott's suggestion

HTH

Sent from Cisco Technical Support iPhone App

14 REPLIES
Hall of Fame Super Silver

Re: Error: Wireless Controller 5508 whit 68 ap

Roni,

Are you able to still access the WLC from Mayberry the service port or console at least?

Sent from Cisco Technical Support iPhone App

-Scott
*** Please rate helpful posts ***
New Member

Re: Error: Wireless Controller 5508 whit 68 ap

Hi Scott

Yes, when the problem occurs we can still access the console.

Hall of Fame Super Silver

Re: Error: Wireless Controller 5508 whit 68 ap

If you do a show ap summary and you still have zero ap's I would issue a save config then a 'y' and reboot the WLC.  YOu can look at the logs and see maybe what happened.

-Scott
*** Please rate helpful posts ***
New Member

Error: Wireless Controller 5508 whit 68 ap

any reason on why this happened? is there a bug related to this?

Hall of Fame Super Silver

Error: Wireless Controller 5508 whit 68 ap

I don't think it is a bug... I have clients running 5508's with more ap's and no problems with code 7.0.98.218, 7.0.116.0 & 7.0.220.0.  Just monitor the wlc and see if it happens again. Something else could of triggered it, but I don't know.

-Scott
*** Please rate helpful posts ***
New Member

Error: Wireless Controller 5508 whit 68 ap

Hi Scott,

But the device already went to this state several times and all access we can get is from the WLC console. and everytime this happens we have to reload the WLC.

the last error we have is the same as stated on the first query on this thread (%CAPWAP-3-DTLS_CONN_ERR),

another error that I see was %DTL-1-ARP_POISON_DETECTED

please see logs before reboot.

Message Log Severity Level ...................... ERROR

*Jan 26 06:04:23.630: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:1b:2b:36:75:50:  DTLS connection not found forAP  10:7:0:150 (50166), Controller: 172:107:80:3 (5246) send packet

*Jan 26 06:04:20.660: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:15:2b:05:b2:c0:  DTLS connection not found forAP  10:115:0:154 (9913), Controller: 172:107:80:3 (5246) send packet

*Jan 26 06:04:17.552: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5671 Invalid AP event (7) and state (4) combination

*Jan 26 06:04:16.279: %CAPWAP-3-DTLS_CONN_ERR: capwap_ac.c:679 00:15:2b:05:b6:a0:  DTLS connection not found forAP  10:115:0:151 (9934), Controller: 172:107:80:3 (5246) send packet

*Jan 26 06:04:15.153: %CAPWAP-3-INVALID_STATE_EVENT: capwap_ac_sm.c:5671 Invalid AP event (7) and state (4) combination

*Jan 26 06:04:08.545: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 06:02:36.943: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 06:01:14.346: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:24:d6:94:e9:6a

*Jan 26 06:01:13.527: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:24:d6:94:e9:6a

*Jan 26 06:01:05.942: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:59:45.540: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:2862 Max EAP identity request retries (3) exceeded for client 64:a7:69:e7:a7:e2

*Jan 26 05:59:34.540: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:58:44.156: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:1f:3c:2e:74:9d

*Jan 26 05:58:43.274: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:1f:3c:2e:74:9d

--More-- or (q)uit

*Jan 26 05:57:54.145: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:22:69:02:11:60

*Jan 26 05:57:53.539: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:57:53.231: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:22:69:02:11:60

*Jan 26 05:56:42.139: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M3 retransmissions exceeded for client 28:6a:ba:b7:c5:e3

*Jan 26 05:56:22.539: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:54:50.939: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:53:19.337: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:51:48.120: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:51:41.519: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M1 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:51:39.530: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:19:d2:bd:58:d2

*Jan 26 05:51:38.660: %DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:705 Received invalid EAPOL-key M2 msg in START state - invalid secure bit; len 40, key type 1, client 00:19:d2:bd:58:d2

*Jan 26 05:51:38.320: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M3 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:50:03.118: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:48:32.115: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:47:29.714: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M3 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:47:26.514: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M3 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:47:23.315: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M3 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:47:16.314: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M1 retransmissions exceeded for client 00:1f:3c:32:16:d0

*Jan 26 05:47:01.114: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:46:05.713: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M5 retransmissions exceeded for client 90:4c:e5:21:31:4c

*Jan 26 05:46:05.713: %DOT1X-3-MAX_EAPOL_KEY_RETRANS: 1x_ptsm.c:407 Max EAPOL-key M5 retransmissions exceeded for client 00:16:44:c7:7b:17

*Jan 26 05:46:02.747: %DOT1X-3-WPA_SEND_STATE_ERR: 1x_kxsm.c:1249 Unable to send EAPOL-key msg - invalid WPA state (0) - client 00:24:d7:86:ff:6c

*Jan 26 05:45:30.112: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:43:59.112: %DOT1X-3-MAX_EAP_RETRANS: 1x_ptsm.c:426 Max EAP retransmissions exceeded for client 00:24:d7:86:ff:6c

*Jan 26 05:43:54.377: %DTL-1-ARP_POISON_DETECTED: dtl_net.c:1394 STA(Target MAC Address) [00:1f:3c:2e:8b:16, 0.0.0.0] ARP (op ARP REQUEST) received with invalid SPA(Source IP Address) 172.107.107.34/TPA(Destination IP Address) 172.107.107.1

Hall of Fame Super Silver

Re: Error: Wireless Controller 5508 whit 68 ap

You’re better off opening up a TAC case so they can review your configuration on the WLC and your network. If it’s a hardware issue, then they will RMA your WLC.

-Scott
*** Please rate helpful posts ***
New Member

Error: Wireless Controller 5508 whit 68 ap

ok thanks scott.

Hall of Fame Super Silver

Re: Error: Wireless Controller 5508 whit 68 ap

If it’s taking down your production network, it’s always a good idea to get TAC involved.

-Scott
*** Please rate helpful posts ***
New Member

Re: Error: Wireless Controller 5508 whit 68 ap

If connected as a LAG interface, from te switch the wlc is connected to shut and no shut the port channel interface.

If seen his in the past and this has occasionally fixed the issue. Otherwise reboot the WLC as per Scott's suggestion

HTH

Sent from Cisco Technical Support iPhone App

New Member

Re: Error: Wireless Controller 5508 whit 68 ap

Hello, I persisted and apparently is a hardware failure.

New Member

Error: Wireless Controller 5508 whit 68 ap

Hi Roni,

Have you found what is the problem, because we have exactly same issue with the 5508 controller.

Thanks

Ali

Error: Wireless Controller 5508 whit 68 ap

Hi!

Did you find a solution for this, because a customer of mine has the same problem!

TAC was involved but he had no solution!

Does is make sense to change the hardware?

Did it help to shut/no shut the switchports?

Regards.

New Member

Error: Wireless Controller 5508 whit 68 ap

On our case I did upgrade the code and everything went fine (7.2.110.0).

Ali

2817
Views
0
Helpful
14
Replies