Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

WLC 2504 sudden network instability

Hello,

we're running a WLC 2504 with two SSIDs on it. It is connected to to a small PoE switch. Standard untagged vlan. A handfull APs connected to it. No DHCP, the APs have all static IP addresses.

All of a sudden we're having a number of issues with the network connection:

APs restarting

The APs restart every now and then reporting that their IP is being used by another device. Looking through the logs there are two MAC addresses that are reported as using the APs IP address. These two MAC addresses have unknown vendor IDs.

Warning: AP with Base Radio MAC f8:72:ea:7c:9d:e3 has found  its IP Address 0.2.146.0 being used by a machine with MAC

Address  04:c6:f8:40:00:00 (The other mac that is reported is 04:cc:90:40:00:00)

AP 'AP5', MAC: 0c:68:03:dd:1b:80 disassociated previously due to Link Failure.  Uptime: 4 days, 00 h 48 m 50 s . Reason: Capwap WTP Event request.

So: There are two MACs that use the IP addresses of 7 APs?!?! And there is no vendor to be found for these MACs?

Ping timouts on the webGUI and CLI

I have a ping running on the IP for managing the device. This is running fine for ages. As soon as I connect via webGUI or CLI I lose packets. Get timeouts etc. some packets get through some don't. More of the latter. So ping is fine but any other traffic seems to be impacted heavily.

What we have done for troubleshooting

Checked duplex/speed settings of the interfaces. Everything ok.

Connected to another switchport. Same.

Changed the IP address of the management port. Same.

Swapped places with a laptop with the same IP address --> Worked fine.

Plugged in a completely new device, installed the latest firmware (7.6) and uploaded the config from the other one. Same.

Restarted the default gateway for the subnet the controler is on.

So now we're at the end of our knowledge. It seems to be a non-physical network issue, but we're a small team and no one has changed anything they say :-/

Any ideas what we could check next?

Kat

6 REPLIES
VIP Purple

WLC 2504 sudden network instability

HI Kat,

First of all :

Warning: AP with Base Radio MAC f8:72:ea:7c:9d:e3 has found  its IP Address 0.2.146.0 being used by a machine with MAC

Address  04:c6:f8:40:00:00 (The other mac that is reported is 04:cc:90:40:00:00)

This error shows that other machine with mention mac address used this IP. The other machine may have manually configured the NIC.  Track down that host and take it off the network.

Please paste more logs from AP to know the root cause.

Regards

Hall of Fame Super Silver

WLC 2504 sudden network instability

One thing to be aware is that even if you have a static address on the AP. if it reboots and can't find the WLC, it will revert to DHCP.  If you don't have dhcp for the subnet the AP's are connected on, the will just fail.  This would mean that connectivity to the WLC broke and made these AP's fallback to DHCP.  In this state, your AP's would need to be rebooted and you need to make sure that they join the WLC.  It's almost better to add a dhcp scope and use mac reservations so that these AP's can get the same address all the time. 

Maybe you need to post your show run-config and your switchport config so we can verify if your configuration is okay.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

-Scott
*** Please rate helpful posts ***
New Member

WLC 2504 sudden network instability

Hello,

thanks for your suggestions. It's hard to find those two MAC addresses. As they seem to be virtual I cannot get a hint from the vendor ID. A show mac-address table on the switch the WLC is connected to doesn't show those two

I found an error in the WLC AP config. AP1 had the same IP as AP5 and a wrong netmask. I changed that. Unfortunately that doesn't solve our problem.

Here are some more messages from the WLC's log:

AP 'AP3', MAC: 0c:68:03:dd:34:00 disassociated previously due to Link Failure.  Uptime: 4 days, 15 h 04 m 15 s . Reason: Capwap WTP Event request.

AP Disassociated. Base Radio MAC:0c:68:03:dd:34:00

AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:0c:68:03:dd:34:00  Cause=Heartbeat Timeout Status:NA

AP 'AP3', MAC: 0c:68:03:dd:34:00 disassociated previously due to Link Failure.  Uptime: 4 days, 15 h 00 m 45 s . Reason: Capwap WTP Event request.

RF Manager updated TxPower for Base Radio MAC: 0c:68:03:dd:34:00 and slotNo: 0.  New Tx Power is: 2

AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:0c:68:03:dd:16:e0  Cause=Max Retransmission Status:NA

IDS Signature attack detected. Signature Type: Standard, Name: Deauth flood,  Description: Deauthentication flood, Track: per-signature, Detecting AP Name:  AP7, Radio Type: 802.11b/g, Preced: 9, Hits: 500, Channel: 6, srcMac:  C2:9F:DB:21:47:60

This is the sh run-config of our WLC including one AP:

>show run-config

System Inventory

NAME: "Chassis"    , DESCR: "Cisco 2500 Series Wireless LAN Controller"

PID: AIR-CT2504-K9,  VID: V01,  SN: PSZ17381EPZ

Burned-in MAC Address............................ 50:17:FF:27:12:80

Maximum number of APs supported.................. 15

System Information

Manufacturer's Name.............................. Cisco Systems Inc.

Product Name..................................... Cisco Controller

Product Version.................................. 7.4.110.0

Bootloader Version............................... 1.0.18

Field Recovery Image Version..................... 1.0.0

Firmware Version................................. PIC 16.0

Build Type....................................... DATA + WPS

System Name...................................... UK-BRI-WFAPC

System Location..................................

System Contact...................................

System ObjectID.................................. 1.3.6.1.4.1.9.1.1279

IP Address....................................... 172.17.128.12

Last Reset....................................... Power on reset

System Up Time................................... 4 days 0 hrs 46 mins 6 secs

System Timezone Location.........................

System Stats Realtime Interval................... 5

System Stats Normal Interval..................... 180

Configured Country............................... GB  - United Kingdom

Operating Environment............................ Commercial (0 to 40 C)

Internal Temp Alarm Limits....................... 0 to 65 C

Internal Temperature............................. +20 C

External Temperature............................. +25 C

Fan Status....................................... 4000 rpm

State of 802.11b Network......................... Enabled

State of 802.11a Network......................... Enabled

Number of WLANs.................................. 3

Number of Active Clients......................... 6

Memory Current Usage............................. Unknown

Memory Average Usage............................. Unknown

CPU Current Usage................................ Unknown

CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 50:17:FF:27:12:80

Maximum number of APs supported.................. 15

AP Bundle Information

Primary AP Image    Size

----------------    ----

ap1g2            9568

ap3g1            11288

ap3g2            11196

ap801            7164

ap802            8568

c1130            5072

c1140            9416

c1250            6944

c1520            8044

c602i            3736

Secondary AP Image    Size

------------------    ----

ap3g1            5792

ap801            5192

ap802            5232

c1100            3084

c1130            4964

c1140            4992

c1200            3364

c1240            4812

c1250            5504

c1310            3136

c1520            6404

c3201            4324

c602i            3716

Switch Configuration

802.3x Flow Control Mode......................... Disable

FIPS prerequisite features....................... Disabled

secret obfuscation............................... Enabled

Strong Password Check Features:

     case-check ...........Enabled

     consecutive-check ....Enabled

     default-check .......Enabled

     username-check ......Enabled

Network Information

RF-Network Name............................. RFGROUP

Web Mode.................................... Disable

Secure Web Mode............................. Enable

Secure Web Mode Cipher-Option High.......... Disable

Secure Web Mode Cipher-Option SSLv2......... Disable

Secure Web Mode RC4 Cipher Preference....... Disable

OCSP........................................ Disabled

OCSP responder URL..........................

Secure Shell (ssh).......................... Enable

Telnet...................................... Disable

Ethernet Multicast Forwarding............... Disable

Ethernet Broadcast Forwarding............... Disable

AP Multicast/Broadcast Mode................. Multicast   Address : 0.0.0.0

IGMP snooping............................... Disabled

IGMP timeout................................ 60 seconds

IGMP Query Interval......................... 20 seconds

MLD snooping................................ Disabled

MLD timeout................................. 60 seconds

MLD query interval.......................... 20 seconds

User Idle Timeout........................... 300 seconds

ARP Idle Timeout............................ 300 seconds

Cisco AP Default Master..................... Disable

AP Join Priority............................ Disable

Mgmt Via Wireless Interface................. Disable

Mgmt Via Dynamic Interface.................. Disable

Bridge MAC filter Config.................... Enable

Bridge Security Mode........................ EAP

Mesh Full Sector DFS........................ Enable

AP Fallback ................................ Enable

Web Auth CMCC Support ...................... Disabled

Web Auth Redirect Ports .................... 80

Web Auth Proxy Redirect  ................... Disable

Web Auth Captive-Bypass   .................. Disable

Web Auth Secure Web  ....................... Enable

Fast SSID Change ........................... Disabled

AP Discovery - NAT IP Only ................. Enabled

IP/MAC Addr Binding Check .................. Enabled

CCX-lite status ............................ Disable

oeap-600 dual-rlan-ports ................... Disable

oeap-600 local-network ..................... Enable

mDNS snooping............................... Disabled

mDNS Query Interval......................... 15 minutes

Port Summary

           STP   Admin   Physical   Physical   Link   Link

Pr  Type   Stat   Mode     Mode      Status   Status  Trap     POE 

-- ------- ---- ------- ---------- ---------- ------ ------- -------

1  Normal  Forw Enable  Auto       1000 Full  Up     Enable  N/A    

2  Normal  Forw Enable  Auto       100 Full   Up     Enable  N/A    

3  Normal  Forw Enable  Auto       1000 Full  Up     Enable  Enable  (Power Off)

4  Normal  Disa Enable  Auto       Auto       Down   Enable  Enable  (Power Off)

AP Summary

Number of APs.................................... 7

Global AP User Name.............................. Not Configured

Global AP Dot1x User Name........................ Not Configured

AP Name             Slots  AP Model              Ethernet MAC       Location          Port  Country  Priority

------------------  -----  --------------------  -----------------  ----------------  ----  -------  ------

AP7                  2     AIR-CAP1602I-E-K9     f8:72:ea:e4:9a:81  default location  1        GB       1

AP1                  2     AIR-CAP1602I-E-K9     f8:72:ea:7c:9d:e3  default location  1        GB       1

AP3                  2     AIR-CAP1602I-E-K9     f8:72:ea:e4:9c:57  default location  1        GB       1

AP6                  2     AIR-CAP1602I-E-K9     f8:72:ea:e4:9a:90  default location  1        GB       1

AP2                  2     AIR-CAP1602I-E-K9     f8:72:ea:7c:9b:63  default location  1        GB       1

AP4                  2     AIR-CAP1602I-E-K9     f8:72:ea:e4:9a:9b  default location  1        GB       1

AP5                  2     AIR-CAP1602I-E-K9     f8:72:ea:e4:9a:cb  default location  1        GB       1

AP Tcp-Mss-Adjust Info

AP Name              TCP State  MSS Size

------------------   --------   -------

AP7                  disabled   -

AP1                  disabled   -

AP3                  disabled   -

AP6                  disabled   -

AP2                  disabled   -

AP4                  disabled   -

AP5                  disabled   -

AP Location

Total Number of AP Groups........................ 0   

Site Name........................................ default-group

Site Description.................................

NAS-identifier................................... UK-BRI-WFAPC

AP Operating Class............................... Not-configured

RF Profile

----------

2.4 GHz band.....................................

5 GHz band.......................................

WLAN ID          Interface          Network Admission Control          Radio Policy

-------          -----------        --------------------------         ------------

1               corporate            Disabled                          None

2               dirtynetwork         Disabled                          None

3               dirtynetwork         Disabled                          None

AP Name             Slots  AP Model             Ethernet MAC       Location          Port  Country  Priority

------------------  -----  -------------------  -----------------  ----------------  ----  -------  --------

AP7                  2     AIR-CAP1602I-E-K9    f8:72:ea:e4:9a:81  default location  1     GB       1

AP1                  2     AIR-CAP1602I-E-K9    f8:72:ea:7c:9d:e3  default location  1     GB       1

AP3                  2     AIR-CAP1602I-E-K9    f8:72:ea:e4:9c:57  default location  1     GB       1

AP6                  2     AIR-CAP1602I-E-K9    f8:72:ea:e4:9a:90  default location  1     GB       1

AP2                  2     AIR-CAP1602I-E-K9    f8:72:ea:7c:9b:63  default location  1     GB       1

AP4                  2     AIR-CAP1602I-E-K9    f8:

RF Profile

Number of RF Profiles............................ 0

Out Of Box State................................. Disabled

RF Profile Name                    Band     Description                         11n-client-only

---------------------------------  -------  ----------------------------------- ---------------

AP Config

Cisco AP Identifier.............................. 15

Cisco AP Name.................................... AP7

Country code..................................... GB  - United Kingdom

Regulatory Domain allowed by Country............. 802.11bg:-E     802.11a:-E

AP Country code..................

................ GB  - United Kingdom

AP Regulatory Domain............................. -E

Switch Port Number .............................. 1

MAC Address...................................... f8:72:ea:e4:9a:81

IP Address Configuration......................... Static IP assigned

IP Address....................................... 172.17.128.24

IP NetMask....................................... 255.255.128.0

Gateway IP Addr.................................. 172.17.128.1

Domain...............

............................

Name Server......................................

NAT External IP Address.......................... None

CAPWAP Path MTU.................................. 1485

Telnet State..................................... Disabled

Ssh State........................................ Disabled

Cisco AP Location................................ default location

Cisco AP Floor Label............................. 0

Cisco AP Group Name.............................. default-group

Primary Cisco Switch Name........................

Primary Cisco Switch IP Address.................. Not Configured

Secondary Cisco Switch Name......................

Secondary Cisco Switch IP Address................ Not Configured

Tertiary Cisco Switch Name.......................

Tertiary Cisco Switch IP Address................. Not Configured

Administrative State ............................ ADMIN_ENABLED

Operation State ....

............................. REGISTERED

Mirroring Mode .................................. Disabled

AP Mode ......................................... Local

Public Safety ................................... Disabled

AP SubMode ...................................... Not Configured

Remote AP Debug ................................. Disabled

Logging trap severity level ..................... informational

Logging syslog facility ..

....................... kern

S/W  Version .................................... 7.4.110.0

Boot  Version ................................... 15.2.2.0

Mini IOS Version ................................ 7.4.1.37

Stats Reporting Period .......................... 180

Stats Collection Mode ........................... normal

LED State........................................ 

Enabled

PoE Pre-Standard Switch.......................... Disabled

PoE Power Injector MAC Addr...................... Disabled

Power Type/Mode.................................. Power injector / Normal mode

Number Of Slots.................................. 2

AP Model......................................... AIR-CAP1602I-E-K9  

AP Image...............................

.......... C1600-K9W8-M

IOS Version...................................... 15.2(2)JB2$

Reset Button..................................... Enabled

AP Serial Number................................. FGL1725W7F7

AP Certificate Type.............................. Manufacture Installed

AP User Mode..................................... AUTOMATIC

AP User Name..................................... Not Configured

AP Dot1x User Mode............................... Not Configured

AP Dot1x User Name............................... Not Configured

Cisco AP system loggi

ng host..................... 255.255.255.255

AP Up Time....................................... 3 days, 23 h 26 m 50 s

AP LWAPP Up Time................................. 0 days, 00 h 14 m 12 s

Join Date and Time............................... Tue Jan 28 18:11:43 2014

Join Taken Time.................................. 0 days, 00 h 11 m 41 s

Attributes for Slot  0

    Radio Type................................... RADIO_TYPE_80211n-2.4

    Administrative State ........................ ADMIN_ENABLED

    Operation State ............................. UP

    Radio Role .................................. ACCESS

    Radio Mode .................................. Local

    CellId ...................................... 0

    Station Configuration

      Configuration ............................. AU

TOMATIC

      Number Of WLANs ........................... 3

      Medium Occupancy Limit .................... 100

      CFP Period ................................ 4

      CFP MaxDuration ........................... 60

      BSSID ..................................... 0c:68:03:dd:16:e0

      Operation Rate Set

        1000 Kilo Bits........................... MANDATORY

        2000 Kilo Bits........................... MANDATORY

        5500 Kilo Bits........................... MANDATORY

        11000 Kilo Bits.......................... MANDATORY

        6000 Kilo Bits........................... SUPPORTED

        9000 Kilo Bits........................... SUPPORTED

        12000 Kilo Bits.......................... SUPPORTED

        18000 Kilo Bits.......................... SUPPORTED

        24000 Kilo Bits.......................... SUPPORTED

36000 Kilo Bits.......................... SUPPORTED

        48000 Kilo Bits.......................... SUPPORTED

        54000 Kilo Bits.......................... SUPPORTED

      MCS Set

        MCS 0.................................... SUPPORTED

        MCS 1.................................... SUPPORTED

        MCS 2.................................... SUPPORTED

        MCS 3.................................... SUPPORTED

        MCS 4.................................... SUPPORTED

        MCS 5.................................... SUPPORTED

        MCS 6.................................... SUPPORTED

        MCS 7.................................... SUPPORTED

        MCS 8.................................... SUPPORTED

        MCS 9.................................... SUPPORTED

        MCS 10................................... SUPPORTED

        MCS 11................................... SUPPORTED

        MCS 12..

................................. SUPPORTED

        MCS 13................................... SUPPORTED

        MCS 14................................... SUPPORTED

        MCS 15................................... SUPPORTED

        MCS 16................................... DISABLED

        MCS 17................................... DISABLED

        MCS 18................................... DISABLED

        MCS 19................................... DISABLED

        MCS 20................................... DISABLED

        MCS 21................................... DISABLED

        MCS 22................................... DISABLED

        MCS 23................................... DISABLED

      Beacon Period ............................. 100

      Fragmentation Threshold ................... 2346

      Multi Domain Capability Implemented ....... TRUE

      Multi Domain Capability Enabled ........... TRUE

      Country String ............................ GB

    Multi Domain Capability

      Configuration ............................. AUTOMATIC

      First Chan Num ............................ 1

      Number Of Channels ........................ 13

    MAC Operation Parameters

      Configuration ............................. AUTOMATIC

      Fragmentation Threshold ................... 2346

      Packet Retry Limit ........................ 64

    Tx Power

      Num Of Supported Power Levels ............. 4

      Tx Power Level 1 .......................... 16 dBm

      Tx Power Level 2 .......................... 13 dBm

      Tx Power Level 3 .......................... 10 dBm

      Tx Power Level 4 .......................... 7 dBm

      Tx Power Configuration .................... AUTOMATIC

      Current Tx Power Level .................... 3

      Tx Power Assigned By ...................... DTPC

    Phy OFDM parameters

      Configuration ............................. AUTOMATIC

      Current Channel ........................... 6

      Channel Assigned By ....................... DCA

      Extension Channel ......................... NONE

      Channel Width.............................. 20 Mhz

      Allowed Channel List....................... 1,2,3,4,5,6,7,8,9,10,11,12,

        ......................................... 13

      TI Threshold .............................. -50

      Legacy Tx Beamforming Configuration ....... CUSTOMIZED

      Legacy Tx Beamforming ..................... ENABLED

      Antenna Type............................... INTERNAL_ANTENNA

      Internal Antenna Gain (in .5 dBi units).... 8

      Diversity.................................. DIVERSITY_ENABLED

      802.11n Antennas

         A....................................... ENABLED

         B....................................... ENABLED

         C....................................... ENABLED

    Performance Profile Parameters

      Configuration ............................. AUTOMATIC

      Interference threshold..................... 10 %

      Noise threshold............................  -70 dBm

      RF utilization threshold................... 80 %

      Data-rate threshold........................ 1000000 bps

      Client threshold........................... 12 clients

      Coverage SNR threshold..................... 12 dB

      Coverage exception level................... 25 %

      Client minimum exception level............. 3 clients

    Rogue Containment Information

    Containment Count............................

    CleanAir Management Information

        CleanAir Capable......................... No

    Radio Extended Configurations

      Beacon period.............................. 100 milliseconds

      Beacon range............................... AUTO

      Multicast buffer........................... AUTO

      Multicast data-rate........................ AUTO

      RX SOP threshold........................... AUTO

      CCA threshold.............................. AUTO

VIP Purple

WLC 2504 sudden network instability

HI Katja,

I will suggest to craete a dhcp pool on Swicth.

Example:

vlan 80 is the management vlan in this example.

ip dhcp excluded-address 10.xx.80.1 10.xx.80.100

ip dhcp excluded-address 10.xx.80.115 10.xx.80.254

!

ip dhcp pool WirelessLAN

   network 10.xx.80.0 255.255.255.0

   default-router 10.xx.80.254

   option 43 ip 10.xx.80.1 --->WLC IP

   lease 3

Where AP connected:

interface FatEthernet1/xx

switchport access vlan 80

switchport mode access

Regards

New Member

WLC 2504 sudden network instability

After more days of troubleshooting I found a reverse dns entry for one of the WLCs interfaces, deleted it. Also two APs had the same IP. Since no one had touched the device since its installation in October, this issue must have existed ever since.

So I got rid of the log messages with the IP conflict. Still the WLC was losing at least 50% of its packets resulting in APs restarting every now and then. The problem is somewhere in the transition between our two subnets, so I swapped IP addresses of the management interface including APs and the interface into our LAN. Works fine now as a workaround.

Thanks for your help.

Katja

VIP Purple

WLC 2504 sudden network instability

Glad that you finally resolved the issue.

But I will recommand you to use DHCP pool on swicth for APs.

Please rate helpful posts accordingly.

Regards

1351
Views
0
Helpful
6
Replies