Wireless LAN Controller looping/flooding DHCP packets
We are facing a strange problem at a client's wireless solution.
Solution with 4x WLCs 5508, running v126.96.36.199, spread across 2x DCs. All 4x are active (no HA) with two mobility groups. The SSIDs are associated with a group interface (vlan select). All configurations are identical, being the IP addresses and hostname more or less the only configurations that change between them. The LAN is constructed pretty much the same way on both DCs and there are no STP loops (all interfaces forwarding).
Recently we activated two external DHCP servers. On 2x of the WLCs we have an issue: for each DHCP request packet from the clients (DHCP DISCOVER) thousands of packets arrive at the dhcp server. And I really mean thousands
The captures from the dhcp servers show that the “hops” field in the dhcp header is increasing every dozen or so packets which means that the packet is relayed more than once but the relay agent IP is always the same (VLAN interface @ WLC). Every time the DHCP hops is increased, there is the addition of one option 82 field so the packets are getting larger and larger until they achieve and impressive 1500 bytes.
The WLC / DHCP don’t cope with this rate and at a certain point the interface is marked as "dirty" because no answers are received from the dhcp server and this stops the “loop”.
A thorough search on bug tool and the web didn’t retrieve any relevant information.
Answering to your questions: DHCP proxy is enabled, pri and sec dhcp configured on all dyn interfaces, wlcs connect to c3750G-12S (soon to be replaced) and 7.3 will be upgraded soon.
It is my pleasure to announce that the root cause was found and a workaround was implemented.
For some unknown reason, the switch where both WLCs are connected (c3750G-12S) was timing out the entries in the CAM relating to the DHCP servers, even when receiving frames/packets from them.
When the wlcs generate dhcp packets to the dhcp server, they send it to the switch and the switch, without a known mac on the cam, was flooding the frame to all interfaces. The frames flooded to the uplink would reach the dhcp server eventually but some frames were injected back to the wlc, strangely enough. When receiving these frames, the wlc would process (adding another option 82 field and incrementing the “hops” field on the dhcp header) and relay them to the dhcp server, back to the switch which would feedback the whole process.
Static mac entries were configured for both dhcp servers’ macs pointing to the uplink
This switch will be replaced soon for a new c4503-E so no more investment on this issue
Any change on the dhcp servers’ mac addresses should be reflected on the switch configuration
Transferring Crash file from standby:
Login to the Active WLC in HA.
(Cisco Controller) >transfer upload datatype crash
(Cisco Controller) >transfer upload filename <Desired filename>
(Cisco Controller) >transfer up...
This is the start of a display filter cross reference between Wireshark and OmniPeek.
The 1st installment is a table of advanced filters. More filters will be added as time allows.
It is a living doc, so check back for changes every so often
Please feel ...
I have created a Powershell script to automatically add a Wireless Guest User on Cisco WLCs. (tested on 2500 Series)
The script should be completely self explanatory.
Powershell SNMP Module (Install-Module -Name SNMP)
SNMP Write Access to...