wlc5508 - dns answer packet is blocked

Florian Brenner · ‎09-21-2010

Hi,

I´ve changed about 60 access points (1242/1231) from standalone-mode to controller-base – mode. We installed and configured three 5508 controllers with firmware-release AIR-CT5500-K9-7-0-98-0.aes. All of them have the same configuration.

We have different clients (Symbol, Dlog) with 802.1x LEAP and EAP-Fast Authentication in use. After authentication they connect to a web-based service that runs on a server on the campus.

After the change to controller-based we have the following problem:

When the clients wake up from sleep mode they authenticate correctly (Ping to the webserver is o.k.), but the login webpage can not be displayed (searching...). Then I`ve sniffered the uplink of the first controller to see what happend. I find out, that the DNS-Requests from the DNS-Server do not achieve the wlan-clients and so the client tries to make the DNS-request several times but without success. This problem occurs on all clients (handhelds, laptops...). To temporarily solve the problem I have to disable/enable the radio-interfaces of the clients.

Then I`ve changed the mode from local to HREAP on one AP and make a local break out at the AP and sniffered the uplink of the HREAP- AP. I´ve repeated the tests serveral times and with HREAP everything is working correctly. The clients get the DNS answer packets and the page is displayed immediately.

So why does the controller block the DNS-Requests after sleep-mode and reauthenticate of the clients ? With release AIR-CT5500-K9-6-0-196-0.aes the same problem occurs.

Thanks

Greeting Flo

Federico Ziliotto · ‎10-12-2010

Hi Flo,

I'd personally suggest to collect two traces while keeping the SSID open (or eventually configure another temporary open SSID):

1. Wireless trace between the client and the AP.

2. Wired trace by spanning the etherchannel interface of the switch where the 5508 is connected:

(config)#monitor session 1 source interface <5508 port-channel> both
(config)#monitor session 1 dest interface encapsulation replicate

Then, according to the OS and the NIC of the machine connected to , you may need to configure some specific registry keys so to preserve the vlan tags:

http://wiki.wireshark.org/CaptureSetup/VLAN

For example, for Intel cards:
http://support.intel.com/support/network/sb/CS-005897.htm

I understand this might sound too demanding, however it would also be one of the safest ways to be 100% sure where the DNS traffic is blocked and why (maybe is being tagged on the wrong vlan).

If this would be a common issue on the 5508 HW/SW, we should have seen it in several deployments and more often, but everything should be clearer by looking at the full DNS traffic flow.

Regards,

Fede

Florian Brenner · ‎10-13-2010

Hi Fede,

thanks for your answer.

I have a few updates:

- All Wireless-Clients have static configured ip addresses

- I´ve temporarily configured a dhcp-scope and configured a few clients with dhcp. Then everything is working fine. After sleep mode the web page is displayed immediately.

- Then I`ve tested a 4404 controller with the same configuration => Everything is working fine, too

- I´ve attached two debug client - files, one of the 4404 and one of the 5508

On the 4404 controller, the client is registered with the correct ip address immediately, but on the 5508 controller the client is first registered with 0.0.0.0. Why does the controller perform the DHCP_REQD - state?

Is there any difference between the two controller platforms in handling clients with static ip addresses?

In the WLAN-Configuration the DHCP required option is deactivated.

There is only one vlan based on a HP-Network.

Greetings, Flo

Federico Ziliotto · ‎10-13-2010

Hi Flo,

I'd agree that the two client connections seem to be treated differently between the 4404 and the 5508.

From the 5508 debugs, the lines that got my attention are the following:

*Oct 07 11:36:21.304: 00:15:70:34:2a:b8 Orphan Packet from DS - IP 172.17.172.101
...

*Oct 07 11:36:22.742: 00:15:70:34:2a:b8 Installing Orphan Pkt IP address 172.17.172.101 for station
...

*Oct 07 11:36:22.742: 00:15:70:34:2a:b8 Assigning Address 172.17.172.101 to mobile

These seem to be the combination of two known defects.

1. CSCsq46427

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsq46427

Even if the symptoms are exactly the same as your case, this should be fixed in 7.0.98.0, until the next one was found.

2. CSCti83830

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti83830

This second one should basically be a regression of CSCsq46427, specific for 5508 platforms.

To confirm whether you may be hitting such a situation, you could test one of the following options:

a) Apply the same workaround as for CSCsq46427.

b) Enable passive client support (only on 7.0):

http://www.cisco.com/en/US/docs/wireless/controller/7.0/configuration/guide/c70wlan.html#wp1391015

Should all of this still not be enough, I would then recommend to open a TAC case if more advanced troubleshooting is needed:

http://tools.cisco.com/ServiceRequestTool/create/

Regards,

Fede

--

If this answers your question please mark the question as "answered" and rate it, so other users can easily find it.

Florian Brenner · ‎10-14-2010

Hi Fede,

I`ve already tested it with version 7.0.98.0, 6.0.196.0 and 6.0.199.4, but on every version I have the same problem. I`ve already tested the passive client feature on 7.0.98.0 without effect as well.

I don`t think that I have to search after a solution with passive clients, because when the clients wake up, they permanently make dns requests. The crazy thing is, that a ping to the webserver, which the web service is running, is o.k. Also the ping to the dns-server is ok. But the page isn`t displayed on the browser (searching for site....). So layer 2/3 is working correctly.

For example:

Client IP: 172.17.75.88 /16

DG: 172.17.0.1

DNS-Server: 172.17.22.23 /16

Webserver: 172.16.0.40 /16

DG: 172.16.0.1

I`ve already opened a TAC case and I have to send cisco some more sniffer-files...

So the next step will be to sniffer the whole way:

Client --------- AP -------------- WLC -------------- Core -------------- DNS

|

Webserver

For any other tips I would be very grateful.

Greetings, Flo

Federico Ziliotto · ‎10-14-2010

Hi Flo,

The next thing I would think of is to test with a pre-authentication ACL for the guest SSID, for all DHCP, DNS and traffic from/to the web server.

Apart from that, I would stick to the sniffer traces to understand the full flow.

Not to close the discussion, but if there is a TAC case open, I'd then rather let the engineer assigned to it keep working with you not to get in conflict with his action plan by giving you different tips and troubleshooting ideas.

Feel free to let me know how it ends ;-)

Regards,

Fede

--

If this answers your question please mark the question as "answered" and rate it, so other users can easily find it.

Florian Brenner · ‎10-14-2010

Hi Fede,

there is no guest-ssid configured. The clients are authenticated with 802.1x EAP/FAST and LEAP with dynamic WPA TKIP encryption and then they connect to a web-based service running on a server at the campus (for goods picking / scanning).

At the moment I can`t see an end, but when we have cracked it :-) I´ll let you know first!

Greetings, Flo

Federico Ziliotto · ‎10-14-2010

Thanks Flo,

Sorry for the previous typo (too many forums at once...;-), not related to any guest SSID in your case of course.

I'd simply focus on the traces and maybe save the effort to collect the wireless one: if the issue is related to DNS traffic you may not be able to see too much there. How about an open SSID? Does the issue still occur?

Regards,

Fede

Florian Brenner · ‎10-14-2010

Hi Fede,

with an open SSID I don`t have tested it until now, so I`ll do this next time.

Thank you

Federico Ziliotto · ‎10-14-2010

Hi Flo,

I would be a little bit surprised if it works with an open SSID. If still failing, that should at least allow you to read a wireless trace (at least more easily than with PSK).

Good luck with the rest!

Regards,

Fede

--

If this answers your question please mark the question as "answered" and rate it, so other users can easily find it.

a.frollano · ‎10-19-2010

Hello there! I'm experiencing the same DNS problem but with the Guest SSID. We have web authentication in place through a NAC guest server and some pre-authentication ACL. The DNS server is in our DMZ. After authenticating everything works fine then after a while I noticed that web pages don't respond correctly so I run a nslookup and I notice it times out! This is a big problem for VPN connected users or if you are downloading big files which interrupt and don't restart.

We are using version 6.0.199.4. So it looks like the controller is cutting out DNS traffic?