vWLC occasionally losing connection to ISE

[Kristian Aasen]

Hi guys

Weird problem.

I have a vWLC (7.5.102.0) set up with a ISE 1.2 cu3 as a RADIUS server. Both running on VMware 5.1 with Nexus 1000v.

I'm running 802.1x with a machine certificate.

Now the ISE and WLC is on the same subnet. No FW between.

Everything is working as planned, but every hour/hour-and-a-half I get this in my WLC log.

129	Mon Nov 25 20:50:36 2013	RADIUS auth-server 10.47.100.199:1812 available
130	Mon Nov 25 20:50:28 2013	RADIUS  server 10.47.100.199:1812 failed to respond to request (ID 234) for  client 6c:88:14:b9:72:fc / user 'host/PFPCNOLAP0131.protector.local'
131	Mon Nov 25 20:50:21 2013	RADIUS  server 10.47.100.199:1812 failed to respond to request (ID 233) for  client 6c:88:14:b9:72:fc / user 'host/PFPCNOLAP0131.protector.local'
132	Mon Nov 25 20:49:51 2013	RADIUS auth-server 10.47.100.199:1812 unavailable

This trigger a bucketload of problems and no clients are able to authenticate/re-authenticate.

vWLC is on a std Trunk port.

ISE on a access port.

I've tried downgrading the WLC, but nogo.

Any idea where I should start looking?

Regards

Kristian

[Nicholas Poole]

I have the same situation with a 5508 on 7.5.102 and ISE VM running 1.2 patch 6.  (both devices mgmt IP is on same VLAN)

Did you get anywhere troubleshooting this?

All I could do is disable and enable the RADIUS server in the WLC to get it working.  I dont know if it is a WLC problem and/or an ISE problem.

[radu.ioncu]

I'm seeing the same issue with my setup - virtual ISE + vWLC in the same subnet

Have you found a cause for this behaviour?

[Kristian Aasen]

Hi guys

As of now the solution is stable (except for the guest network running through the ISE, dunno if it's due to the same thing).

So what "solved" it for us is as follows.

Now, I'm not a ISE guy so go gentile on me...

The ISE on VMware has 2 NICs, 1 mgmt and 1 virtual (dummy) card. Now no traffic goes through the dummy one.

However, if the dummy nic was online, we had the issues. Then he disabled it, the problem stopped.

Why? No idea...

I'm now running the latest vWLC and ISE path 5, but will be upgrading to latest patch next week.

As for the guest network, guest are disconnected and reconnected avery 3-5 minutes. If anyone hade the same issues, please let me know.

Kristian

[radu.ioncu]

Hi,

First off don't upgrade to patch 7 just yet, there are some bugs with the sponsor/guest portal right now.

For the guest user disconnection, you should check the session timeout (per-wlan) and the idle timeout (global setting) on the WLC. Unless the guest account expires, ISE shouldn't send a CoA to the controller.

[Kyle Nielsen]

The 7.5 code can be quite problematic in general, I would suggest moving to 7.6 if you can. 

 What is your server timeout set to under the RADIUS server on ISE? (Default is 2 seconds, but i have seen cases like this when TAC had advised to jump that to 10)

[radu.ioncu]

I am running version 7.4.121, should be the most stable right now. Putting ISE and the vWLC into separate subnets seems to solve the issue completely.

I have tried various server timeout values up to 20 seconds, I usually get the same result. The loss of communication occurs when an endpoint errors out with "5434 Endpoint conducted several failed authentications of the same scenario" - this only happens when the two are in the same subnet.

[routerhand99]

Using a vWLC  - experiencing the same problem with RADIUS authentication.  Have upgraded in stages to version  7.6.130.0.   I think we need to concentrate on WLC configuration and possible code problems. The symptoms are the same "available - unavailable messages" between a RADIUS pair (not ISE systems).  It is like the WLC shuns both RADIUS boxes.

It would not be unreasonable to suggest that this is a new problem introduced in the vWLC code as these servers are working just fine with two physical 5508 WLCs during periods when the virtual WLC starts flipping.  This is a pretty serious problem when it happens. The 7.6 line of code so far - has the same problem.

[routerhand99]

It looks like there is an Open Caveat (BUG) in 7.6.130.0. 

CSCun62368: RADIUS NAC Client auth issues for 7.6. 

"RADIUS NAC Client auth issues for 7.6"

***************************************************************************************************************

And another: CCSCun18315

"RADIUS server anomalies with controller. When the primary RADIUS server fails, the secondary or tertiary controllers fail within 2 seconds."