Another strange one I am throwing out to the forum. Basically I have a 5 node deployment (1 x Primary Admin, 1 x Primary Monitoring, 1 x Secondary Admin/Monitoring and 2 x Policy Nodes). The primary authentication method is EAP-TLS or PEAP for wireless only. The deployment in question has been in pilot for about 3 weeks with no issues what so ever.
As of this morning we rolled into production and all seemed well - about 100 users successfully authed against PSN1 (PSN2 is configured in the WLC as a secondary radius). About 30 minutes after the production rollout authentications began failing for the exact same reason (see attached radius log). I checked all of the certificates as recommended in the log but this was a matter of course in that everything is as it should be.
My next step was to essentially stop PSN1 (application stop ise) to see if the issue was a problem on the second PSN. All authentications were now succeeding via PSN2. I left it this way for 30 minutes with no drama. I started PSN1 again and authentications began to work....20 minutes later the issue was back. I replicated this issue again to be sure.
At this point I decided to deregister PSN1 and application reset the node before rejoining with the ISE deployment. Authentications worked well until about 30 minutes later when the issue reappeared. At this point I reloaded all nodes in the ISE deployment to see if this made a difference but the issue still remained.
Currently I have PSN1 shutdown and all is functioning well - anyone have any ideas??