Help me please to undestand.
I have OOB mode on next equipment:
same hosts: winxp and win7 with nacagent 4.7.2
AD SSO on CAM started, ports on switch is controled.
On layer 3 network works fine, but nacagent on hosts don't started.
In your network.png screen capture you do not have "Enable L3 support" checked. You need this checked if the computer is one or more L3 hop away from the NAC server. Since you're running in OOB Real IP GW mode, I'm assuming this is the case. You'll just have to run through these steps
1. check the box
2. reboot the NAC server
3. try again
Could you provide a diagram of your scenario? That would help out a lot.