The reason you are getting this is either the certificate is not installed correctly on the ACS or you have validate server certificate on the client side, preventing the certificate to be used. Try to uncheck that in the client side.
there is an option of not using certs for peap, right? i do not want to use the cert for the authentication, but the cert is installed (generated) in the ACS. client side is disabled for getting the certs from the ACS..
PEAP like any EAP type, needs a certificate installed. I have tried to generate a certificate from ACS, but never got that to work. I got the same SSL error you got. Users have to obtain that cert form the ACS in order to continue with the authentication process.
The easiest thing to so is to obtain a cert for the ACS SE from an online CA. The one I always recommend is www.rapidssl.com as they are reasonably cheap and the whole order process takes about half an hour to work through. If you generate the CSR on the ACS, obtain your cert and install it you can leave the check boxes checked on your clients as the Rapidssl root cert is built into Windows/IE.
The only thing to be careful of is that before you generate the CSR, remove the existing self-signed cert from the ACS SE. Failure to do so can sometimes lead to problems.