Yes, you certainly should be able to SSH directly to a context. The only difference with SSH'ing to a non-admin context is that you only get access to that context, you can't change contexts like you can if you SSH ito the admin context. The theory behind this is that ISP's can provide access for a customer directly into their context without fear that they could get into someone else's context.
As for why it's not working, try running "debug ssh" and see what it says.
