We use Cisco ACE modules [build 3.0(0)A2(3.0)].
Is there a way to configure a sticky-group based on the clients mac address instead the ip?
ACE allows sticky based on Source IP, Http cookie, http content, http header, radius, sip header, rtsp header, and layer 4 payload data - however, not by mac address.
Question back to you - MAC sticky would only be useful if the ACE were in the same subnet as the client. In that case - Every client making a connection would in theory have a have a unique source IP. If a client was being natted, the nat device uses its own mac for each unique IP in the pool. I can't see where sticking based on a source MAC would be useful where sticky based on IP was not. What exactly is going on in your environment where this is a requirement?
Sidenote - ACE does have a feature named "mac-sticky enable" which is appliend under a vlan interface. The purpose of this is that ACE will send a SYN,ACK back to the source MAC that the SYN for that flow came from. Normally, ACE would look up the source IP that the SYN,ACK is destine to in the routing table and send it to the appropriate gateway/default route.