Hi Silju,
As I understand all your users are on the 3524 and they are a single VLAN and the servers are in a different VLAN and subnet, thus the users have to cross the 4908 switch to reach the servers....
If the servers were also on the same 3500 as the users, then port protection would have done the job, else if the users have to cross the 4908 switch to reach the servers then the restriction shall have to be by IP addresses only.
Pls revert.
Rgds