Are you running the FWSM in single or multiple context mode?
Sounds like you have reached the ACL limit set by your resource partition.
If running in multi context mode by default 12 partition are configured, and depending on your software version, you will have a maximum of approx 11000 ACLs you can configure.
When using the object group depending on the number of src, dst addresses and services you could be adding a large number of rules, hence why it works when just adding the single port instead of the OG.
You can re-partition the firewall to increase the number of resources available per partition. This does require a reboot to take effect.
Using the 'resource acl-artition ' command.
Reducing to 8 partitions for instance would increase the acls limit to approx 20k.
If you running v4.x code you can now manually adjust the number of ACLs per partition, without re-paritioning the firewall.
HTH
Stu