Yes, the interesting traffic should be mirrors of each other on either end. For instance...
Site A
access-list crypto extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
Site B
access-list crypto extended permit ip 192.168.2.0 255.255.255.0 192.168.1.0 255.255.255.0
There are 2 options for restricting the vpn traffic.
1. remove "sysopt connection permit-ipsec/vpn". This will force all ipsec traffic to be filtered in your regular interface access lists. So to allow traffic from site B to site A...
Site A
no sysopt connection permit-vpn
access-list outside_access_in extended permit tcp eq 3389
access-list outside_access_in extended permit tcp eq 21
access-list outside_access_in extended permit tcp eq 25
access-group outside_access_in in interface outside
Site B
no sysopt connection permit-vpn
access-list outside_access_in extended permit tcp eq 25
access-list outside_access_in extended permit tcp eq http
access-list outside_access_in extended permit tcp eq 3389
access-group outside_access_in in interface outside
2. Use the vpn-filter attribute in the tunnel group policy to restrict the traffic.
This example is for remote access vpn but also works for l2l.
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a0080641a52.shtml