TCP Resets are sent out the sniffing interface of the sensor to each end host in the connection, NOT to your router or PIX. Your router software version has noting to do with TCP RST's not working. In fact they may well be working, it's just the P2P app is rebuilding the connection again. In fact a lot of P2P apps will use UDP for file transfers, so TCP RST obviously won't help you in this case.
To really confirm that TCP Reset is working first off make sure the particular signature is actually firing off alerts when P2P traffic is seen. Then enable TCP RST on that signature, and always remember that the RST is sent out the sniffing interface of the sensor, so if you have spanned a port on your switch to that sniffing interface you need to allow input packets on that span port, by default span ports are only outbound packets. Use the "help" on your span command to see the input packet options.
