I just resolved a VPN connectivity issue using "ip tcp adjust-mss" as noted here: http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml
But I used this more on a hunch than on direct evidence from the router. The symptom was that the user could ping sites on the far end of the LAN-to-LAN tunnel but he could not load the sites in a browser. These "sites" were actually printer management site and likewise the use could not print to these printers. As I have seen similar in the past my hunch was MTU issue and that did the trick.
But it took a long time to resolve this as the user was a VIP and I could not get him to test scenarios in live time. I would just hear the complaints after he went home and things didn't work as expected. My question: how could I have used the SOHO router (871 in this case) to trouble-shoot without the need for the user to be home? Maybe there is just no test like the end user really trying to do their thing. But perhaps some thoughts on how to verify MTU will not be a problem over the VPN? Or a way to emulate downloading a full web page or other test?
"Could it be that my initial setting of 1452 actually made the situation worse?"
Unlikely. If starting with standard Ethernet, MSS is usually 1460 (1500 MTU less 20 bytes IP header and less 20 bytes TCP header). Your initial setting just wasn't small enough to account for all the IPSec overhead, which in the link you referenced is up to 58 bytes. I.e. maximum MSS should be about 1402. Smaller works, just not as optimal, which is why 1200 works.
Other than TCP, many other protocols don't often completely fill packets, which is probably the reason everything else works fine.
You can leave things as they are, and odds are, you'll see few in any problems. (You could also try increasing adjust-mss for sligthly better performance.) Or, you can also add IP MTU in addition to adjust-mss, or you can try to resolve the breakdown in PMTU. Even if you discover and fix the latter, adjust-mss is still good to use to avoid initial fragmentation in TCP.
"ip tcp adjust-mss" is really an efficiency option. If its application fixed an MTU issue, you have an MTU discovery processing problem. Other IP (non-TCP) packets can still be impacted. The latter, assuming the MTU discovery issue can be resolved, can be addressed using IP MTU before you get to your VPN tunnel. This isn't to say not to use "ip tcp adjust-mss".
Testing MTU should be possible using extended ping.