The server can join an AD domain, but all LMS management operations must happen as a local Administrator.
No anti-virus software is certified with LMS. Internally, we have used McAfee and Norton, but CA should be fine. Just disable on-access scanning of the NMSROOT directory.
We've seen issues with SMS and LMS. We never tested it. If SMS stays out of LMS's way, and doesn't mess with the casuser account, it should be okay for you.
You may want to test everything without these services installed first to make sure LMS is working. Then, add them one at a time, and make sure things are still working as expected.