I'm not aware of any particular incompatibilities between VCS and this firewall, but if the firewall actively sends RST to the source of H460 traffic, it definitely sounds like the H323 ALG is in fact not disabled (If this behavior only occurs for H460 traffic).
RST could also simply mean that the firewall does not allow outbound traffic between this source and destination address/port, but I assume that you have already checked that?
Assent and H460 uses different ports for H225/H245 so that might explain the different behavior.
Are you getting this error “Installer User Interface Mode Not Supported. The installer cannot run in this UI mode. To specify the interface mode, use the -i command-line option, followed by the UI mode identifier. The value UI mode identifiers...
The below trick might come handy when you have to add a new node to a cluster but you don't have or is unsure of the security password for the publisher. This procedure has been around for ages.
1) Login into the CLI of the Publisher.