Hi,
yes a loaded VIP could affect performances, but not in a drastic manner. The jitter, etc can be in the order of few msec, should not be more.
Of course if you have a spare bay in VIP4 it would be good to place your loaded LAN interfaces into it. It is not just a matter of RAM,the VIP4 is much faster as CPU.
Hope this helps,pleaserate post if it does!