My initial thought is that that will wreak havoc on the PIX state table by introducing potentially assymetric routing.
Also, given two different ISPs, you defeat BGPs ability to provide optimal routing by using equal cost load balancing. If provider A has a one hop path to destination that is 5 hops via provider B, load balancing may send it the long way.