Hey all, we've done a lot of back and forth between our Internet provider and the destination host network support team and have not been able to find a solution. By now it's pretty clear the "ghost" is downstream somewhere on the Internet but I just thought I throw this out here just for the kicks and see maybe someone can shed a light or two.
We have approx. 1700 retail store outlets nationwide, all equiped with debit/credit over IP that connect to Chase Paymentech over the web for each transaction over SSL. This has been setup for last 2-3 years and all has been working fine until last May. We occasionally get a store complaining they are no longer able to process their debits. When we perform a quick telnet to the debit host at paymentech site over port 443, usually we see an Open which confirms there is connectivity. However during the reported outage this no longer works. We have to then route them via their DMVPN tunnel back to our data center in Nova Scotia and out to the Internet from there. It works. Few days later, same store again complains about their debits outage, so we go in and simply remove the static route we put in and transactions once again start flowing right from the store to the host (split tunnel). Back in May we had approx. 80 stores all in the same province who reported this issue (on a weekend) and we had to this manually for all these stores. We've gone back and forth with the provider who has contacted a bunch of other circuit providers down stream but no one has been able to find a definitive answer. Any thoughts?
