I'm working on an encryption delay problem with an ASA 5510. We have a server hosting a web service - when accessing that site directly over the internet from a specific location, we're seeing sub 100ms responses consistently. We then configured a site-to-site VPN tunnel to that location, and we sometimes see >5s responses. Looking through the logs, everything points to the encryption on the ASA. I checked resources, and there are no bandwidth problems (virtually nil bandwidth), CPU is consistenly less than 10% and memory usage is 130MB out of 256MB. Running version 8.2(2).
The ASA's only purpose is VPN tunnels, and it has 6 configured. Network-wise, both the ASA and the server in question are plugged into the same 2960 switch (which connects up to the core via a 2GB trunk). Looking at the 2960, I'm seeing no bandwidth or CPU issues there. I currently do not have any QoS or bandwidth policing policies defined on the ASA. The tunnel in question currently has the highest priority crypto map, so I am changing it to the lowest priority during the next maintenance window, but I can't imagine that would cause this delay. This is 100% data traffic (no voice), and they are very small messages (just a web service).
Any ideas as to what might be causing the encryption delays on the ASA? I'd expect some latency from IPSec, but from sub 100ms to >5s??? From what I can tell, it's not when the tunnel is being established - we're seeing these delays well after phase 1 and phase 2 have been negotiated. Any config changes I might be able to make to reduce that encryption delay? Would LLQ help here?