ASR HTTP inspection slowing up long HTTP downloads?

b.julin · ‎06-25-2010

Anyone seen this happen on an ASR?

If we inspect HTTP traffic as HTTP, then short (<100K) sessions seem to work just dandy. But longer HTTP downloads slow to a complete crawl -- 5KBps or less.

If we inspect HTTP traffic as TCP, or just pass it, then it's all normal again.

This happens reguardless of the amount of congestion on the network and whether the ASR is extremely lightly loaded (10-20 sessions), or carrying thousands of sessions.

I'm think latency and TCP windowing. This is version 2.5.0 on a 1002 (non-fixed chassis).

Yeah I know, ASR, what's that? Here's wishing you all a huge equipment budget so I'm not alone here on the bleeding edge trying to do small boy stuff with big boy equipment much longer :-)

Panos Kampanakis · ‎06-25-2010

Can you take a capture at the end host that is accessing the long url?

Please check to see if the slowness is introduced due to long URL HTTP GETs (drops) or drops within the HTTP data.

Then I would suggest opening a case with TAC. They need to look into the ASR ZBF design as the platform is different than traditional routers.

I hope it provides a little guidance on next steps.

PK

b.julin · ‎06-25-2010

Yeah that's what I just got back from doing. Looks like there are actually drops, not just latency, though the ACK-->packet latency seems mostly higher. Also the problem apparently is not completely going away with TCP inspection only, it is just greatly reduced compared to HTTP inspection.

I can't seem to find any commands to dig deeper into session statistics in this software load... every release there are tons of new features but some basic things remain missing. So I guess I'm going to have to wait till the week starts to get in there where I can take some inter-link captures and make sure it is definitely the ASR dropping traffic and not something else in the chain. But at least as long as there are packet drops I can do that... it would be harder to pull off if this were a latency-only problem.

Panos Kampanakis · ‎06-25-2010

Sometimes dup-acks can be normal in a WAN environment, depending on how many they are of course.

I understand that it can be a little tough with the ASRs. The main reason is the architecture and the packet processing being pushed lower close to the hardware (too general of a statement, I know).

Work with TAC on this, they will be able to dig deeper (troublehsooting commands etc) and work with you on the ASR, as it is not traditional in its ZBF functionality.

Good luck,

PK

b.julin · ‎06-28-2010

Here's an update. I'm still collecting data to create a TAC ticket, but it looks like there are two problems now.

First, it looks like when inspection is tcp, or when there is no inspection, the ASR does something to the traffic that causes our ellacoya (Arbor) box to swat more packets than when the traffic goes through our ASA (which we are trying to replace.) That will cut a download that normally goes at 170K or so down to 20 or 30, but it's dependent on the subscriber contract.

The second problem is with http inspection, where there are NO packet drops occuring in the Arbor device at all -- packets are just arriving at the very slow rate of 2 per second. I have yet to find out what this scenario looks like on the outside of the ASR.

Does anyone have a URL to more detailed docs about ASR's http inspection procedure? The manuals aren't proving to be especially informative.