T3 throughput

[dtrevino]

I have three sites, intermeshed using full T3's, 6506's w/720 sups, flexwans w/2ds3+ modules, using different providers.

In each case I can't get more than about 7mb session. I've tried several tools (ftp, ttcp, mgen) and they all stop at about 7mb. I can run 8 sessions in parallel and get the full 44mb, but not anything higher than 7mb for an individual session.

Except for the carriers, all three sites (26506's per site) are identical.

Did I miss something in the wan design?

[tbaranski]

What is the round trip latency between the sites? The "bandwidth X delay" product is usually what ends up limiting per-flow TCP performance over WAN links. See http://www.psc.edu/networking/perf_tune.html

For example, a transfer between two devices using standard 64KB TCP windows over a T3 with with an RTT of 50ms will max out at ~10Mbits.

[dtrevino]

My average RTT is 68ms and 53ms from my central site to my primary satellites.

We've come to the conclusion that it's a window scaling issue. TCP wasn't really designed to handle very large (30+ gbit) sessions.

We're now working with the programers to update the app.

What we don't know is, will the network support RFC 1323.

[tbaranski]

Usually you can turn on window scaling in the OS. Most if not all modern OS's support it. I'm not a software guy, so I don't know if OS's tend to even allow an application to control such things.

The network, however, doesn't care about window scaling -- it's just routing packets, and not concerning itself with the intricacies of TCP.

[jblalock1337]

I've had a similar problem with a pipe not being fully utilized....had 2 T1's load balancing with ip cef per-packet between two sites, however I hardly got anything over 1 mb transfer of any single session at a time. Turns out what it was is the nature of TCP..the packets were arriving out of order due to cef, and the other side thought it was dropping packets because one side was sending too fast so it sent those little requests to "slow down" your transmission speed.

Don't know your topology/configs exactly, but this sounds similar.