GRE Loadsharing over ADSL

[howard.cisco]

Having problems with increasing throughput between 2 sites joined by 2 ADSL based GRE tunnels. Tunnel 1 is over ADSL links from ISP 1, the other tunnel is over ADSL from ISP 2.

Using EIGRP and getting dual routes appearing in routing table (using variance etc)

If we shut 1 tunnel down we get the expected 250 kbit/s running ftp between 2 hosts (MS 2000 server)

When we open the other tunnel, we still get only 250 kbit/s even though the traffic is balanced nicely over the 2 tunnels.

Have tried many options such as MTU changes, setting DF flags.

Have noticed this:

For 1 tunnel up only

Interface running 250kbit/s at 18 Packets/sec

With both tunnels up

Interface running 250kbit/s at 32 Packets/sec

Ok the packet size has changed but it still doesn't get up to expected 500kbit/s

Using Cisco 1710 c1700-k9o3sy7-mz.122-8.T5.bin

Also tried newer versions

Not tried CEF

Any ideas

[ruwhite]

You're probably seeing the impact of out of order packets across the two links--you state you are using variance, which almost assures me they are unequal cost, and out of orders can be very severe on unequal cost links.

http://www.cisco.com/en/US/partner/tech/tk827/tk831/technologies_tech_note09186a0080094806.s

html

I would try CEF, which does per flow load sharing (source/destination address), and then try to see if you get ore throughput when you open two or three ftp sessions through the links, rather than one. If you do only one session, you're not going to see the second tunnel come into play at all with per flow load sharing.

:-)

Russ.W

[howard.cisco]

Thanks Russ

Problem is that the customer wants to copy files from one 2000 server to another 2000 server using the combined 512kbit/s. The drag and drop operation results in a single large MS files copy that is one flow.

Do you think the main problem is oo-sequenced packets. The characteristics of the loadsharing seems almost capped in some way to around 250 kbit/s with packet sizes halving with both tunnels up.

[mherald]

I believe the problem you are observings has to do with your load balancing methods. The GRE Tunnel (regardless what traverses the tunnel) counts as 1 conversation.

By default, your router will do per session load balancing. Where 1 conversation takes 1 path, the next conversation takes the next path.

It is possible that both of your tunnels come up and utilize just one circuit. Depending on what order packets leave your router.

What I believe you want is per packet load balancing, were 1 packet leaving the router goes via circuit a, the next packet going via circuit b?

First, your routing must be set up correctly. The tunnels must load balance per packet on the serial interfaces. that way traffic on any one tunnel will traverse both serial interfaces. Under the serial interfaces, look for a no ip route-cache command.

Then you must load balance via your routing protocol, static? OSPF, EIGRP? to use both tunnels for any given destination. Either one tunnel will use both serial interface, so its not that critical. Or you can further more do load balancing between the two tunnels (that is allready being load balanced across the same two interfaces) on a per packet basis as well, a tad redundant.

Now it is HIGHLY likely packets will arrive out of order. However that is what the sequence number is for in the IP packet.

I hope that helps,

Mike

[howard.cisco]

We are currently running per packet loadsharing with eigrp showing two routes to the remote LAN subnet

The tunnels are running over 2 ethernet interfaces to a pair of local ADSL routers at each site. When we either off-balance the routes by temp changing the bandwidth on 1 tunnel OR shutdown 1 tunnel we get 250kbits down the enabled tunnel. But when we bring up both tunnels we still get an aggregate of 250kbit/s transfer rate shared very accurately across both tunnels BUT the packet size has halved. We notice that the packet per second throughput has doubled but the Bit/s throughput is the same.

IE : 1 Tunnel, 250kbit/s @ 20.83 pps with packetsize 1500

2 tunnels, 250kbit/s total @ 41.66 pps with packetsize 850.

We just cannot push the combined rate upwards to 512kbit/s which is what we expect.

We are not seeing fragmentation counts (sh ip traffic)

We have tried setting the DF bit using a policy route map and played with the MTU which all leads to fragmentation

We have also set interface bandwidth to actual bandwidth available via each tunnel

My experience of oos packets is a drop of maybe 10-15% but not 50%

Our customer is not amused andwe have had to abandon similar solutions for other customers. I also noticed doing a search for "ADSL LOAD" on all forums that many others are experiencing such problems so maybe an investigation in the Lab by Cisco may help us all

Any ideas greatfully received

[ruwhite]

The problem is still out of order packets, form what you are describing. The traffic originators are probably reducing their TCP segment size in accordance with what gets dropped, and even a 10% to 15% drop rate due to out of orders can cause exactly what you are seeing--a 50% reduction in effeciency across the circuits. TCP is designed to do just what you are seeing--back off heavily--in the face of just about any drop rate, which is why out of orders and single drops have such a disproportionate impact on performance.

I'd simply not use per packet load sharing on these links.

:-)

Russ