load balancing

petewa · ‎05-21-2007

I have 2 1750 routers back to back with two point to point T1's and OSPF running. Why would one of the T1's (either) (show interface S0 or S1)intermittantly show either no transmit or no receive? All stats in service-module look clean. See attachment Thanks for anyone's input. Pete

Edison Ortiz · ‎05-21-2007

What does the show ip route looks like on this router ?

Does it have multiple entries for each destination network ?

In addition to OSPF load balancing, I suggest enabling CEF and have per-packet load balance on the interfaces. By default, you will have per-destination load balance which may produce unequal circuit utilization.

petewa · ‎05-22-2007

Hi,I'll take a look at the ip route next time I get to the site (no remote access). Thanks for your suggestion on CEF. I think that's the road I was heading down!. You may of confirmed it for me. Thank you, Pete

Richard Burts · ‎05-22-2007

Peter

While CEF may have some advantage over Fast Switching, and therefore worth configuring, I would be very cautious about specifying per packet load balancing for your traffic. While it will produce more even load on the circuits, it will also increase the probability of out of order packets. For some traffic that is a good trade-off. For VOIP I believe that it is problematic.

HTH

Rick

HTH

Rick

Richard Burts · ‎05-21-2007

Peter

Without some specific information about your environment we must make some assumptions. You tell us that there are 2 T1s and OSPF so I will assume that there are 2 equal cost paths to all destinations on the other side of the link. I will also assume that you are running CEF on the router and that the default of per destination load balancing is running. (If any of my assumptions are not correct then please provide clarification). I also note that you have configured the measurement period down to 30 seconds.

We do not know specifics about the traffic flows that are going through the router and it might be helpful to know how many sources are sending traffic and how many destinations are receiving traffic. It would also be good to know if each packet transmitted 1 packet in response or whether there is an imbalance in packet count in some direction.

Having said all the above, I will take a guess at the explanation. I believe that with per destination load share that from time to time there is a 30 second interval where all traffic being transmitted is to a single destination address and therefore is sent over a single T1. In the example that you posted it seems that all transmitted traffic is over serial 1. In looking at the packet counts in this 30 second measurement window I note that the number of packets transmitted (354) is much larger than the number of packets received on that interface (216) and is slightly less than the combined total of packets received on both interfaces (462).

So I am not convinced that you have a real problem here. I believe that it is just the variability that you may get with a 30 second window of measurements and with per destination load balancing.

[edit] Edison makes an interesting point in suggesting that you configure per packet load balancing. With a small number of destinations he may be correct that per destination may create unbalanced loads. With a larger number of destinations it is likely to become much more even. And you should consider that when you configure per packet balancing that you increase the probability of out of order packets. Depending on the applications being run out of order packets may or may not be problematic. I had a colleague once who was working on performance problems with an application and brought up a second link and configured load balancing with per packet balancing. The performance of the application dramatically decreased with load sharing because out of order packets had a significant impact on this application.

HTH

Rick

HTH

Rick

petewa · ‎05-21-2007

Hi, Attached are the router configs. No, CEF is not running. Ah, the data that these routers pass is VoIP traffic from Nortel 9150 boxes. Lots of UDP and small packets. Interesting enough, This has been in for the last 4 years, virtually undisturbed, and now all of a sudden they are having voice quality issues. There is no QOS being passed. The only devices on this "network" are the 9150's. Nothing else. Thanks for your help, Pete

Richard Burts · ‎05-21-2007

Peter

Thanks for posting the configs. They are quite simple. You are correct that it does not appear to run CEF. The only real difference that makes in this discussion is that the way to achieve per packet balancing in this environment would be to force process switching and I strongly feel that you do not want to go there. And I believe that VOIP is an example of the applications that have real problems with out of order packets. So that is another reason not to try to do per packet sharing.

Without CEF these routers will default to fast switching which still gives you the default behavior of per destination load sharing. And I believe that my proposed explanation that within a given 30 seconds that there might be traffic to only a single destination is still a likely explanation.

I did notice that each of the serial interfaces is reporting some output drops:

Total output drops: 3759

VOIP does not like packet loss and I suggest that you might look into output drops as a possible source of the problems that they are experiencing.

One other observation: I am not sure that it matters much but I am puzzled why the serial interfaces are configured with:

ip ospf network broadcast

on a point to point link why would they be configured to act as broadcast. It would be interesting to see the output of show ip ospf interface.

HTH

Rick

HTH

Rick

Richard Burts · ‎05-21-2007

Peter

As I continue to think about your issue I suspect that with a small number of destination addresses that there will always be an amount of imbalance in the use of the T1 links. If you want to achieve more equal use of the T1s I believe that you might consider configuring the T1s with multilink. The multilink feature takes several physical links, creates a logical bundle which has a single IP addres. There is a certain amount of overhead in running multilink, but it would be the way to get really equal utilization of the T1s.

HTH

Rick

HTH

Rick

petewa · ‎05-22-2007

Thank you for you input. I do have a "future" multilink configuration that I'm going to install in the future. Right now, the "remote" side might as well be on the moon. It's in a high security area that takes an act of congress to physically access. So, I have to be very careful when changing anything, for if I lose communication with the far end, it's more then egg on my face! For the meantime, I'll program each side with CEF and see if that helps any. As far as the output drops, I'm puzzled about that. I guess I don't fully understand what it really means... Interestingly enough, at the HQ end, all equipment is full-duplex. At the remote end, the router e-net port is 100 full to a 2950 switch, and the rest of it is 10 half. This is due to a physical limitation of the remote end. From the remote end's 9150, I do see a lot of collisions. But with half-duplex, I suspect it's as good as it's gonna get.

Anyway, thanks to ALL!! Pete

Wilson Samuel · ‎05-22-2007

Hey,

Just a tip for all your future "Area 51" sort of Router / Switch configurations i.e. if you dread loosing communication after a config err to a remote site, just follow the basic steps:

1. Give the command 'reload in' with a suitable period of time e.g. 30 mins

2. If you DO NOT revoke this command in next 30 mins the router/switch WILL reload

Hence will come back with the old working config and you shall be able to access your device once again, provided you haven't given any 'wr me' command.

HTH,

Please rate if it was helpful,

Kind Regards,

Wilson samuel

petewa · ‎05-29-2007

Nice!... Thank you Wilson. I'm just a "novice" Cisco technician,... Yeah, the site is a tough one on us to get in and out, so this one is already on the printer and now in my bag of tricks. Thanks, Pete