Load balancing over FR not working

rhaskell · ‎10-04-2010

Hey gang! I'm having some issues with what would appear to be a straightforward config. I've got two frame relay T1s going into a 7204, and am trying to per-packet load balance across the two using static routes. The problem is that I can only get 1.6 Mb/s throughput max when trying to load balance, even though I can get 1.4 Mb/s throughput on each T1 individually. In other words, if I take out either of the routes or shut down either T1, I can get a solid 1.4 Mb/s transfer rate across the remaining link. But with both T1s up and both routes in place, I'm only getting 1.6 Mb/s (distributed evenly across both T1s.) One fly in the ointment that I don't know the ramifications of is that the other end of the link is ATM, and I don't have that config immediately handy but can get it changed if necessary. Any ideas what might be going on? Selected config is below, along with "show interface" during a transfer with both T1s up. Thanks!

!
version 12.2
!
ip cef
!
controller T1 1/0
framing esf
linecode b8zs
channel-group 1 timeslots 1-24
!
controller T1 1/1
framing esf
linecode b8zs
channel-group 1 timeslots 1-24
!
interface FastEthernet0/0
ip address 33.74.246.130 255.255.255.248
load-interval 120
duplex full
!
interface Serial1/0:1
bandwidth 2000
no ip address
encapsulation frame-relay IETF
load-interval 120
fair-queue
frame-relay lmi-type ansi
!
interface Serial1/0:1.2 point-to-point
ip address 33.74.252.14 255.255.255.252
ip load-sharing per-packet
frame-relay interface-dlci 16
!
interface Serial1/1:1
bandwidth 2000
no ip address
encapsulation frame-relay IETF
load-interval 120
fair-queue
frame-relay lmi-type ansi
!
interface Serial1/1:1.1 point-to-point
ip address 33.74.252.10 255.255.255.252
ip load-sharing per-packet
frame-relay interface-dlci 17
!
ip classless
ip route 0.0.0.0 0.0.0.0 Serial1/0:1.2
ip route 0.0.0.0 0.0.0.0 Serial1/1:1.1
!

#sh int
FastEthernet0/0 is up, line protocol is up
Hardware is DEC21140A, address is 0018.18f8.2e00 (bia 0018.18f8.2e00)
Internet address is 33.74.246.130/29
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:56:28, output 00:00:00, output hang never
Last clearing of "show interface" counters 10:34:28
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
2 minute input rate 1616000 bits/sec, 156 packets/sec
2 minute output rate 76000 bits/sec, 143 packets/sec
     731598 packets input, 149259920 bytes
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     1229096 packets output, 1512570370 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Serial1/0:1 is up, line protocol is up
Hardware is Multichannel T1
MTU 1500 bytes, BW 2000 Kbit, DLY 20000 usec,
     reliability 255/255, txload 102/255, rxload 4/255
Encapsulation FRAME-RELAY IETF, crc 16, Data non-inverted
Keepalive set (10 sec)
LMI enq sent 3807, LMI stat recvd 3807, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 634/0, interface broadcasts 0
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 10:34:29
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations 0/7/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1500 kilobits/sec
2 minute input rate 34000 bits/sec, 73 packets/sec
2 minute output rate 807000 bits/sec, 82 packets/sec
     617010 packets input, 750004472 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     370924 packets output, 71287468 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
no alarm present
Timeslot(s) Used:1-24, subrate: 64Kb/s, transmit delay is 0 flags
Serial1/0:1.2 is up, line protocol is up
Hardware is Multichannel T1
Internet address is 33.74.252.14/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 102/255, rxload 4/255
Encapsulation FRAME-RELAY IETF
Serial1/1:1 is up, line protocol is up
Hardware is Multichannel T1
MTU 1500 bytes, BW 2000 Kbit, DLY 20000 usec,
     reliability 255/255, txload 102/255, rxload 4/255
Encapsulation FRAME-RELAY IETF, crc 16, Data non-inverted
Keepalive set (10 sec)
LMI enq sent 3807, LMI stat recvd 3807, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 634/0, interface broadcasts 0
Last input 00:00:04, output 00:00:00, output hang never
Last clearing of "show interface" counters 10:34:29
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations 0/9/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1500 kilobits/sec
2 minute input rate 34000 bits/sec, 73 packets/sec
2 minute output rate 806000 bits/sec, 82 packets/sec
     616782 packets input, 750083692 bytes, 0 no buffer
     Received 0 broadcasts, 8 runts, 0 giants, 0 throttles
     8 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     370991 packets output, 71400072 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
no alarm present
Timeslot(s) Used:1-24, subrate: 64Kb/s, transmit delay is 0 flags
Serial1/1:1.1 is up, line protocol is up
Hardware is Multichannel T1
Internet address is 33.74.252.10/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 102/255, rxload 4/255
Encapsulation FRAME-RELAY IETF

paolo bevilacqua · ‎10-04-2010

Since per-packet load balancing causes out-of-order packet arrival, what likely it's happening, is that the systems are wasting a lot of bandwidth in trying to recover from that, with continuous retransmission and transmit window resets. You can verify that looking at the TCP statistics of the systems involved.

Conclusion: either use a mechanism like MLPPP that guarantees in-sequence packet arrival, or do not use per-packet load balancing.

rhaskell · ‎10-04-2010

Thanks Paolo. We did try MLPPP (as illustrated here: https://supportforums.cisco.com/message/740711#740711 ,) but had difficulty getting the multilink interfaces to come up and couldn't figure it out (possibly due to the fact that it's FR on one end and ATM on the other.) I'll have to take a closer look at the traffic content and check for retransmissions as you suggest.

paolo bevilacqua · ‎10-04-2010

I understand and believe that for some reason is often difficult to have a stable MLPPP over a FR/ATM network like your, however if your objective is the maximum single-session performance, I think there is no alternative.