Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Trouble with Ethernet WAN

Hi! This is my first post here. Thanks to Cisco for providing this valuable resource. I'm primarily a Unix system administrator, but I wear many hats do to the small size of the company I'm currently working for. In other words, we don't have a dedicated network engineer.

We recently turned up an AT&T Ethernet WAN circuit between our two data centers (one in Boston, the other in New Jersey). The circuit is supposed to provide 150 Mbit/s. Testing with iperf, I see the expected throughput running from New Jersey to Boston (>100 Mbit/s), but only about one fifth what is expected running from Boston to New Jersey (30 Mbit/s). There are no errors on the Cisco 3560 switches at either end. The odd thing is, if I run 5 iperf clients simultaneously in Boston, each will show just over 20 Mbit/s, which means just about 100 Mbit/s total.

Any ideas what might be wrong? Configuration snippets and iperf test results can be found at the end of the post. Thanks in advance for any suggestions.

Best wishes,

Myles

Boston Switch:

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

cisco WS-C3560G-24TS (PowerPC405) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1028Z3DD

#show interfaces gi0/27 status

Port      Name               Status       Vlan       Duplex  Speed Type

Gi0/27                       connected    26           full   1000 1000BaseLX SFP

#show interfaces gi0/27

GigabitEthernet0/27 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 0018.b954.bd1b (bia 0018.b954.bd1b)

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive not set

  Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseLX SFP

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:34, output 00:00:01, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     119154244 packets input, 47574275052 bytes, 0 no buffer

     Received 7261 broadcasts (7234 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 7234 multicast, 0 pause input

     0 input packets with dribble condition detected

     189612796 packets output, 266293462185 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

Boston Test Host:

# ifconfig eth3

eth3      Link encap:Ethernet  HWaddr 00:24:81:7C:79:43

          inet addr:192.168.26.4  Bcast:192.168.26.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:1128881484 errors:0 dropped:0 overruns:0 frame:0

          TX packets:5901215833 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:126526667110 (117.8 GiB)  TX bytes:8700597434340 (7.9 TiB)

          Interrupt:185 Memory:fdea0000-fdec0000

# ethtool eth3

Settings for eth3:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

        Port: Twisted Pair

        PHYAD: 1

        Transceiver: internal

        Auto-negotiation: on

        Supports Wake-on: d

        Wake-on: d

        Current message level: 0x00000001 (1)

        Link detected: yes

Single iperf client running in Boston:

# iperf -c 192.168.26.5 -n 1073741824 -m

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 36403 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-278.6 sec  1.00 GBytes  30.8 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

Five iperf clients running simultaneously in Boston:

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 54037 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-81.5 sec   205 MBytes  21.1 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 54038 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-83.4 sec   205 MBytes  20.6 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 54039 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-85.7 sec   205 MBytes  20.0 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 54040 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-85.7 sec   205 MBytes  20.1 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 54041 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-83.0 sec   205 MBytes  20.7 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

New Jersey Switch:

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

cisco WS-C3560G-48TS (PowerPC405) processor (revision D0) with 131072K bytes of memory.

Processor board ID FOC1047Z59G

#show interfaces gi0/52 status

Port      Name               Status       Vlan       Duplex  Speed Type

Gi0/52                       connected    26           full   1000 1000BaseLX SFP

#show interfaces gi0/52

GigabitEthernet0/52 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 001a.6c4b.08b4 (bia 001a.6c4b.08b4)

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive not set

  Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseLX SFP

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:01, output 00:00:48, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     188270859 packets input, 264260280083 bytes, 0 no buffer

     Received 224109 broadcasts (223448 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 223448 multicast, 0 pause input

     0 input packets with dribble condition detected

     119392994 packets output, 47761060680 bytes, 0 underruns

     0 output errors, 0 collisions, 2 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

New Jersey Test Host:

# ifconfig eth3

eth3      Link encap:Ethernet  HWaddr 00:15:17:16:DF:3B

          inet addr:192.168.26.5  Bcast:192.168.26.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:188140437 errors:0 dropped:0 overruns:0 frame:0

          TX packets:119225785 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:263306036792 (245.2 GiB)  TX bytes:46892828314 (43.6 GiB)

          Interrupt:177 Memory:fdfa0000-fdfc0000

# ethtool eth3

Settings for eth3:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

        Port: Twisted Pair

        PHYAD: 1

        Transceiver: internal

        Auto-negotiation: on

        Supports Wake-on: d

        Wake-on: d

        Current message level: 0x00000001 (1)

        Link detected: yes

Single iperf client running in New Jersey:

# iperf -c 192.168.26.4 -n 1073741824 -m

------------------------------------------------------------

Client connecting to 192.168.26.4, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.5 port 13424 connected with 192.168.26.4 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-77.7 sec  1.00 GBytes   111 Mbits/sec

[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

Everyone's tags (3)
5 REPLIES
Super Bronze

Re: Trouble with Ethernet WAN

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Might be lost packets in the MetroE providers cloud.

Might be one set of clients RWIN isn't large enough for th bandwidth delay product.

New Member

Re: Trouble with Ethernet WAN

I think frames are being dropped in the "cloud". I tried testing with iperf using UDP instead of TCP, setting a target bandwidth of roughly 140 Mbit/s. Quite a few datagrams were dropped.

# iperf -c 192.168.26.5 -u -b 140000000 -t 60 -i 6

------------------------------------------------------------

Client connecting to 192.168.26.5, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size:  126 KByte (default)

------------------------------------------------------------

[  3] local 192.168.26.4 port 34346 connected with 192.168.26.5 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0- 6.0 sec   100 MBytes   140 Mbits/sec

[  3]  6.0-12.0 sec   100 MBytes   140 Mbits/sec

[  3] 12.0-18.0 sec   100 MBytes   140 Mbits/sec

[  3] 18.0-24.0 sec   100 MBytes   140 Mbits/sec

[  3] 24.0-30.0 sec   100 MBytes   140 Mbits/sec

[  3] 30.0-36.0 sec   100 MBytes   140 Mbits/sec

[  3] 36.0-42.0 sec   100 MBytes   140 Mbits/sec

[  3] 42.0-48.0 sec   100 MBytes   140 Mbits/sec

[  3] 48.0-54.0 sec   100 MBytes   140 Mbits/sec

[  3] 54.0-60.0 sec   100 MBytes   140 Mbits/sec

[  3]  0.0-60.0 sec  1001 MBytes   140 Mbits/sec

[  3] Sent 714197 datagrams

[  3] Server Report:

[  3]  0.0-60.0 sec  1000 MBytes   140 Mbits/sec   0.168 ms  954/714196 (0.13%)

[  3]  0.0-60.0 sec  1 datagrams received out-of-order


Hall of Fame Super Gold

Trouble with Ethernet WAN

Who's doing packet shaping of your WAN traffic?

New Member

Re: Trouble with Ethernet WAN

Any shaping or QoS is being done by AT&T. To us, it's just Ethernet.

Thanks,

Myles

New Member

Re: Trouble with Ethernet WAN

AT&T shed some light on this for me, and it is now performing satisfactorily in both directions.

At each end, the 1 Gbit/s ethernet is multiplexed on 3 50 Mbit/s SONET STS channels (this is how 150 Mbit/s is achieved). The frames that wouldn't fit down the 3 STS channels were being discarded.

I added the following to the switch ports at both ends:

srr-queue bandwidth limit 14

(14% of 1 Gbit/s is roughly 140 Mbit/s)

Best wishes,

Myles

318
Views
0
Helpful
5
Replies
CreatePlease to create content