Odd throughput "simple" test

cisco_lad2004 · ‎04-04-2007

Hi all

I am testing throughput on following topology.

TestBox2-7600-7600-4500-3400-TestBox1

TestBox1 emulate a CE connecting thru L2 which is terminated at 7600.

My test is basic, I send packets and watch them coming back, making sure the QOS policing is working as it should (25Mb allowed, any excess droped).

the capacity end to end is 1Gb.

the odd problem is that I can only pump 5Mb and no more, anything after that start showing packets loss.

what I checked so far:

1-removed all policing on path.

2-checked all interfaces on path for errors or packets drops.

3-ensured interfaces are 1Gb.

4-changed packets size from 64 to 1500 and tested with random sizes.

to recap. from 1Mb to 5Mb all works great, confirmed by tester as well as load on interfaces where I can see input = output. however when I go over 5Mb it goes pear shape.

any pointers or suggestions would be great help as I think I have checked all what needs to be checked on this simple set up.

TIA

Sam

ilya.varlashkin · ‎04-05-2007

Could you post part of the config showing definition of the policy? Also, make sure that you have correct duplex setting all the way through (assuming you use ethernet everywhere) and for GigE interfaces you should use autonegotiation so that master/slave are correctly set.

cisco_lad2004 · ‎04-05-2007

Thanks Ilya

Duplex is hard coded to full, auto negotiation is disabled. service policy has been removed while I am testing, so it is not in the equation...until I know what is killing my eating up my packets.

Thanks

Sam

ilya.varlashkin · ‎04-05-2007

You should enable autonegotiation on Gigabit interfaces in order to have correct master/slave setting.

How do you perform measurements and what do you use to generate traffic?

cisco_lad2004 · ‎04-05-2007

I am using an traffic generator, where I typically start with lower throughput of 1Mb at constant rate, then gradually increase it once satisfied with correct throughput.

I also take test with various packet size ( one at the time) then various sizes in one test.

MTU has also been increased on teh whole path so it is out of the equation.

Thanks

Sam

cisco_lad2004 · ‎04-05-2007

I will also add, the test is fairly simple. I expect to see number of packets sent equal the received. my interfaces, shoudl also show same traffic in as out.

teh above test results are postive when I test from 1Mb to 5Mb, above this. I have 20% packet loss.

avmabe · ‎04-05-2007

what did you set the MTU to?

cisco_lad2004 · ‎04-05-2007

4470, but the packet loss is seen for packets as small as 512Byte.

ilya.varlashkin · ‎04-05-2007

Could you post config of the edge interfaces?

cisco_lad2004 · ‎04-05-2007

Here it is.

interface TenGigabitEthernet2/3

dampening

mtu 9216

no ip address

load-interval 30

carrier-delay msec 16

udld port aggressive

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

hold-queue 4096 in

hold-queue 4096 out

!

interface GigabitEthernet6/6

ip address x.x.x.x 255.255.255.252

load-interval 30

mls qos trust dscp

end

ilya.varlashkin · ‎04-05-2007

I'm not sure about what are you doing. Is it ordinary ethernet switching or do you have pseudowire? It's confusing that one of the interfaces you posted is with IP addr, the other is without.

Check that your configuration is consistent and packets are not mis-routed/switched somewhere else. Also, what kind of traffic generator do you use - is it hardware like Agilent or do you use some unix box running Iperf?

cisco_lad2004 · ‎04-05-2007

Apologies, corrected and more info:

tester1-7600A-7600B-4500-tester2

This is on the 7600A. so L3 terminated on subif while next 7600B is configured as a trunk allowing vlan 1401 all the way thru tester2

From tester2, I generate traffic using acterna box.

I cannot see any drops, errors etc.. on the path at all.

***************7600A**************************

interface TenGigabitEthernet2/3.1401

encapsulation dot1Q 1401

ip address X.X.X.X 255.255.255.252

!

interface TenGigabitEthernet2/3

dampening

mtu 9216

no ip address

load-interval 30

carrier-delay msec 16

udld port aggressive

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

hold-queue 4096 in

hold-queue 4096 out

***************7600B***************************

interface TenGigabitEthernet2/3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1401

switchport mode trunk

mtu 9216

no ip address

udld port aggressive

wrr-queue threshold 2 100 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 1 40 70 100 100 100 100 100 100

wrr-queue random-detect min-threshold 2 70 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 3 70 100 70 100 100 100 100 100

wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

mls qos vlan-based

mls qos trust dscp

spanning-tree portfast trunk

ilya.varlashkin · ‎04-06-2007

Ok, try first to remove all 'wrr-queue cos-map' and especially 'wrr-queue random-detect' commands on all involved interfaces (edge and transit), check the throughput. Also, make sure that none of the interfaces (including your traffic generator) is sending PAUSE frames; for this apply following on all interfaces:

flowcontrol receive off

flowcontrol send of

It's good idea to baseline tester itself - connect source port to the sink port, and check the performance.

cisco_lad2004 · ‎04-06-2007

Hi Ilya

Thanks for the suggestions!

I will follow up and hopefully get to the bottom of this issue.

Eitherway, I will come back with feedback.

Once again , many thanks

Sam

cisco_lad2004 · ‎04-09-2007

Hi Ilya

After base lining the Acterna and checking the routing path. I have managed to narrow down where the packets are lost it is between 2 7600 connected by Etherchannel.

No stats on either interface or QOS/WRED are confirming this drops.

Thanks again for teh suggestions !

Sam