Odd throughput "simple" test

Unanswered Question
Apr 4th, 2007
User Badges:
  • Gold, 750 points or more

Hi all


I am testing throughput on following topology.


TestBox2-7600-7600-4500-3400-TestBox1


TestBox1 emulate a CE connecting thru L2 which is terminated at 7600.

My test is basic, I send packets and watch them coming back, making sure the QOS policing is working as it should (25Mb allowed, any excess droped).

the capacity end to end is 1Gb.


the odd problem is that I can only pump 5Mb and no more, anything after that start showing packets loss.


what I checked so far:

1-removed all policing on path.

2-checked all interfaces on path for errors or packets drops.

3-ensured interfaces are 1Gb.

4-changed packets size from 64 to 1500 and tested with random sizes.



to recap. from 1Mb to 5Mb all works great, confirmed by tester as well as load on interfaces where I can see input = output. however when I go over 5Mb it goes pear shape.


any pointers or suggestions would be great help as I think I have checked all what needs to be checked on this simple set up.


TIA


Sam

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3 (1 ratings)
Loading.
ilya.varlashkin Thu, 04/05/2007 - 00:53
User Badges:
  • Silver, 250 points or more

Could you post part of the config showing definition of the policy? Also, make sure that you have correct duplex setting all the way through (assuming you use ethernet everywhere) and for GigE interfaces you should use autonegotiation so that master/slave are correctly set.

cisco_lad2004 Thu, 04/05/2007 - 01:36
User Badges:
  • Gold, 750 points or more

Thanks Ilya


Duplex is hard coded to full, auto negotiation is disabled. service policy has been removed while I am testing, so it is not in the equation...until I know what is killing my eating up my packets.


Thanks


Sam

ilya.varlashkin Thu, 04/05/2007 - 02:19
User Badges:
  • Silver, 250 points or more

You should enable autonegotiation on Gigabit interfaces in order to have correct master/slave setting.


How do you perform measurements and what do you use to generate traffic?

cisco_lad2004 Thu, 04/05/2007 - 02:27
User Badges:
  • Gold, 750 points or more

I am using an traffic generator, where I typically start with lower throughput of 1Mb at constant rate, then gradually increase it once satisfied with correct throughput.

I also take test with various packet size ( one at the time) then various sizes in one test.

MTU has also been increased on teh whole path so it is out of the equation.


Thanks


Sam

cisco_lad2004 Thu, 04/05/2007 - 02:30
User Badges:
  • Gold, 750 points or more

I will also add, the test is fairly simple. I expect to see number of packets sent equal the received. my interfaces, shoudl also show same traffic in as out.

teh above test results are postive when I test from 1Mb to 5Mb, above this. I have 20% packet loss.

avmabe Thu, 04/05/2007 - 05:43
User Badges:
  • Bronze, 100 points or more

what did you set the MTU to?

cisco_lad2004 Thu, 04/05/2007 - 05:50
User Badges:
  • Gold, 750 points or more

4470, but the packet loss is seen for packets as small as 512Byte.

ilya.varlashkin Thu, 04/05/2007 - 06:54
User Badges:
  • Silver, 250 points or more

Could you post config of the edge interfaces?

cisco_lad2004 Thu, 04/05/2007 - 08:21
User Badges:
  • Gold, 750 points or more

Here it is.


interface TenGigabitEthernet2/3

dampening

mtu 9216

no ip address

load-interval 30

carrier-delay msec 16

udld port aggressive

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

hold-queue 4096 in

hold-queue 4096 out

!

interface GigabitEthernet6/6

ip address x.x.x.x 255.255.255.252

load-interval 30

mls qos trust dscp


end




ilya.varlashkin Thu, 04/05/2007 - 12:44
User Badges:
  • Silver, 250 points or more

I'm not sure about what are you doing. Is it ordinary ethernet switching or do you have pseudowire? It's confusing that one of the interfaces you posted is with IP addr, the other is without.


Check that your configuration is consistent and packets are not mis-routed/switched somewhere else. Also, what kind of traffic generator do you use - is it hardware like Agilent or do you use some unix box running Iperf?

cisco_lad2004 Thu, 04/05/2007 - 14:48
User Badges:
  • Gold, 750 points or more

Apologies, corrected and more info:


tester1-7600A-7600B-4500-tester2


This is on the 7600A. so L3 terminated on subif while next 7600B is configured as a trunk allowing vlan 1401 all the way thru tester2


From tester2, I generate traffic using acterna box.


I cannot see any drops, errors etc.. on the path at all.


***************7600A**************************

interface TenGigabitEthernet2/3.1401

encapsulation dot1Q 1401

ip address X.X.X.X 255.255.255.252

!

interface TenGigabitEthernet2/3

dampening

mtu 9216

no ip address

load-interval 30

carrier-delay msec 16

udld port aggressive

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

hold-queue 4096 in

hold-queue 4096 out


***************7600B***************************


interface TenGigabitEthernet2/3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1401

switchport mode trunk

mtu 9216

no ip address

udld port aggressive

wrr-queue threshold 2 100 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 1 40 70 100 100 100 100 100 100

wrr-queue random-detect min-threshold 2 70 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 3 70 100 70 100 100 100 100 100

wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100

wrr-queue cos-map 3 1 3

wrr-queue cos-map 3 2 4

wrr-queue cos-map 3 3 6 7

mls qos vlan-based

mls qos trust dscp

spanning-tree portfast trunk





ilya.varlashkin Fri, 04/06/2007 - 02:57
User Badges:
  • Silver, 250 points or more

Ok, try first to remove all 'wrr-queue cos-map' and especially 'wrr-queue random-detect' commands on all involved interfaces (edge and transit), check the throughput. Also, make sure that none of the interfaces (including your traffic generator) is sending PAUSE frames; for this apply following on all interfaces:


flowcontrol receive off

flowcontrol send of


It's good idea to baseline tester itself - connect source port to the sink port, and check the performance.


cisco_lad2004 Fri, 04/06/2007 - 08:21
User Badges:
  • Gold, 750 points or more

Hi Ilya


Thanks for the suggestions!

I will follow up and hopefully get to the bottom of this issue.


Eitherway, I will come back with feedback.


Once again , many thanks


Sam

cisco_lad2004 Mon, 04/09/2007 - 02:36
User Badges:
  • Gold, 750 points or more

Hi Ilya



After base lining the Acterna and checking the routing path. I have managed to narrow down where the packets are lost it is between 2 7600 connected by Etherchannel.

No stats on either interface or QOS/WRED are confirming this drops.


Thanks again for teh suggestions !


Sam

cisco_lad2004 Tue, 04/10/2007 - 10:07
User Badges:
  • Gold, 750 points or more

Hi Ilya


I am pretty sure it is caused by CSCee10005.

so next thing will be to remove DEC and then enable ETC on same module.


Sam

ilya.varlashkin Wed, 04/11/2007 - 12:11
User Badges:
  • Silver, 250 points or more

Samir,


it should be possible to verify whether you're hitting that bug or something else. Since bug is due to cross-module etherchannel, try using only single link for the test.

Actions

This Discussion