I can certainly try iperf but how is that going to help me with my problems? Citrix, remote desktop protocols are TCP and that's the problem I'm facing now. Because in the first topology, the server side is where all the servers are located (central office) and offices connected to the central office to use Citrix and RDP. I'll let you know how I go with iperf.

Has anyone have a suggestion on which parameter should I be using when testing using iperf? My circuit is 150Mbps. Thanks.

The server/client are exactly the same. Both have only one NIC that's connected to the network and I've checked the speed/duplex setting on both already just to make sure it's not the PC/server.

Regarding the link utlization, I've already checked that and it's running at about 70-80Mbps on a 1 1Gbps interface and the access on the MPLS cloud is 150Mbps. This is a L3 MPLS cloud where all CE nodes can be directly onnected to each other without having to go through a central hub.

Yes I know about the limited QoS capability on the 3750 switch.

Well I agree, your results are strange when you switch host roles as FTP server/client.

As to 70 to 80 Mbps across 150, especially if based on a 5 minute average, although that's only about 50%, it's possible that Citrix is being impacted by traffic bursts. A quick test might be to run a series of pings (hundred or so) and pay attention to the deltas between min/avg/max. What you want to see is little difference between the values, and a min latency that's close to the distance ideal.

Relating to the UDP test using iperf, I found another method of carrying out the UDP test. downloaded a Windows TFTP client from http://www.tftp-server.com/tftp-client.html and using the same 3Com Daemon installed at both ends. Here are the results:

Topology 1:

Client----Router----MPLS----Router----Server

C:\Download>TFTP -i 10.10.51.251 GET abc

WinAgents TFTP Client version 1.4 Copyright (c)2004-2007 by Tandem Systems,Ltd.

http://www.winagents.com - Software for network administrators

Transfering file abc from server in octet mode...

Using blocksize = 512

Using TFTP timeout = 10s

Transfer size = 9141594 bytes

File abc was transferred successfully.

9141594 bytes transfered for 78 seconds, 117199 bytes/second

C:\Download>TFTP -i 10.10.51.251 PUT abc

WinAgents TFTP Client version 1.4 Copyright (c)2004-2007 by Tandem Systems,Ltd.

http://www.winagents.com - Software for network administrators

Transfering file abc to server in octet mode...

Using blocksize = 512

Using TFTP timeout = 10s

Transfer size = 9141594 bytes

File abc was transferred successfully.

9141594 bytes transfered for 78 seconds, 117199 bytes/second

Topology 2:

Server----Router----MPLS----Router----Client

With GET command, I get 121887 bytes/second transfer (75 seconds) and with PUT command, I get 120284 bytes/second transfer (76 seconds)

Basically I get good results for both topology when using UDP as the protocol.

The major difference between FTP and TFTP, the former will ramp up its bandwidth demand, the latter doesn't. Also FTP should attempt to use max MTU; unsure about TFTP, likely up to the application and yours might be limiting itself to 512. Was the network under typical load during these tests?

The nature of the two applications bandwidth usage is so different, TFTP being good doesn't really tell us much.

One important point; which issue is most important to you, finding the cause of the strange FTP performance results, or the cause of the poor Citrix performance? Solving both would be ideal, and they might be related, or might not too.

Earlier you noted Citrix runs over TCP, but at the application level, if you're using it for screen scraping, its network usage is often much different from FTP. Packets shouldn't be as large, bandwidth demand usually much, much ligher, won't ramp up bandwidth demand, more two way interaction, etc.

For Citrix, ping tests when there's a normal production load, might be very informative, since at least for its screen scraping, you want no packet loss and minimum possible latency.

PS:

BTW:

In my earlier post I mentioned checking ping times against ideal latency. NetQoS has a free on-line latency calculator that you can estimate what latency ought to be.

Thanks for the feedback. I went to www.netqos.com to look for that latency calculator but could not find it. Anyway, the WAN is a private network so I doubt that the calculator would work since it's an online tool.

Thanks for clarifying the diff b/w UDP and TCP. I understand that Citrix and FTP behaves differently and Citrix usually requires little bandwidth comparing to FTP and it usually conform to fixed/standard packet size. Problem is when the problem occur, it usually lasted for quite a long time (several hours). I've checked the Citrix servers many times over and can't see any problem with them. Network troubleshooting tool is very limited (basically little) because of $ hence I have to resort to these basic tools to troubleshoot things on the network. The ISP is not that fantastic neither and most of the time I have to tell them what is the problem.

I've installed a free tool called PingPlotter on the client side and do a continuous ping/trace to the data centre (DC) and have captured the ideal (now) delay time and have informed the client to contact me when the problem occur. Also, Solarwinds is setup to poll every 60 seconds and collect stat every 5 mins. Can't see anything high delay during problematic window. Also carry out manual ping between client and DC and again cannot see any spike with delay.

http://www.netqos.com/resourceroom/calculator/

I've used PingPlotter myself, nice little app to graph ping response performance.

Do I understand that you've done manual pings when a Citrix performance problem has been reported and problem is currently active, but ping results are the same as when there is no reported problem?

For another inexpensive (as in free) tool, that might help in your investigation, you might look into obtaining and using WireShark (http://www.wireshark.org/).

Thanks for the link, I'll check it out ASAP. Yes, your understanding is correct. Manual ping during the period when problem was reposrted is the same as when users reported normal response so there is no ping delay regardless of the situation.