Hi Eric,

I would start doing a wireshark capture while FTP and windows copies are done to see if there are retransmissions.

Then I would follow that line and see if you have drops along the way, congested interfaces or MTU sizes mismatches along the path.

Check also if the mms is correctly negotiated between the sender and the receiver.

regards,

Riccardo

We have gathered a few packet captures and I don't see any fragmentation which I think would indicate a MTU issue. No re-transmissions or anything that would indicate an issue. Window sizing looks ok as well.

We are about 90% sure its all windows traffic that is slow.

Response time on the circuit is around 15ms.

I'd would have a look at the application side. Antiviruses, Firewalls, disk subsystems.

Also, does write operation experience the same delay?

Is delay could be seen in both directions?

Easy, but need to rule out - any QoS application in between on the network?

What about other type of traffic? HTTP, ICMP for ex.

Did you try for different file types?

Did you try from different pair of PCs?

What about the access switches - anything unusual there?

Ivan,

I'd would have a look at the application side. Antiviruses, Firewalls, disk subsystems.

- We have used several different variations of desktops/laptops but the one common factor is windows (both XP and 7)

Also, does write operation experience the same delay?

- Yes

Is delay could be seen in both directions?

- Yes

Easy, but need to rule out - any QoS application in between on the network?

- We've actually gone as far as removing QoS from our side and the providers side to see if there is a difference. No difference was noticed.

What about other type of traffic? HTTP, ICMP for ex.

- Good thought, I will try with HTTP. I have to find out what our backup server is using because that works fine. Just seems to be isolated to windows file copies.

Did you try for different file types?

- Yes, zip files and text files both with the same result

Did you try from different pair of PCs?

- Yes, both laptops and desktops

What about the access switches - anything unusual there?

- Nope, 4510's with dual 10gb sup, gig ports

Cisco TAC is pushing the issue on windows and SMB sizes but I'm not 100% convinced. If it is that issue, none of my other sites (non-mpls) are having the issue and that is a huge undertaking to tweak the server farm.

Thanks,
Eric

Thanks,

Eric

Eric,

As I understood - you're trying to download from a server, i.e. every time from a centralized location, right? I'd suggest to give a try to a pc2pc scheme, with one pc installed next to the server;

What about linux/samba - any issues with downloads from such a system installed next to the server?

What maximum size of ICMP with the DF bit turned on is passing the circuit? What is the size of packets in your captures?

Also following Riccardo's question - have you checked together with your MPLS provider whether there're any drops being observed on the interfaces along the path?

Is there any caching engine on the network?

Is there any firewall(s) installed for this particular site(s)?

Very generic question which I am sure you have already answered while dealing with TAC - have this setup ever been working for you? and of course - what changes have been made since then :)

Ivan,

Answers below to your questions. We have found that HTTP transfers also seem slow. Looking at the captures, we are seeing a few "TCP segment of  a reassembled PDU" messages. I'm not sure what would cause that or if its relevent.

As I understood - you're trying to download from a server, i.e. every  time from a centralized location, right? I'd suggest to give a try to a  pc2pc scheme, with one pc installed next to the server;

- yes, multiple servers (all windows). The centralized servers are all at the datacenter. The remote site connects to the datacenter via the MPLS VPN. pc2pc on the remote LAN are fine but a pc on the remote lan to a pc in the datecenter is also slow

What about linux/samba - any issues with downloads from such a system installed next to the server?

- a great test but unfortunetely we are a complete windows shop. One of our next steps is to get a linux server setup 

What maximum size of ICMP with the DF bit turned on is passing the circuit? What is the size of packets in your captures?

- The size of the frames are 78 bytes. The maximum ping size using ping -l was I believe 1476. I will have to verify in the morning.

Also  following Riccardo's question - have you checked together with your  MPLS provider whether there're any drops being observed on the  interfaces along the path?

- the provider has provided stats and they say everything is fine. Unfortunetely there are two providers involved and I can only believe in what the tell me and we all know how honest they are!

Is there any caching engine on the network?

- No

Is there any firewall(s) installed for this particular site(s)?

- No

Very  generic question which I am sure you have already answered while  dealing with TAC - have this setup ever been working for you? and of  course - what changes have been made since then

- Its been slow since day 1

Thanks,

Eric

Hi Eric,

just to recap it would seem that (can you please confirm or correct the following statements):

1) Slowness is experienced between the datacenter and a remote site which is the only one connected via MPLS VPN managed by 2 different ISP's.

2) All data transfers from remote location to datacenter, and vice versa, are affected. Exception is FTP transfers which are ok.

3) Data transfers of the same type between devices in the datacenter or between the datacenter and a different remote site (not connected via MPLS VPN) are fine.

4) Only Windows based PC/servers are affected (can you list the non windows ones you have and the type of tests you did on them?)

About  the FTP transfers, which application do you use? FTP should use TCP and therefore in theory should also be affected. Can you check on the capture that traffic is really sent via TCP segments?

Then we should see gather some baseline information.

Can you install iPerf on a PC in the DC and one in the remote site?

Then you should perform some iperf tests, TCP first and then UDP ones, and see the max bandwidth you can get and whether you have packet loss.

A nice iperf tutorial (in case you are not familiar with it) is here: http://openmaniak.com/iperf.php

After the test I would like you to try reducing the MTU on both PCs to the highest value before fragmentation occurs (apparently you mentioned 1476).

The procedure to do that is here: http://help.expedient.com/broadband/mtu.shtml

After that can you try the same iperf tests and also your FTP/Windows file transfers to see if you have any benefit/change?

Lastly, can you give me your SR number as I could not find it? I would like to have a look at it I am not completely sure it makes too much sense keep on discussing about your issue here as a TAC case is already opened. The ratio behind having Cisco chaps in the forum helping out is to avoid that you ppl open TAC cases in the first place for issues that can be solved somewhere else. Anyway let's keep on rolling here.

Riccardo

Riccardo,

Thanks for the response. So late last night we received some interesting news from the provider about how they want to "redesign" the circuit. Apparently with a redesign of the circuit, they can get the 16ms response time down to about 2ms.

Here are the answers to your questions though.

1 - So there is one provider of the MPLS VPN design; however, there is a second provider on the last mile only providing physical connectivity.

2 & 4 - Only traffic involving windows shares. FTP and backups using UDP are fine.

3. Data transfers between other remote sites not using the MPLS are fine.

- The FTP application was filezilla and here is the SR 618860547. They were basicially focussing in on windows SMB being the issue.

At this point, since the ISP has suggested a redesign, I'm suspecting they have something going on internally.

Since this has been a very interesting topic, I'll continue to update everyone once the provider completes whatever they are changing to see if there is a difference.

Thanks for all the advice and information!

Steve,

ok let us know later on.

I quickly checked filezilla specs and I saw that you can also use UDP for transfers. I tend to believe that you configured it to use udp.

If this is the case the explanation is simple, even though in the capture you did not see retransmissions and the ISP denied it (until last night) there were drops somewhere along the path which affected TCP based transfers only due to the TCP sliding window.

iperf could have given you an answer in that sense too, I suppose.

cheers,

Riccardo

Gilles,

That is exactly our problem. The slow transfers are  related to our CIFS environment and the servers are pre-2008. The MPLS  speed is 50 Mbps. We were able to get around 28Mbps through FTP but  about a third of that through windows copy.

Thanks for the information.

Eric

 The slow transfers are  related to our CIFS environment and the servers are pre-2008.

time to open a case with Microsoft TAC? :)