Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

Application slowness over T-1 but not backup link

If you can figure out this you are indeed among the elite of troubleshooting. We have a remote site connected to our main site over a Full T-1 connection. We also have a backup DSL connection from this remote site to the main site connected using DMVPN (Dynamic Multipoint VPN) in case the T-1 link goes down. Now here is the weird issue. Users at the remote site "connected via the T-1" while trying to use a certain "web based http" (the application in stored at the main site on a server) application experience applicaiton lockup and extreme slowness, now when they try to use "https" the issue goes away and they experience no applicaiaton lockups of slowness. On the other hand when testing if we take the T-1 down and use the much slower backup DMVPN connection both http and https work just fine! Granted the applicaiton is a little slow because they are going through the internet and it is encrypted, but the application does not lockup using http or https. Thanks in advance for any suggestions.

28 REPLIES
Hall of Fame Super Bronze

Re: Application slowness over T-1 but not backup link

I've seen in occasions where browser lockup during a session due to MTU issues.

Are you running some kind of GRE tunnel between these T1 link ?

Can you check if the packet is being fragmented by using this command:

ping -l 1500 -f x.x.x.x

x.x.x.x being the target ip at the other side of the link.

Community Member

Re: Application slowness over T-1 but not backup link

I don't think we are running a GRE tunnel between these T1 link. I'll check the ping test you mentioned. Do you think possibly setting "ip tcp adjust-mss 1436" on each serial interface of the T1 link could help?

Hall of Fame Super Bronze

Re: Application slowness over T-1 but not backup link

Yes, as explained on this URL

http://www.cisco.com/warp/public/105/56.html

Or you can implement PMTUD by following this link:

http://www.cisco.com/warp/public/105/pmtud_ipfrag.html

Community Member

Re: Application slowness over T-1 but not backup link

It's kind of strange that this just recently started happening? Do you suspect this could be a carrier issue? It's just strange that with the same application https works, but http causes a lockup.

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

My first reaction was that it might help and would not hurt to specify tcp adjust-mss (and if it were me I would put it on the Ethernet interface rather than the serial).

My second reaction is that I am not sure that it is an issue with fragmentation if there is a problem with HTTP but there is not a problem with HTTPS. Both HTTP and HTTPS are TCP based and if they are going to transmit the same information then why would there be an issue with one but not with the other? It makes me wonder if there is some involvement with the server at the other end of the connection.

This leads me to question whether it really would be transmitting the same information between HTTP and HTTPS? How do you run the "same" application over HTTP and HTTPS? I wonder what the server is doing differently when data is transported over HTTP than what it does when data is transported over HTTPS?

So I think that you might go ahead and try tcp adjust-mss. But I am not optimistic that it will solve the issue.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

The server has a security certificate for this web application since this site/application reached either internally (intranet) or externally from the internet. Users internally can just put an "s" https and access the web application just the same as using plain ole http.

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

So you would put this command on the physical ehternet interfaces of both the A and Z end routers of the T-1?

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

Yes I would. I believe that I remember when I first discovered the adjust-mss that the documentation discussed it in terms of assignment on LAN interfaces rather than others. I have seen some things that suggest that now it works on physical interfaces in general (not restricted to LAN) but I have continued to put it on LAN interfaces and it works reliably for me that way.

It would be an interesting experiment to try adjust-mss on the LAN interface and if it does solve the problem to move it to the serial interface and determine if it also works there.

Give it a shot and let us know the outcome.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

Any suggestion to what I should set the adjust-mss to for starters?

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

My suggestion would be to start with something very low - lets see if it makes a difference. So something like 1200 might be where I would start. If it makes a difference then you can start experimenting with various values trying to find the optimum. If it does not make any difference at a low value then lets move on and look for a different theory of how to solve it.

It occurs to me that no one has yet made the usual request: can we see configs from the routers. If a low value of adjust-mss does not make any difference then I would ask if you could post router configs. Perhaps we might find something in the configs that would point us in the right direction.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

I'll give this a try. I'll put the command on the router's LAN FA0/0 interface of the router at the end that is having the trouble.

Thanks

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

Would you also try setting the ip mtu size on the interface as shown in the example on this link?

http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t4/ft_admss.htm

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

I might be inclined to try ip tcp adjust-mss by itself first and then to try adding the ip mtu size to see if it changes anything.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

Just curious, but is this typically type of issue that could be spotted/detected with a protocol analyzer such as Wireshark?

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

If the issue is that packets are being sent that are large and need fragmentation but fragmentation is not possible (which is the basic problem that tcp adjust-mss will solve) the router that drops the packet should generate an ICMP error message which indicates that fragmentation required but DF set. Assuming that those messages are not filtered out somewhere, then a protocol analyzer such as Wireshark should see them and it would be a good identifier of the problem.

Especially if the adjust-mss (and ip mtu) do not improve things it might be very helpful to have a Wireshark capture of packets to see if it shows anything significant about the problem.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

I do have a Wireshark capture. Attached is a screen shot of a frame and it looks like DF is set.

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

It is pretty clear that the DF bit is set in this packet. Having DF set is only an issue if the frame needs to go over some segment in the data path which has a smaller size. The capture indicates a length of 1500 which in general should be ok. One of the questions is whether the frame will go over some link with a smaller maximum size.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

I am posting some of the config snipet's from the HUB and remote router. I have two questions. Do you think an issue like we are experiencing is a usually a configuration issue or could this ever be a "carrier" issue? And what is the best way to find out if there is a link with a smaller maximum seg size?

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

I will take a look at the config snipets. In the meantime issues like this could possibly (depending on the carrier's environment) be a carrier issue.

If you want to find whether there is a link with a smaller maximum size I would suggest sending pings from one end to the destination. In the ping set the DF bit (from router it is available in extended commands of extended ping or from Windows it is the -f option in ping) and experiment with various sizes.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

I am posting some of the config snipet's from the HUB and remote router. I have two questions. Do you think an issue like we are experiencing is a usually a configuration issue or could this ever be a "carrier" issue? And what is the best way to find out if there is a link with a smaller maximum seg size?

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

I have looked at the config parts that you posted. It looks to me like if segment size (tcp adjust-mss) were going to be an issue anywhere it would be an issue on the DMVPN tunnel since GRE and IPSec VPN add extra headers. I do not see anything that suggests that there is a fragmentation issue with traffic over the T1. At this point it might be interesting to know if tcp adjust-mss makes any difference. But I am not at all optimistic that it will change anything.

HTH

Rick

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

My mind is going back to the description of the problem where you say that the remote users have a problem if they use HTTP but do not have a problem is they use HTTPS. That would seem to indicate that it is less likely to be a networking problem since the transmission of data through the network, requirements for fragmentation, etc are the same for HTTP and HTTPS.

Do you happen to have a Wireshark capture of packets when a user is having problems going over the T1? Does it indicate any issues?

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

Rick,

I do happen to have a few wireshark captures while users are experiencing issues. To be honest I am a bit new to properly learning how to interpret protocol analyzer output. I know a little about interpretation, but am not sure I could spot the issue we are having in the capture. Should I post the capture here or email it to you? The captures are 4 and 5MB's.

Thanks,

Brandon

Hall of Fame Super Gold

Re: Application slowness over T-1 but not backup link

Brandon

Somehow I missed this post yesterday. If you want to post the capture files that probably is ok (I am not clear whether there is any size limitation in posting files on NetPro). Or if you wish you can just email them to me. My email address is in my profile on NetPro. As the comment in the profile says please identify the email as related to NetPro, otherwise my spam filter might not allow it through.

HTH

Rick

Community Member

Re: Application slowness over T-1 but not backup link

I would be interested in those too and they would probably prove insightful. Also, back in the day, I used SSLdump to do some fantastic troubleshooting on various SSL server issues. Perhaps that is still a viable option.

http://www.rtfm.com/ssldump/

Community Member

Re: Application slowness over T-1 but not backup link

Brandon,

Hopefully you have sorted the issue. If not you may find the following helpful.

When using the T-1, in a sunny day scenario, you are connecting directly to your hub router. The only carrier is the one that is providing the T service. If this is a correct statement this is not a carrier issues since they are not cognizant of the upper layer protocols.

Did you try the ip tcp adjust-mss command? I have found it to be very helpful in the past.

Silver

Re: Application slowness over T-1 but not backup link

DSL line Do you have connected with Ethernet Interface???

If yes, You should also look up on IP Fragmentation.

This feature allows multilink PPP (MLPPP) encapsulation over a single slow link to fragment and interleave packets to a small enough size that the delay requirements of delay-sensitive traffic will be met.

To resolve it Configure properly MTU and fragmentation.

See more on

http://www.cisco.com/en/US/products/hw/routers/ps221/prod_configuration_guide09186a00800993f1.html

http://www.cisco.com/en/US/products/hw/routers/ps221/prod_configuration_guide09186a00800993f1.html

Regards,

Dharmesh Purohit

Community Member

Re: Application slowness over T-1 but not backup link

Dharmesh,

What is happening is while users access the webserver via http the application locks up if they use https the application does not lockup. For troubleshootng if we take the T-1 down and use the backup DMVPN DSL connection then the application works just fine with either http or https. The problem is with http over the T-1 link.

144
Views
5
Helpful
28
Replies
CreatePlease to create content