MPLS drops

[rywatters]

We have set up an MPLS with our telco. Main office is in Idaho, remote branch is in New Mexico. When we try to copy large files across the pipe, we get errors. If I've done an SSH session to the router in the remote office, I randomly lose my connection. We use UltraVNC as a support tool to do remote sessions with our users in New Mexico, and it will frequently lose connection and we have to reconnect. If you do a MSTSC remote desktop session out to a server in New Mexico, it will have frequent hiccups where it has lost the connection and has to re-establish it.

When we opened a ticket with the telco to try and figure out why we're having these problems they ran some tests overnight and just sent us back results saying: "Tested clean to the NIU and the CSU." Which is great, but doesn't really help us address the problem. The only thing I can see on the routers when I do a 'sh int' on the routers is that they have a large number of total output drops. This would generally indicate congestion on the interface, but there just isn't that much traffic flowing across. I've used the SolarWinds Netflow Analyzer and I don't really see any more than about 75Kb/s worth of bandwidth being used.

The question I'd have then is that the router is connected into my network at 100Mbps, and the serial link is 1.5Mbps. Am I just overwhelming the interface by squeezing from 100Mbps to 1.5Mbps from one interface to the other? Anyone have any ideas of why I'm having these problems?

[Edison Ortiz]

It sounds like your provider is policing any exceeded packet causing you to drop the connection.

Your router is also showing that you are trying to exceed its egress interface.

To mitigate excess traffic, you need to implement QoS in your WAN router so traffic coming from a 100Mbps towards a 1.5Mbps can be delayed during congestion.

If you are overwhelming your circuit over 75% of the time, you may need to consider upgrading its bandwidth since QoS can only do so much.

HTH,

__

Edison.

[rywatters]

Thank you for the response. I've mostly been working on Pix and ASA firewalls, so these are the first routers we've had in for our company. I haven't really had a lot of exposure to QoS, and the telco set up the routers they sent out to us. I see 5 class maps defined in the configuration, and I see 15 policy maps. However, on my 'sh int' I don't see the service-policy line actually attaching any of the policy maps to the serial interface.

Is there a good primer I can go through that goes through understanding and setting up QoS?

[Edison Ortiz]

You can verify your service-policy with the 'show policy-map interface' command.

It will show you the interface that is attached and what's actually doing - i.e; priority queueing, bandwidth guarantee, etc and if there are any drops within those classes.

A good read would be this document:

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND/QoS-SRND-Book.html

HTH,

__

Edison.

[Joseph W. Doherty]

If you see a large number of output drops, it's possible your circuit is so overstressed it drives the actual average utilization down (not to mention many other issues, such as those you also describe).

Since you mention MPLS, but not detail further your WAN topology, it's also possible problems can be happening within the "cloud" (cloud-to-site egress drops often being one).

Odds are if TelCo and provider say tested circuit is okay, it usually is, and such issues as you described are more often due to user congestion. However, they sometimes are wrong too.

The root cause of performance problems isn't always easy to find, and if the problem isn't quickly found and resolved, you might want to consider obtaining additional consulation.

[edit]

"15 policy maps. However, on my 'sh int' I don't see the service-policy line actually attaching any of the policy maps to the serial interface. "

15 policy maps? That's quite a few! (I'm guessing so many might be for different QoS profiles the MPLS provide might support.) None applied; so what's the queuing on the interface now?

[edit2]

If Cisco routers, QoS, MPLS, etc., are all new to you, then you really might want to consider obtaining extra consultation. I'm sure you might be able to figure this all out by yourself, but is time to do so an important consideration?

[rywatters]

The queueing strategy is weighted fair. Time isn't a huge consideration at the moment. The MPLS is really for future expansion, not really for the load we're putting out to it now. The concern is that we want to get these issues addressed before we moved on to really beginning to utilize the circuit in the future.

When I do the 'show policy-map interface' command, I don't get anything returned. That pretty much confirms for me that while there's a ton of policy-maps defined, none of them are actually getting applied. At this point, we have more time than money to put towards the project, so that's where we're headed. We're hoping to move forward with a VOIP project next year, so we want to at least get a rough working knowledge of working with QoS anyway in preparation for that. Even if we have a Cisco tech do all of the work when we implement our VOIP strategy, I want to have some idea of what's going on for troubleshooting and tweaking things as the network grows.

[Joseph W. Doherty]

WFQ normally does a very good job of interface congestion management and usually doesn't have the issues you describe. As Edison described in his first post, your provider might be policing within the path, this might be especially true if they sell a data rate less than link's bandwidth.

Since you have time, references like the one Edison also provided are very good and you can learn much. However, my recommendation for consultation was that, consultation; not just obtaining someone to solve the problem at hand (although that, alone, can be provided too).