Hi Suresh,

It may also depend on what device is at Hop 13. For example, certain firewalls may report a slow response time to ICMP, but perform much better when ICMP is passed through it.

Reason being that the ICMP response processing is low priority compared to the day-to-day functional task of checking and passing (or not) traffic.

Regards,

David

Thanks everyone for the nice explanation.

Will it be the ICMP packets only see the delay or general traffic also face the delay.

As far as i know, Router can be configured to give low priority for ICMP packets. But dont think ISP routers are give this much of delay ( Because they will use High end routers).

what about the ping, it is ok between the source and destination, does it mean traffic between source and destination is flowing with the same delay which i am able to see on the ping response.

one more simple question, My BW utilization is just 40% and client end utilization also less than 60%, But VPN tunnel traffic shows delay where as non vpn traffic (Ping from VPN source device to Destination device) is normal.

To marikakis 2nd point. Don't get hung up troubleshooting the troubleshooting tool. I know it's intresting stuff but you'd be better served learning about tracert in an environment that you control. Troubleshooting tracert over the Internet exposes you to so many variables beyond your control.

Just an opinion,

James

In my opinion, engineers have to do the best they can when a complaint from a client comes. Performing a few traceroutes is not too much work if you have a client that is hard to convince. As I said before, the real issue is whether the client does face connectivity or performance issues for real traffic together with the troubleshooting tools showing something not easily explained. What was the reason for the client to decide to perform the traceroute? You cannot know if a real problem exists unless you explore it. You cannot just say "traceroute is worthless" to the client when problems occur, and use it at other times as a clue to convince the client everything is fine.

You cannot know if real packets face the delay reported by traceroute unless you interact with the site. In this case, things are easy. Delays in the order of 400ms are noticable by humans. If you interact with the site very quickly when clicking here and there at the web interface, then the delay is just traceroute delay. If site responds slowly, you need to dig deeper, but still over the Internet you cannot know if this is a problem of the web server or a problem of the intermediate network and that's why you need a baseline traceroute. If you have the same traceroute delays when the site is slow and when it is fast responding, then you have a clue pointing towards the site being the problem.

Also have in mind that no matter how annoying a single client complain about a single site might look, you can see it as a help from the client that gives you a hint to diagnose early a potential issue that might affect other clients soon. In my own opinion, a client complaint should not be left unexplored. This procedure is part of the quality of the service you provide to the client.

Just because ISP routers are monsters this does not mean they are left unprotected to respond to every ping/traceroute their clients initiate. The real traffic they receive is also huge, so the numbers are relative. For example, they also receive huge amounts of troubleshooting traffic, which is not only their own direct clients' traffic, but also from all over the world.

You are right that ping delay is probably better, because it does not require responses from intermediate devices. Ping delay is closer to delay of forwarded traffic. As always, tools are tools. I could argue that real delay can be better than ping delay like this: When you download a file you can have many packets on the way between source and destination. The overall delay could be much lower than sending one packet and waiting for a response to send the next one. Of course, interactive traffic of non-file transfer applications might not be as good and depends more on end-to-end delay.

Bandwidth utilization is important to know and cannot be very easily measured (many overheads might not be reported by routing devices). If we assume that your numbers are accurate enough, then congestion due to bandwidth might not be an issue in your case (though over the internet you never know either the utilization of all the links or the cpu utilizations of the devices or anything). The traceroute over the VPN might look slower in traceroute because packets from intermediate devices might follow different return path for the different source addresses you use (i.e. your source addresses are destination addresses from the point of view of the routers in the path). Anyway, the most important thing is to determine whether the client does have connectivity or performance issues when browsing the site. Application performance is your real measure of whether a problem does exist. Also try to see if client has issues with only this site or this is more general. This detail is easily forgotten and I also forgot to mention it earlier. If client cannot see a thing then we move troubleshooting more towards the client side.