Suboptimal routing for Google crawler traffic?

Unanswered Question
Oct 14th, 2008


We have a customer who's business hinges a lot on Google page rank (nothing new). They are very focused on latency to the Google crawler farms which of course randomly change. They are constantly sending us traceroutes to the bot farms and when they go through Verizon, our primary upstream provider, the traffic to the farms hop across the pond and end up in Europe. However, when they trace to these same farms on another network, they remain domestic routes and stay within the US. Of course there is much lower latency because of this and they believe their page rank is directly affected by the latency to the bot farms crawling their sites.

So I have a few questions:

1) Can anyone validate if latency would affect their page rank? I know it's probably part of Googles secret sauce but thought I would ask.

2) Has anyone had any similar experience? I think Google is obviously doing this on purpose but that doesn't help me respond to my customer. Looking at the routes, it's obvious that the route they are taking is the one Google is preferencing to Verizon. It looks like Google is padding domestic routes to Verizon on purpose specifically for some of these bot farms.

3) Being that it looks like Google is pushing traffic from Verizon to Europe in some situations, it seems odd to me that this latency would affect their page rank. If that's the case and Google didn't have the intelligence built in to compesate for diverse geographical distribution, it wouldn't work as well as it does.

Any ideas on where to go here? We have temporarily preferenced some of the crawler farm locations but they're constantly changing so this obviously isn't maintainable. Thanks for any help you can provide.


I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)


This Discussion