Oct162009Server Hardware, Googlebot and SEO
As the prevalance of more and more web pages strewn about the Internet as well as increasingly larger web sites becomes commonplace, the level of investment you make in web site hardware can actually have implications on your search engine optimization.
Operators of very large sites have often assumed that powerful servers or multi-server solutions, along with excessive bandwidth, are necessary to ensure performance during spikes in visitor traffic and activity. E-commerce web site operators have long known that each additional second that a customer waits for a shopping cart page to load can lead to a 5% – 10% loss in conversion at that step.
The above are well known facts that impact web site hardware and bandwidth selection.
But SEO?
As it turns out, Googlebot, the Google search engine spider, is a prime culprit of server and bandwidth resource theft. The more external links to your web site, or internal links between pages, the more likely you are to receive a visit from Googlebot or a host of other search engine spiders. As the number of pages on your site increases, and assuming that you have interlinked your pages using SEO best practices, the number of potential paths that a spider can take through your site increases exponentially. The larger the site and the more complex the linking between pages, the more hits that spiders will make on your site and the more bandwidth, memory and CPU resources will be consumed. We have witnessed sites with over 10,000 pages exceed bandwidth limitation due solely to a combination of good SEO and limited hardware resoures.
There are ways to reduce Googlebot and other spiders’ visits in order to accommodate hardware and bandwidth limitations, but from an SEO perspective, living with IT constraints is not a good option since webmasters should desire that spiders to visit their websites as often as possible.
How to Solve / Anticipate Spiders’ Impact on your Servers
- Host a site with a lot of pages (+10,000) and complex interlinking of pages (average of 20 internal links per page or more with a non-heirarchical linking strategy) on a dedicated server to ensure the best performance and handle many spider visits.
- Install Google Webmaster Tools on your websites; you can control the rate at which the Googlebot visits your site.
