Skip to main content

Article | 10 Ways to Increase Your Site Crawl Rate

Popularity Report

Total Popularity Score: 0
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...

Rank

URL Tag Cloud

Bookmark History

Saved by 1 people (0 private), first by anonymouse user on 2008-10-11


Public Sticky notes

Highlighted by suzannah

on 2008-10-12 by suzannah

Smarty, Ann. 2008. 10 Ways to Increase Your Site Crawl Rate. Blog. Search Engine Journal. June 24. http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/.

Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Highlighted by suzannah

Note that you can’t force Googlebot to visit you more often - what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

Highlighted by suzannah

Update your content often and regularly (and ping Google once you do) - well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).

Highlighted by suzannah

Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.

Highlighted by suzannah

Mind your page load time: note that the crawl works on a budget - if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.

Highlighted by suzannah

Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.

Highlighted by suzannah

Get more back links from regularly crawled sites.

Highlighted by suzannah

Adjust the crawl speed via Google Webmaster tools.

Highlighted by suzannah

Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).

Highlighted by suzannah

Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.

Highlighted by suzannah

Make sure you have unique title and meta tags for each of your pages.

Highlighted by suzannah

Monitor Google crawl rate for your site and see what works and what not:

Highlighted by suzannah

access crawl stats via Google Webmaster tools:

Highlighted by suzannah

crawl rate via Google Webmaster tools

Highlighted by suzannah

Take advantage of this great Wordpress Plugin that tracks crawl rate for Google, Yahoo and MSN:

Highlighted by suzannah

crawl rate tracker

Highlighted by suzannah

Readers (1)