Being a webmaster, I faced the problem of Google Crawler aka Googlebot eating up huge bandwidth on more than a few occasions. In worst case, I even faced a 5GB bandwidth usage by google bot alone. While other search engine bots like Bing bot, Yahoo bot etc. seemed taking nominal bandwidth compared to google bot, made the matter yet more confusing. Of course, it is very easy to stop Googlebot from eating up huge bandwidth by simply disallowing it – but you can’t do that. Because, it would make your website invisible to the web world.
Here are a few things you can do to prevent Googlebot eating up too much bandwidth without stopping it from indexing or crawling your site and keeping your PR, i.e., Page Rank same as before.
How to stop Googlebot eating up huge bandwidth
1. Sign In to your Google Adsense Account
2. Click on Site Diagnostics on Top
3. Click on Google Webmaster Central at bottom
4. Click on Sign In To Webmaster Tools in the page that appears.
5. Now on the left click on Site Configuration and then Settings.
6. Look for the Crawl Rate – You’ll see “Let Google determine my crawl rate (recommended)” is selected by default.
7. Click on ‘Set Custom Crawl Rate’. Move the slider to left direction and click save.
8. You’re done! Your google bot bandwidth problem is solved.
However, you know a big step always leaves a big gap as well – so, it would be wise to check the following.
Bandwidth Issues with GoogleBot
1. Do your site use Session Ids? Then check for session related problems first. Generally google bot takes too much bandwidth when it gets stuck on a script with sessions Ids.
2. If you just have a huge website with lots of contents and if you maintain fresh content google bot will be visiting your site more frequently eating up more of your bandwidth. Not only that, but google bot also has a knack for re-indexing pages that don’t change between crawling sessions. I know that’s good news – but if google bot is using up most of your bandwidth – then you may want to prevent it effectively.
3. Is Google indexing your multimedia (which is the common case with e-commerce sites)? If this is the case, you can stop it from indexing anything but your text content with a couple of lines in your robots.txt file.
If all of your media are contained in a different folder than where your contents are stored, you can easily tell bots not to look inside of those folders. However, if you’ve embedded media into pages, it might be in the course of indexing text that the bandwidth gets chewed automatically.
If you have a template page that calls in videos, you may want to ban Search Engines from indexing that page.
4. If you’ve done the above and still the problem persists then try to put the next 2 lines of code in robots.txt
User-agent: Googlebot
Crawl-delay: 10
Google bot still be visiting your site every day, but with a lower bandwidth usage.
5. I know you don’t want to piss your google bot off and screw your unique traffic by experimenting things. So, if you are confused about what to do, the you should contact google about the matter.