Continue reading "How to stop Googlebot eating up huge bandwidth"
How to stop Googlebot eating up huge bandwidth
Being a webmaster, I faced the problem of Google Crawler aka Googlebot eating up huge bandwidth on more than a few occasions. In worst case, I even faced a 5GB bandwidth usage by google bot alone. While other search engine bots like Bing bot, Yahoo bot etc. seemed taking nominal bandwidth compared to google bot, …