Thread: Google is Leeching Bandwidth!
View Single Post
  #8  
Old November 19th, 2004, 01:56 PM

Baron Munchausen Baron Munchausen is offline
General
  
Join Date: Aug 2000
Location: Ohio, USA
Posts: 4,323
Thanks: 0
Thanked 0 Times in 0 Posts
Baron Munchausen is on a distinguished road
Default Re: Google is Leeching Bandwidth!
Also see Google's own information page about their webcrawler:

http://www.google.com/bot.html

It's possible to set a config file (robots.txt) that keeps Googlebot from accessing certain things.
Reply With Quote