Robots

Web directories and search engines use robots to traverse web documents for inclusion in their search indexes. Web robots are sometimes referred to as web wanderers, crawlers, or spiders. Webmasters can deploy a robots.txt file to their website as a way to inform visiting robots which parts of their site should be included in its index. Learn more about robots at www.robotstxt.org.
More Website and Internet