Definition of Robots.txt

Trial Download
Search Dictionary:
Search Web Search Dictionary
Robots.txt Definition from Encyclopedia Dictionaries & Glossaries
Wikipedia Dictionaries
Wikipedia English - The Free Encyclopedia
The robots exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is, otherwise, publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard complements Sitemaps, a robot inclusion standard for websites.

See more at Wikipedia.org...
This article uses material from Wikipedia® and is licensed under the GNU Free Documentation License