Robots.txt
Please Note:
The text on this page was translated by translation software. A revised version from our editors will be available soon.
The file robots.txt is a text file that can be stored on the server of a website. It determines whether and which subpages of the website may be visited by search engine crawlers and included in their index.
This allows certain subpages to be excluded from the search result.
Example: You can use the file robots.txt to exclude all archive contents of a website from the search result.
However, some search engine crawlers ignore the robots.txt file. Therefore, subpages that must not appear in search engines should be password protected.