Robots.txt
A small text file in your website's root directory. It tells search engine crawlers which areas they should and shouldn't crawl. Essentially, it's your house rules for automated visitors. Important: It keeps friendly bots on the right track, but it doesn't block attackers. For real protection, you need access restrictions and security settings.
The robots.txt file tells search engines which areas they should not crawl. It is a request, not a block, and above all, it is not protection: The file is publicly readable, so anyone who lists secret directories there has effectively made them public.
The most common misconception: A page blocked via robots.txt can still end up in the index if other pages link to it. Google then displays it without a description. To truly keep a page out of the index, you need a noindex directive in the page header, and for this to work, Google must be allowed to access the page. Blocking and using noindex simultaneously are therefore mutually exclusive.
Back to the glossary