What is Robots.txt

Robots.txt - or the Robots Exclusion Protocol - is a text file that tells search engine crawlers of any specific website areas which they can access.