What are Crawler Directives?

Crawler directives tell search engines how to behave when crawling and indexing your website. Examples of crawler directives are the robots meta tag, the x-robots tag and the robots.txt file.