Crawler directives tell search engines how to behave when crawling and indexing your website.Examples of crawler directives are the robots meta tag, the x-robots tag and the robots.txt file.