What are Crawler Directives?
Free Hub Resources
Free Hub Resources
Crawler directives tell search engines how to behave when crawling and indexing your website. Examples of crawler directives are the robots meta tag, the x-robots tag and the robots.txt file.