The Marketing Dictionary

SEO and Digital Marketing terms and definitions you need to know.
Marketing dictionary
Back to Marketing Dictionary

Crawler directives tell search engines how to behave when crawling and indexing your website.

Examples of crawler directives are the robots meta tag, the x-robots tag and the robots.txt file.

NO COMMITMENT
GET A 14-DAYS FREE TRIAL

ALL FEATURES INCLUDED - NO CREDIT CARD NEED

Loved by 30.000 Agencies & Brands All Over The World