The Marketing Dictionary

SEO and Digital Marketing terms and definitions you need to know.
Marketing dictionary
Back to Marketing Dictionary

A robots.txt file tells search engine crawlers which URLs the crawler can or cannot access on a website. It’s mainly used to prevent search engines from wasting crawl budgets on unimportant pages. The robots.txt file doesn’t, however, prevent webpages from being indexed by search engines.

NO COMMITMENT
GET A 14-DAYS FREE TRIAL

ALL FEATURES INCLUDED - NO CREDIT CARD NEED

Loved by 30.000 Agencies & Brands All Over The World