WordPress robots.txt file is a text file located at the root of your site that “tells search engine crawlers which URLs the crawler can access on your site”. Here is an example of a robots.txt file for a WordPress site that allows web crawlers to crawl the site and access all pages and files: User-agent: *Disallow: