🤖

Robots.txt Generator

Generate a robots.txt file to control search engine crawling.

Rule Block 1

User-agent

Allow paths (one per line)

Disallow paths (one per line)

Crawl-delay (seconds, optional)

Sitemap URL

Generated robots.txt

User-agent: *
Allow: /

How to Use This Tool

A robots.txt file tells search engine crawlers which pages they can and cannot access on your site. It is placed in the root directory of your website.

Set the User-agent (use * for all search engines)
Add Allow and Disallow paths
Optionally add a Sitemap URL
Copy the generated file and save as robots.txt in your website root

FAQ

What is robots.txt?

Robots.txt is a text file placed at the root of a website that tells web crawlers which pages to crawl and which to skip.

Where do I put robots.txt?

Place the robots.txt file in the root directory of your website, accessible at yourdomain.com/robots.txt.