Robots.txt Generator

Create a customised robots.txt file with crawler directives, WordPress rules, path controls and sitemap instructions. Review your rules before publishing.

Website Configuration

Add your website and sitemap information.

Website URL *

Enter your preferred canonical website URL.

Sitemap URL

Add your XML sitemap or sitemap index URL.

Default behaviour: When no blocking rule is specified, crawlers are generally allowed to access URLs. You do not need to add an Allow rule for every public page.

Crawler Settings

Choose which crawler groups should receive your rules.

All standard crawlers (*)General rules for crawlers using the wildcard user-agent. GooglebotGoogle’s main web crawler. BingbotMicrosoft Bing’s search crawler. Googlebot-ImageGoogle’s image crawler.

Important: Specific crawler groups do not automatically inherit the rules in the wildcard (*) group in Google’s robots.txt interpretation. Select crawler groups carefully and review the generated output.

Common Website Rules

Select only the paths you actually want to restrict.

WordPress Admin/wp-admin/ — administrative area. WordPress Login/wp-login.php — login page. Shopping Cart/cart/ — common ecommerce cart path. Checkout/checkout/ — common checkout path. My Account/my-account/ — common account path. Internal Search/search/ — only if this is your actual search URL. CGI Bin/cgi-bin/ — if used by your website. Private Directory/private/ — only a crawl restriction, not security.

These are optional path examples, not universal SEO recommendations. Blocking a URL does not protect private data or guarantee that a URL will disappear from search results.

Custom Crawl Rules

Add your own Allow or Disallow directives.

Paths must start with /. Example: /wp-content/uploads/ or /sample-page/

Advanced Options

Add optional directives and comments to your robots.txt file.

File comment (optional)

Crawl-delay (optional)

Crawl-delay compatibility: Google does not support Crawl-delay. Some other crawlers may interpret it differently. This option is disabled by default and should only be used when you understand your target crawler’s support.

Generated Robots.txt

Review, edit, copy or download your generated file.

0Lines

0Path rules

0 BFile size

Robots.txt content

Before publishing: Review every directive. The generated file is a draft; this tool does not upload it to your server or check your live robots.txt file.

Robots.txt controls compliant crawler access; it is not a password, security mechanism or guaranteed deindexing method. Google supports User-agent, Allow, Disallow and Sitemap. Check crawler-specific compatibility before using additional directives.