Home / Tools
Tools

robots.txt Generator for WordPress

Assemble a robots.txt file for your WordPress site: set the sitemap rule, block the back end, limit AI crawlers and the internal search. Everything is computed in your browser; nothing is sent. Upload the finished file to your site's root directory. More on the role of robots.txt in technical SEO for WordPress.

The standard, open rule for all crawlers. Uncheck only if you want to block the whole site (e.g. a staging environment).

Blocks indexing of the entire site. Use for staging and test sites. Note: this is not password protection, only a request to crawlers.

The standard WordPress rule: blocks the back end but keeps access to admin-ajax.php, which many plugins need.

Prevents indexing of search result pages, which create thin content and waste crawl budget.

Hides readme.html and license.txt, which can reveal the WordPress version.

Blocks known bots that collect content for training AI models (GPTBot, CCBot, Google-Extended, ClaudeBot and others). Note: this is a voluntary request and not every bot honors it.


                    
Place the robots.txt file in the root directory (available at /robots.txt). Remember: robots.txt is a hint for crawlers, not a security measure. To hide content use a password (.htpasswd) or noindex. Blocking in robots.txt does not remove a page from the index if it is already there.

                

            

        

    


    

        

            

                
What robots.txt is and how it works

                
robots.txt is a simple text file placed in the site's root directory that tells search engine crawlers what they should not visit or index, and points to the sitemap address. Crawlers fetch it as one of the first resources and treat it as a set of hints. Important: it is a hint, not a security measure. Only well-behaved bots respect the rules, and a blocked URL may still end up in the index if links point to it. A well-configured file does, however, help save crawl budget and keep pages that should not appear out of the results. It is a part of technical SEO for WordPress.

            


            

                
What is worth configuring in WordPress

                

                    

                        Block wp-admin
                        The standard: block the /wp-admin/ directory but leave access to /wp-admin/admin-ajax.php, which many plugins and themes use. The generator sets both conditions automatically.
                    

                    

                        Sitemap address
                        Pointing to the full sitemap URL helps crawlers discover subpages. If you use an SEO plugin, use its sitemap address (e.g. /sitemap_index.xml). The sitemap is a pillar of indexing.
                    

                    

                        Internal search
                        Blocking search result URLs (?s=) prevents indexing hundreds of thin, repetitive pages that waste crawl budget and weaken the site's quality in Google's eyes.
                    

                    

                        AI crawlers
                        If you do not want your content feeding AI models, you can add rules for bots like GPTBot or CCBot. It is a voluntary request that only standards-respecting bots honor.
                    

                    

                        Staging environment
                        For a staging site use a full block (Disallow: /), and ideally also protect it with a password via .htpasswd. robots.txt alone will not hide content from someone who knows the address.
                    

                

            


            

                
Frequently asked questions about robots.txt

            

            

                
A text file in the site's root that tells crawlers what not to index and points to the sitemap. It is a hint, not a security measure, so it does not protect content from access.

                
Only partially. Disallow asks crawlers not to visit a URL, but with external links the page may still reach the index without content. For real removal use noindex or a password.

                
Good practice, except for admin-ajax.php which plugins need. The standard blocks /wp-admin/ and allows /wp-admin/admin-ajax.php. The generator sets it up automatically.

                
Yes, with Disallow rules for bots like GPTBot, CCBot, Google-Extended or ClaudeBot. It is a voluntary request honored only by bots that respect the robots.txt standard.

                
In the root directory of the domain, at yourdomain.com/robots.txt. Each subdomain has its own file. A physical file in the root takes precedence over the virtual one from WordPress.

            


            

                
Tool and write-up: Marcin Szewczyk-Wilgan, WebOptimo. Rules consistent with the robots.txt standard and WordPress indexing practice. The generator runs in the browser, with no data sent.

            

        

    


    
Contact
We help with WordPress technical SEO
We configure indexing, sitemaps, structured data and performance for WordPress sites. No obligation — a concrete proposal after a short conversation.
Phone
+48 608 271 665
Mon–Fri, 8:00–16:00
E-mail
kontakt@weboptimo.pl
We reply within 24h
Company
WebOptimo
VAT ID: PL6391758393











	Marcin Szewczyk-Wilgan - m4c.pl  Powered by FreeBSD  Edited in Vim