Robots.txt file for wordpress

Making the right robots.txt file for WordPress is very important. Thanks to him, search engines will know which pages to show in the search. That is, the search results will be just what you need, without duplicating WordPress pages.

Correct robots.txt file

User-Agent: *
Disallow: /wp-
Disallow: /xmlrpc.php
Disallow: /?s=
Allow: /wp-content/uploads/
Allow: /wp-content/themes/
Allow: *.js 
Allow: *.css
Allow: /wp-content/plugins/*.gif
Allow: /wp-content/plugins/*.png

Sitemap: https://site.ru/sitemap_index.xml

A sitemap will be this if you use the Yoast SEO plugin.

The robots.txt file does not guarantee that the pages that are closed in it will not be indexed, therefore, duplicate content must be additionally hidden using 301 redirects in the .htaccess file:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} ^replytocom= [NC]
RewriteRule (.*) $1? [R=301,L]
RewriteRule ^feed$ /$1 [R=301,L]
RewriteRule (.*)/feed$ /$1 [R=301,L]
RewriteRule (.+)/comment-page /$1 [R=301,L]
RewriteRule (.+)/trackback /$1 [R=301,L]
RewriteRule (.+)/comments /$1 [R=301,L]
RewriteRule (.+)/attachment /$1 [R=301,L]
RewriteCond %{QUERY_STRING} ^attachment_id= [NC]
RewriteRule (.*) $1? [R=301,L]
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Thanks to the Yoast SEO plugin, robots.txt and .htaccess files can be edited from the WordPress admin panel (menu item "Tools" -> "File Editor").

For WordPress multisite, there is a https://wordpress.org/plugins/multisite-robotstxt-manager/ plugin that allows you to create a virtual robots.txt file for each site from a network of sites.


Updated: 11.01.2019

Did this article help you? Rate it!
(No Ratings Yet)
Loading...


Leave a Reply

Your email address will not be published. Required fields are marked *