Robots.txt file for WordPress – plugin

robots.txt

Making the right robots.txt file for WordPress is very important. Thanks to him, search engines will know which pages to show in the search. That is, the search results will be just what you need, without duplicating WordPress pages.

Correct robots.txt file

User-Agent: *
Disallow: /cgi-bin 
Disallow: /wp-
Disallow: /?s=
Disallow: *&s=             
Disallow: /search          
Disallow: /author/         
Disallow: *?attachment_id= 
Allow: /wp-content/uploads/
Allow: /wp-content/themes/
Allow: /*/*.js             
Allow: /*/*.css            
Allow: /wp-*.png           
Allow: /wp-*.jpg           
Allow: /wp-*.jpeg          
Allow: /wp-*.gif           
Allow: /wp-*.svg           
Allow: /wp-*.pdf 

Sitemap: https://site.com/sitemap_index.xml

How to add a robots.txt file to WordPress

You can download our plugin Robots.txt Editor:

Features

  • Works with multisite network on Subdomains;
  • An example of the correct file for WordPress;
  • Works out of the box;
  • Totally Free.

Or add the following code to the functions.php file:

add_filter( 'robots_txt', 'add_robotstxt' );
function add_robotstxt($text){
    $site_url = get_site_url();
    $text .= "User-Agent: *
Disallow: /cgi-bin 
Disallow: /wp-
Disallow: /?s=
Disallow: *&s=             
Disallow: /search          
Disallow: /author/         
Disallow: *?attachment_id= 
Allow: /wp-content/uploads/
Allow: /wp-content/themes/
Allow: /*/*.js             
Allow: /*/*.css            
Allow: /wp-*.png           
Allow: /wp-*.jpg           
Allow: /wp-*.jpeg          
Allow: /wp-*.gif           
Allow: /wp-*.svg           
Allow: /wp-*.pdf 

Sitemap: $site_url/sitemap_index.xml";

    return $text;
}

This method works in the multisite version of WordPress. For each site network will display the correct Sitemap. 

The robots.txt file does not guarantee that the pages that are closed in it will not be indexed, therefore, duplicate content must be additionally hidden using 301 redirects in the .htaccess file:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} ^replytocom= [NC]
RewriteRule (.*) $1? [R=301,L]
RewriteRule ^feed$ /$1 [R=301,L]
RewriteRule (.*)/feed$ /$1 [R=301,L]
RewriteRule (.+)/comment-page /$1 [R=301,L]
RewriteRule (.+)/trackback /$1 [R=301,L]
RewriteRule (.+)/comments /$1 [R=301,L]
RewriteRule (.+)/attachment /$1 [R=301,L]
RewriteCond %{QUERY_STRING} ^attachment_id= [NC]
RewriteRule (.*) $1? [R=301,L]
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Thanks to the Yoast SEO plugin, .htaccess files can be edited from the WordPress admin panel (menu item "Tools" -> "File Editor").


Updated: 06.05.2019

Did this article help you? Rate it!
(2 votes, average: 5.00 out of 5)
Loading...


Leave a Reply

Your email address will not be published. Required fields are marked *