Many people does not know the importance of robots.txt file. The robots.txt file is very important as per search engine and SEO prospective.
What is robots.txt file?
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
In wordpress you should put robots.txt file. you can use following code in robots.txt file and put that file under your ROOT directory( wordpress main folder)
User-agent: * Allow: / Disallow: /*?s= Disallow: /wp-admin/* Disallow: /wp-content/* Disallow: /wp-includes/* Disallow: /wp-content/cache Disallow: /wp-content/themes/*</code><code> Disallow: /wp-content/plugins/*</code>
Important Note: if you are using the QA or testing enviourment then put following lines in your theme header.php file. You can put the domain name condition also.
<?php
if($SERVER)['HTTP_HOST']=='Testing domain name') {
?>
<meta name='robots' content='noindex,nofollow' />
<?php } ?>
You should not disallow the images and comments also because that is more important for SEO purpose. I recommend to use always good permalink structure for your wordpress installation.
I have also "Sitemap:http://suckup.de/sitemap.xml" in my robots.txt is this wrong, old style or something like this ???
I included your blog to my RSS feed, thanks for the article! Hoping for another article soon! Thanks!
I would like to say “wow” what a inspiring post. This is really great. Keep doing what you’re doing!!
Interesting read, perhaps the best article iv’e browse today. We learn everyday cheers to you!
I’ve just started off a blog, the knowledge you give on this site has aided me extremely. Thank you for all your time & work.
I’ve been checking your blog for a while now, seems like everyday I learn something new
Thanks
Awesome post. I so good to see someone taking the time to share this information
Good! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog?
Nice post! You truly have a wonderful way of writing which I find captivating! I will definitely be bookmarking you and returning to your blog. In fact, your post reminded me about a strange thing that happened to me the other day. I’ll tell you about that later…
Of course, what a great site and informative posts, I will add backlink – bookmark this site? Regards, Reader
Great read. Thanks for the info!
Good! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog?
Hello, this is my first time i visit here. I found so many interesting in your blog especially on how to determine the topic. keep up the good work.
I was just having a conversation over this I am glad I came across this it cleared some of the questions I had.
Of course, what a great site and informative posts, I will add backlink – bookmark this site? Regards, Reader