Disallow Robots Using Robots.txt
I develop customer websites on a publicly accessible web server so that my customers may check the progress of their website at any given time. I could use .htaccess to require username and password for the site but then I'm constantly needing to remind customers what their password is. My big concern is preventing search engines from finding their way to my development server. Luckily I can add a robots.txt file to my development server websites that will prevent search engines from indexing them.
The Robots.txt
User-agent: * Disallow: /
The above directive prevents the search engines from indexing any pages or files on the website. Say, however, that you simply want to keep search engines out of the folder that contains your administrative control panel. You'd code:
User-agent: * Disallow: /administration/
Or if you wanted to allow in all spiders except Google's GoogleBot, you'd code:
User-Agent: googlebot Disallow: /
What would you prevent the search engines from seeing?
Comments
Be Heard!
Share your thoughts without being a jerk! And wrap your code in <code> tags, f00!
Sweet! Now I can block all the back-up’d pronz :p
Nice article David, but we must be extra careful not to block Google from legit content!
There is one small problem with this. robot.txt is often a hotlist for hackers. If someone wants to hack your site, robot.txt shows them all the best spots.
When I was getting my blog ready, Google somehow found my public testing server (which was hosted in my basement no less) and decided to index a couple of my test posts. I have used Perishable Press’ User Agent Blacklist since then and for the most part bots haven’t broken through.
(Sorry if this posted twice, I submitted the first time but it seems that wp-post timed out.)
Binny V A: Good point. I believe a nice .htaccess hack would take care of that.
As I understand it, the robots.txt file doesn’t actually prevent search engines from indexing you, it just tells them not to. If they want to index you they can just ignore the file.
http://www.robotstxt.org/faq/blockjustbad.html
Mike
Binny is correct… but that shouldn’t stop you from using robot.txt!
I block all my cms folders and private stuff using robot.txt
Hey David, How can you protect the file using .htaccess ? I believe this will block the searchbots too right?
Maybe this would help:
Aren’t you relying on spider programs actually following the rules? What prevents someone from writing a spider which ignores your robots.txt file and actually indexing your entire site?
It appears some are thinking this post is about how to secure your site …
@Binny VA, there’s no reason you can’t sprinkle in a few random directories to mislead your hacker audience. A little light security by obscurity.
User-agent: *
Disallow: /adm/
Disallow: /administration/
Disallow: /admin/
Disallow: /adminportal/
Disallow: /drupal/
Disallow: /joomla/
Thank you, Ive been looking for this info for 3 hours now.
Ryan’s correct, this post isn’t about securing your site.
I had never considered doing a .htaccess hack to exclude items. What a marvelous idea!
Thanks for that. I am going to work on implementing it ASAP.
If you block the robots.txt file using .htaccess rules, not only the hackers wouldn’t be able to read it, but also the search bots. That means it’s useless to do it – it would be easier to simply delete robots.txt in that case.
Hi, I need to know how to block ROBOTS.txt using HTACCESS, means I want to block robots.txt from spiders but from htaccess.
One should be careful not to rely on robots.txt as a means of security. It offers no security.
The robots.txt file is only a directive to cooperating web crawlers/bots on what to index. The file(s) you’re trying to restrict the access of are still publicly accessible.
If you don’t want a part of your website to be publicly accessible then password protect it.