Comments on: Disallow Robots Using Robots.txt

By: Lucas

Lucas — Thu, 06 Oct 2011 09:38:07 +0000

One should be careful not to rely on robots.txt as a means of security. It offers no security.
The robots.txt file is only a directive to cooperating web crawlers/bots on what to index. The file(s) you’re trying to restrict the access of are still publicly accessible.

If you don’t want a part of your website to be publicly accessible then password protect it.

By: Vipin Sharma

Vipin Sharma — Sat, 01 Oct 2011 07:38:46 +0000

Hi, I need to know how to block ROBOTS.txt using HTACCESS, means I want to block robots.txt from spiders but from htaccess.

By: Dmitri Ponomarjov

Dmitri Ponomarjov — Sun, 18 Sep 2011 10:54:18 +0000

If you block the robots.txt file using .htaccess rules, not only the hackers wouldn’t be able to read it, but also the search bots. That means it’s useless to do it – it would be easier to simply delete robots.txt in that case.

By: Omega-3's

Omega-3's — Sun, 27 Jun 2010 13:50:20 +0000

I had never considered doing a .htaccess hack to exclude items. What a marvelous idea!

Thanks for that. I am going to work on implementing it ASAP.

By: Obri

Obri — Sun, 21 Feb 2010 11:00:59 +0000

Ryan’s correct, this post isn’t about securing your site.

By: Web Host Directory

Web Host Directory — Wed, 16 Sep 2009 00:56:30 +0000

Thank you, Ive been looking for this info for 3 hours now.

By: Mike V - RaleighNC

Mike V - RaleighNC — Wed, 08 Jul 2009 12:45:24 +0000

@Binny VA, there’s no reason you can’t sprinkle in a few random directories to mislead your hacker audience. A little light security by obscurity.

User-agent: *
Disallow: /adm/
Disallow: /administration/
Disallow: /admin/
Disallow: /adminportal/
Disallow: /drupal/
Disallow: /joomla/

By: Ryan Florence

Ryan Florence — Mon, 06 Jul 2009 01:56:24 +0000

It appears some are thinking this post is about how to secure your site …

By: Dave

Dave — Sun, 05 Jul 2009 21:27:01 +0000

Aren’t you relying on spider programs actually following the rules? What prevents someone from writing a spider which ignores your robots.txt file and actually indexing your entire site?

By: digital

digital — Sun, 05 Jul 2009 11:57:45 +0000

Maybe this would help:

order allow,deny
deny from all