robots.txt Rerouting on Development Servers

By  on  

Every website should have a robots.txt file.  Some bots hit sites so often that they slow down performance, other bots simply aren't desirable.  robots.txt files can also be used to communicate sitemap location and limit request rate.  It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file.  Here's a quick .htaccess snippet you can use to make that happen:

RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]

The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint:  you really don't want this).

Recent Features

  • By
    39 Shirts – Leaving Mozilla

    In 2001 I had just graduated from a small town high school and headed off to a small town college. I found myself in the quaint computer lab where the substandard computers featured two browsers: Internet Explorer and Mozilla. It was this lab where I fell...

  • By
    Facebook Open Graph META Tags

    It's no secret that Facebook has become a major traffic driver for all types of websites.  Nowadays even large corporations steer consumers toward their Facebook pages instead of the corporate websites directly.  And of course there are Facebook "Like" and "Recommend" widgets on every website.  One...

Incredible Demos

  • By
    Facebook-Style Modal Box Using MooTools

    In my oh-so-humble opinion, Facebook's Modal box is the best modal box around. It's lightweight, subtle, and very stylish. I've taken Facebook's imagery and CSS and combined it with MooTools' awesome functionality to duplicate the effect. The Imagery Facebook uses a funky sprite for their modal...

  • By
    MooTools FontChecker Plugin

    There's a very interesting piece of code on Google Code called FontAvailable which does a jQuery-based JavaScript check on a string to check whether or not your system has a specific font based upon its output width. I've ported this functionality to MooTools. The MooTools...

Discussion

  1. Here’s an example showing how to include multiple development domains:

    RewriteCond %{HTTP_HOST} ^localhost [OR]
    RewriteCond %{HTTP_HOST} ^example.dev [OR]
    RewriteCond %{HTTP_HOST} ^test.example.com [OR]
    RewriteCond %{HTTP_HOST} ^staging.example.com
    RewriteRule ^robots.txt$ robots-disallow.txt [L]
    
  2. Steve

    use vagrant

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!