robots.txt Rerouting on Development Servers

By  on  

Every website should have a robots.txt file.  Some bots hit sites so often that they slow down performance, other bots simply aren't desirable.  robots.txt files can also be used to communicate sitemap location and limit request rate.  It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file.  Here's a quick .htaccess snippet you can use to make that happen:

RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]

The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint:  you really don't want this).

Recent Features

  • By
    CSS 3D Folding Animation

    Google Plus provides loads of inspiration for front-end developers, especially when it comes to the CSS and JavaScript wonders they create. Last year I duplicated their incredible PhotoStack effect with both MooTools and pure CSS; this time I'm going to duplicate...

  • By
    Camera and Video Control with HTML5

    Client-side APIs on mobile and desktop devices are quickly providing the same APIs.  Of course our mobile devices got access to some of these APIs first, but those APIs are slowly making their way to the desktop.  One of those APIs is the getUserMedia API...

Incredible Demos

  • By
    jQuery Random Link Color Animations

    We all know that we can set a link's :hover color, but what if we want to add a bit more dynamism and flair? jQuery allows you to not only animate to a specified color, but also allows you to animate to a random color. The...

  • By
    Create a Simple Dojo Accordion

    Let's be honest:  even though we all giggle about how cheap of a thrill JavaScript accordions have become on the web, they remain an effective, useful widget.  Lots of content, small amount of space.  Dojo's Dijit library provides an incredibly simply method by which you can...

Discussion

  1. Here’s an example showing how to include multiple development domains:

    RewriteCond %{HTTP_HOST} ^localhost [OR]
    RewriteCond %{HTTP_HOST} ^example.dev [OR]
    RewriteCond %{HTTP_HOST} ^test.example.com [OR]
    RewriteCond %{HTTP_HOST} ^staging.example.com
    RewriteRule ^robots.txt$ robots-disallow.txt [L]
    
  2. Steve

    use vagrant

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!