robots.txt Rerouting on Development Servers

By  on  

Every website should have a robots.txt file.  Some bots hit sites so often that they slow down performance, other bots simply aren't desirable.  robots.txt files can also be used to communicate sitemap location and limit request rate.  It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file.  Here's a quick .htaccess snippet you can use to make that happen:

RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]

The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint:  you really don't want this).

Recent Features

  • By
    Page Visibility API

    One event that's always been lacking within the document is a signal for when the user is looking at a given tab, or another tab. When does the user switch off our site to look at something else? When do they come back?

  • By
    Responsive and Infinitely Scalable JS Animations

    Back in late 2012 it was not easy to find open source projects using requestAnimationFrame() - this is the hook that allows Javascript code to synchronize with a web browser's native paint loop. Animations using this method can run at 60 fps and deliver fantastic...

Incredible Demos

  • By
    Multiple Background CSS Animations

    CSS background animation has been a hot topic for a long time, mostly because they look pretty sweet and don't require additional elements.  I was recently asked if it was possible to have multiple background animations on a given element and the answer is yes...with...

  • By
    AJAX For Evil:  Spyjax with jQuery

    Last year I wrote a popular post titled AJAX For Evil: Spyjax when I described a technique called "Spyjax": Spyjax, as I know it, is taking information from the user's computer for your own use — specifically their browsing habits. By using CSS and JavaScript, I...

Discussion

  1. Here’s an example showing how to include multiple development domains:

    RewriteCond %{HTTP_HOST} ^localhost [OR]
    RewriteCond %{HTTP_HOST} ^example.dev [OR]
    RewriteCond %{HTTP_HOST} ^test.example.com [OR]
    RewriteCond %{HTTP_HOST} ^staging.example.com
    RewriteRule ^robots.txt$ robots-disallow.txt [L]
    
  2. Steve

    use vagrant

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!