Popular:
 Building Resilient Systems on AWS: Learn how to design and implement a resilient, highly available, fault-tolerant infrastructure on AWS.

robots.txt Rerouting on Development Servers

By  on  

Every website should have a robots.txt file.  Some bots hit sites so often that they slow down performance, other bots simply aren't desirable.  robots.txt files can also be used to communicate sitemap location and limit request rate.  It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file.  Here's a quick .htaccess snippet you can use to make that happen:

RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]

The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint:  you really don't want this).

Request Metrics real user monitoring
Request Metrics real user monitoring
Request Metrics real user monitoring
Request Metrics real user monitoring

Recent Features

Incredible Demos

  • By
    WebKit Marquee CSS: Bringin’ Sexy Back

    We all joke about the days of Web yesteryear.  You remember them:  stupid animated GIFs (flames and "coming soon" images, most notably), lame counters, guestbooks, applets, etc.  Another "feature" we thought we had gotten rid of was the marquee.  The marquee was a rudimentary, javascript-like...

  • By
    MooTools 1.2 OpenLinks Plugin

    I often incorporate tools into my customers' websites that allow them to have some control over the content on their website. When doing so, I offer some tips to my clients to help them keep their website in good shape. One of the tips...

Discussion

  1. Fervor Creative

    Here’s an example showing how to include multiple development domains:

    RewriteCond %{HTTP_HOST} ^localhost [OR]
RewriteCond %{HTTP_HOST} ^example.dev [OR]
RewriteCond %{HTTP_HOST} ^test.example.com [OR]
RewriteCond %{HTTP_HOST} ^staging.example.com
RewriteRule ^robots.txt$ robots-disallow.txt [L]
  2. Steve

    use vagrant

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!