robots.txt Rerouting on Development Servers
Every website should have a robots.txt file. Some bots hit sites so often that they slow down performance, other bots simply aren't desirable. robots.txt files can also be used to communicate sitemap location and limit request rate. It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file. Here's a quick .htaccess snippet you can use to make that happen:
RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]
The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint: you really don't want this).
![5 HTML5 APIs You Didn’t Know Existed]()
When you say or read "HTML5", you half expect exotic dancers and unicorns to walk into the room to the tune of "I'm Sexy and I Know It." Can you blame us though? We watched the fundamental APIs stagnate for so long that a basic feature...
![9 Mind-Blowing WebGL Demos]()
As much as developers now loathe Flash, we're still playing a bit of catch up to natively duplicate the animation capabilities that Adobe's old technology provided us. Of course we have canvas, an awesome technology, one which I highlighted 9 mind-blowing demos. Another technology available...
![MooTools Wall Plugin]()
One of the more impressive MooTools plugins to hit the Forge recently was The Wall by Marco Dell'Anna. The Wall creates an endless grid of elements which can be grabbed and dragged, fading in elements as they are encountered. Let me show...
![Using Opacity to Show Focus with MooTools]()
I'm a huge fan of using subtle effects like link nudging (jQuery, MooTools) to enhance the user experience and increase the perceived dynamism of my websites. Trust me -- a lot of little things are what take websites to the next level.
Here’s an example showing how to include multiple development domains:
RewriteCond %{HTTP_HOST} ^localhost [OR] RewriteCond %{HTTP_HOST} ^example.dev [OR] RewriteCond %{HTTP_HOST} ^test.example.com [OR] RewriteCond %{HTTP_HOST} ^staging.example.com RewriteRule ^robots.txt$ robots-disallow.txt [L]use vagrant