robots.txt Rerouting on Development Servers
Every website should have a robots.txt file. Some bots hit sites so often that they slow down performance, other bots simply aren't desirable. robots.txt files can also be used to communicate sitemap location and limit request rate. It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file. Here's a quick .htaccess snippet you can use to make that happen:
RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]
The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint: you really don't want this).
![Regular Expressions for the Rest of Us]()
Sooner or later you'll run across a regular expression. With their cryptic syntax, confusing documentation and massive learning curve, most developers settle for copying and pasting them from StackOverflow and hoping they work. But what if you could decode regular expressions and harness their power? In...
![5 Awesome New Mozilla Technologies You’ve Never Heard Of]()
My trip to Mozilla Summit 2013 was incredible. I've spent so much time focusing on my project that I had lost sight of all of the great work Mozillians were putting out. MozSummit provided the perfect reminder of how brilliant my colleagues are and how much...
![Truly Responsive Images with responsive-images.js]()
Responsive web design is something you hear a lot about these days. The moment I really started to get into responsive design was a few months ago when I started to realise that 'responsive' is not just about scaling your websites to the size of your...
![CSS Tooltips]()
We all know that you can make shapes with CSS and a single HTML element, as I've covered in my CSS Triangles and CSS Circles posts. Triangles and circles are fairly simply though, so as CSS advances, we need to stretch the boundaries...
Here’s an example showing how to include multiple development domains:
RewriteCond %{HTTP_HOST} ^localhost [OR] RewriteCond %{HTTP_HOST} ^example.dev [OR] RewriteCond %{HTTP_HOST} ^test.example.com [OR] RewriteCond %{HTTP_HOST} ^staging.example.com RewriteRule ^robots.txt$ robots-disallow.txt [L]use vagrant