robots.txt Rerouting on Development Servers
Every website should have a robots.txt file. Some bots hit sites so often that they slow down performance, other bots simply aren't desirable. robots.txt files can also be used to communicate sitemap location and limit request rate. It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file. Here's a quick .htaccess snippet you can use to make that happen:
RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]
The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint: you really don't want this).
![5 Awesome New Mozilla Technologies You’ve Never Heard Of]()
My trip to Mozilla Summit 2013 was incredible. I've spent so much time focusing on my project that I had lost sight of all of the great work Mozillians were putting out. MozSummit provided the perfect reminder of how brilliant my colleagues are and how much...
![Introducing MooTools Templated]()
One major problem with creating UI components with the MooTools JavaScript framework is that there isn't a great way of allowing customization of template and ease of node creation. As of today, there are two ways of creating:
new Element Madness
The first way to create UI-driven...
![AJAX For Evil: Spyjax with jQuery]()
Last year I wrote a popular post titled AJAX For Evil: Spyjax when I described a technique called "Spyjax":
Spyjax, as I know it, is taking information from the user's computer for your own use — specifically their browsing habits. By using CSS and JavaScript, I...
![QuickBoxes for Dojo]()
Adding to my mental portfolio is important to me. First came MooTools, then jQuery, and now Dojo. I speak often with Peter Higgins of Dojo fame and decided it was time to step into his world. I chose a simple but useful plugin...
Here’s an example showing how to include multiple development domains:
RewriteCond %{HTTP_HOST} ^localhost [OR] RewriteCond %{HTTP_HOST} ^example.dev [OR] RewriteCond %{HTTP_HOST} ^test.example.com [OR] RewriteCond %{HTTP_HOST} ^staging.example.com RewriteRule ^robots.txt$ robots-disallow.txt [L]use vagrant