robots.txt Rerouting on Development Servers
Every website should have a robots.txt file. Some bots hit sites so often that they slow down performance, other bots simply aren't desirable. robots.txt files can also be used to communicate sitemap location and limit request rate. It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file. Here's a quick .htaccess snippet you can use to make that happen:
RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]
The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint: you really don't want this).
![Chris Coyier’s Favorite CodePen Demos]()
David asked me if I'd be up for a guest post picking out some of my favorite Pens from CodePen. A daunting task! There are so many! I managed to pick a few though that have blown me away over the past few months. If you...
![Designing for Simplicity]()
Before we get started, it's worth me spending a brief moment introducing myself to you. My name is Mark (or @integralist if Twitter happens to be your communication tool of choice) and I currently work for BBC News in London England as a principal engineer/tech...
![QuickBoxes for Dojo]()
Adding to my mental portfolio is important to me. First came MooTools, then jQuery, and now Dojo. I speak often with Peter Higgins of Dojo fame and decided it was time to step into his world. I chose a simple but useful plugin...
![MooTools 1.2 Tooltips: Customize Your Tips]()
I've never met a person that is "ehhhh" about XHTML/javascript tooltips; people seem to love them or hate them. I'm on the love side of things. Tooltips give you a bit more information about something than just the element itself (usually...
Here’s an example showing how to include multiple development domains:
RewriteCond %{HTTP_HOST} ^localhost [OR] RewriteCond %{HTTP_HOST} ^example.dev [OR] RewriteCond %{HTTP_HOST} ^test.example.com [OR] RewriteCond %{HTTP_HOST} ^staging.example.com RewriteRule ^robots.txt$ robots-disallow.txt [L]use vagrant