robots.txt Rerouting on Development Servers
Every website should have a robots.txt file. Some bots hit sites so often that they slow down performance, other bots simply aren't desirable. robots.txt files can also be used to communicate sitemap location and limit request rate. It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file. Here's a quick .htaccess snippet you can use to make that happen:
RewriteCond %{HTTP_HOST} devdomain
RewriteRule ^robots.txt$ robots-go-away.txt [L]
The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint: you really don't want this).
![5 Ways that CSS and JavaScript Interact That You May Not Know About]()
CSS and JavaScript: the lines seemingly get blurred by each browser release. They have always done a very different job but in the end they are both front-end technologies so they need do need to work closely. We have our .js files and our .css, but...
![Designing for Simplicity]()
Before we get started, it's worth me spending a brief moment introducing myself to you. My name is Mark (or @integralist if Twitter happens to be your communication tool of choice) and I currently work for BBC News in London England as a principal engineer/tech...
![WebSocket and Socket.IO]()
My favorite web technology is quickly becoming the WebSocket API. WebSocket provides a welcomed alternative to the AJAX technologies we've been making use of over the past few years. This new API provides a method to push messages from client to server efficiently...
![Input Incrementer and Decrementer with MooTools]()
Chris Coyier's CSS-Tricks blog is everything mine isn't. Chris' blog is rock star popular, mine is not. Chris prefers jQuery, I prefer MooTools. Chris does posts with practical solutions, I do posts about stupid video-game like effects. If I...
Here’s an example showing how to include multiple development domains:
RewriteCond %{HTTP_HOST} ^localhost [OR] RewriteCond %{HTTP_HOST} ^example.dev [OR] RewriteCond %{HTTP_HOST} ^test.example.com [OR] RewriteCond %{HTTP_HOST} ^staging.example.com RewriteRule ^robots.txt$ robots-disallow.txt [L]use vagrant