Disallow Robots Using Robots.txt
I develop customer websites on a publicly accessible web server so that my customers may check the progress of their website at any given time. I could use .htaccess to require username and password for the site but then I'm constantly needing to remind customers what their password is. My big concern is preventing search engines from finding their way to my development server. Luckily I can add a robots.txt file to my development server websites that will prevent search engines from indexing them.
The above directive prevents the search engines from indexing any pages or files on the website. Say, however, that you simply want to keep search engines out of the folder that contains your administrative control panel. You'd code:
Or if you wanted to allow in all spiders except Google's GoogleBot, you'd code:
What would you prevent the search engines from seeing?
It's no secret that Facebook has become a major traffic driver for all types of websites. Nowadays even large corporations steer consumers toward their Facebook pages instead of the corporate websites directly. And of course there are Facebook "Like" and "Recommend" widgets on every website. One...
With Firefox OS, asm.js, and the push for browser performance improvements, canvas and WebGL technologies are opening a world of possibilities. I featured 9 Mind-Blowing Canvas Demos and then took it up a level with 9 Mind-Blowing WebGL Demos, but I want to outdo...
Google Analytics provides a wealth of information about who's coming to your website. One of the most important statistics the service provides is the referrer statistic -- you've gotta know who's sending people to your website, right? What about where you send others though?
A while back David Walsh published a list of
7 MooTools Plugins You Should Use on Every Website
which included 'AutoGrow' a text area expander plugin. 'AutoGrow' is very similar in results to the class I wrote for Education.com, Flext. I decided to release this...