Disallow Robots Using Robots.txt
I develop customer websites on a publicly accessible web server so that my customers may check the progress of their website at any given time. I could use .htaccess to require username and password for the site but then I'm constantly needing to remind customers what their password is. My big concern is preventing search engines from finding their way to my development server. Luckily I can add a robots.txt file to my development server websites that will prevent search engines from indexing them.
The above directive prevents the search engines from indexing any pages or files on the website. Say, however, that you simply want to keep search engines out of the folder that contains your administrative control panel. You'd code:
Or if you wanted to allow in all spiders except Google's GoogleBot, you'd code:
What would you prevent the search engines from seeing?
It's no secret that Facebook has become a major traffic driver for all types of websites. Nowadays even large corporations steer consumers toward their Facebook pages instead of the corporate websites directly. And of course there are Facebook "Like" and "Recommend" widgets on every website. One...
Two years ago I documented my struggles with Imposter Syndrome and the response was immense. I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions. I've even caught myself reading the post...
I'm back! David asked me to rope up some of my favorite stuff on CodePen again, which I both love doing, and wince at the thought of having to pick so few favorites. I like a ton of stuff on...
There you have it. Opacity is a very simple but effective...