Disallow Robots Using Robots.txt
I develop customer websites on a publicly accessible web server so that my customers may check the progress of their website at any given time. I could use .htaccess to require username and password for the site but then I'm constantly needing to remind customers what their password is. My big concern is preventing search engines from finding their way to my development server. Luckily I can add a robots.txt file to my development server websites that will prevent search engines from indexing them.
The above directive prevents the search engines from indexing any pages or files on the website. Say, however, that you simply want to keep search engines out of the folder that contains your administrative control panel. You'd code:
Or if you wanted to allow in all spiders except Google's GoogleBot, you'd code:
What would you prevent the search engines from seeing?
I spent a few months experimenting with different approaches for writing simple, elegant and maintainable media queries with Sass. Each solution had something that I really liked, but I couldn't find one that covered everything I needed to do, so I ventured into creating my...
I work with an awesome cast of developers at Mozilla, and one of them in Daniel Buchner. Daniel's shared with me an awesome strategy for detecting when nodes have been injected into a parent node without using the deprecated DOM Events API.
My team mate Edna Piranha is not only an awesome hacker; she's also a fantastic philosopher! Communication and online interactions is a subject that has kept her mind busy for a long time, and it has also resulted in a bunch of interesting experimental projects...