O'Reilly

Prevent Robot Indexing with Response Headers

By on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

Track.js Error Reporting

Recent Features

  • 9 Mind-Blowing WebGL Demos

    As much as developers now loathe Flash, we're still playing a bit of catch up to natively duplicate the animation capabilities that Adobe's old technology provided us.  Of course we have canvas, an awesome technology, one which I highlighted 9 mind-blowing demos.  Another technology available...

  • Creating Scrolling Parallax Effects with CSS

    Introduction For quite a long time now websites with the so called "parallax" effect have been really popular. In case you have not heard of this effect, it basically includes different layers of images that are moving in different directions or with different speed. This leads to a...

Incredible Demos

  • WebKit-Specific Style:  -webkit-appearance

    I was recently scoping out the horrid source code of the Google homepage when I noticed the "Google Search" and "I'm Feeling Lucky" buttons had a style definition I hadn't seen before:  -webkit-appearance.  The value assigned to the style was "push-button."  They are buttons so that...

  • Create Twitter-Style Buttons with the Dojo Toolkit

    I love that JavaScript toolkits make enhancing web pages incredibly easy. Today I'll cover an effect that I've already coded with MooTools: creating a Twitter-style animated "Sign In" button. Check out this five minute tutorial so you can take your static...

Discussion

  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!

Recently on David Walsh Blog

  • JavaScript Polling

    Polling with JavaScript is one of those ugly but important functions within advanced front-end user experience and testing practices.  Sometimes there isn't the event you can hook into to signify that a given task is complete, so you need to get your hands dirty and simply poll for...

  • OSCON Portland:  Conference Giveaway and Discount!

    O'Reilly puts on the best web industry conferences in the world.  These conferences include Fluent Conference, Velocity Conference, and the upcoming OSCON in Portland, Oregon from July 20-24.  Open Source Convention (OSCON) is a conference that focuses specifically on open source developers and the tools and possibilities...

  • Prevent Chrome from Translating a Page

    A while back I shared my favorite Google Chrome extension:  Google Art Project.  I've enjoyed seeing beautiful art when I open a new tab -- it's brought genuine happiness to my day, however small that happiness may be.  About a week ago, however, the art presented had...

  • Create Any Type Of Website With These Multi-Purpose Themes

    We have selected what we believe are the very best multipurpose WordPress themes on the market today. Our list contains a number of best sellers, several newcomers that are proving to be highly popular, and a few themes that are ideal for creating the types of...

  • An Introduction to Static Site Generators

    Static site generators seem to have been becoming more and more popular recently, but they’re not one of those ephemeral novelty things that grow in popularity as quickly as they fall into oblivion shortly after. For over a decade, many different projects — 394 of...