Prevent Robot Indexing with Response Headers

By  on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

Recent Features

  • By
    Create a CSS Cube

    CSS cubes really showcase what CSS has become over the years, evolving from simple color and dimension directives to a language capable of creating deep, creative visuals.  Add animation and you've got something really neat.  Unfortunately each CSS cube tutorial I've read is a bit...

  • By
    Conquering Impostor Syndrome

    Two years ago I documented my struggles with Imposter Syndrome and the response was immense.  I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions.  I've even caught myself reading the post...

Incredible Demos

  • By
    HTML5&#8217;s window.postMessage API

    One of the little known HTML5 APIs is the window.postMessage API.  window.postMessage allows for sending data messages between two windows/frames across domains.  Essentially window.postMessage acts as cross-domain AJAX without the server shims. Let's take a look at how window.postMessage works and how you...

  • By
    Create a Download Package Using MooTools Moousture

    Zohaib Sibt-e-Hassan recently released a great mouse gestures library for MooTools called Moousture. Moousture allows you to trigger functionality by moving your mouse in specified custom patterns. Too illustrate Moousture's value, I've created an image download builder using Mooustures and PHP. The XHTML We provide...

Discussion

  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

  2. This a really old post, but for those who are using nGinx instead of Apache, you can do

    location ~* \.(doc|pdf)$ {
        add_header  X-Robots-Tag "noindex, noarchive, nosnippet";
    }
    

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!