Prevent Robot Indexing with Response Headers

By  on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

Raygun Pulse

Recent Features

  • By
    9 Mind-Blowing Canvas Demos

    The <canvas> element has been a revelation for the visual experts among our ranks.  Canvas provides the means for incredible and efficient animations with the added bonus of no Flash; these developers can flash their awesome JavaScript skills instead.  Here are nine unbelievable canvas demos that...

  • By
    Send Text Messages with PHP

    Kids these days, I tell ya.  All they care about is the technology.  The video games.  The bottled water.  Oh, and the texting, always the texting.  Back in my day, all we had was...OK, I had all of these things too.  But I still don't get...

Incredible Demos

  • By
    Using Opacity to Show Focus with MooTools

    I'm a huge fan of using subtle effects like link nudging (jQuery, MooTools) to enhance the user experience and increase the perceived dynamism of my websites. Trust me -- a lot of little things are what take websites to the next level.

  • By
    MooTools Window Object Dumping

    Ever want to see all of the information stored within the window property of your browser? Here's your chance. The XHTML We need a wrapper DIV that we'll consider a console. The CSS I like making this look like a command-line console. The MooTools JavaScript Depending on what you have loaded...

Discussion

  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!