O'Reilly

Prevent Robot Indexing with Response Headers

By on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

O'Reilly Velocity Conference
Save 20% with discount code AFF20

Recent Features

  • LightFace:  Facebook Lightbox for MooTools

    One of the web components I've always loved has been Facebook's modal dialog.  This "lightbox" isn't like others:  no dark overlay, no obnoxious animating to size, and it doesn't try to do "too much."  With Facebook's dialog in mind, I've created LightFace:  a Facebook lightbox...

  • CSS 3D Folding Animation

    Google Plus provides loads of inspiration for front-end developers, especially when it comes to the CSS and JavaScript wonders they create. Last year I duplicated their incredible PhotoStack effect with both MooTools and pure CSS; this time I'm going to duplicate...

Incredible Demos

Discussion

  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!

Recently on David Walsh Blog

  • Preventing Side Effects in JavaScript

    JavaScript is very dynamic these days but I still see a lot of legacy code, whether it be for optimal backward compatibility or simply that the code hasn't been maintained.  One of the practices that makes me cringe is coding that creates unwanted side effects.  What's a side...

  • Access Mac Camera by Command Line

    With all of my recent command line tutorials, I've really gotten excited about the shell's simplicity and realized the true power of using the underlying technology of pretty UIs.  Since I work from home, I spend a lot of time on video calls, so when...

  • Open Files from Command Line on OS X

    I'm as much of a fan of application UIs as anyone else but I'm finding myself working more and more from the command line lately.  Much of that is becoming obsessed with media manipulation but I'm forcing myself to use less UIs so that I...

  • Get Stock Quotes From Command Line

    When I conned my way into my first professional programming gig, I didn't really think much about money -- just that I was getting my foot in the door.  But as my career has gone on, I've been more aware of money, investing, and retirement.  I've recently...

  • Geolocation API

    One interesting aspect of web development is geolocation; where is your user viewing your website from? You can base your language locale on that data or show certain products in your store based on the user's location. Let's examine how you can...