Prevent Robot Indexing with Response Headers

By  on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

Track.js Error Reporting

Upcoming Events

Recent Features

  • Chris Coyier&#8217;s Favorite CodePen Demos

    David asked me if I'd be up for a guest post picking out some of my favorite Pens from CodePen. A daunting task! There are so many! I managed to pick a few though that have blown me away over the past few months. If you...

  • JavaScript Promise API

    While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...

Incredible Demos

  • MooTools-Like Element Creation in jQuery

    I really dislike jQuery's element creation syntax. It's basically the same as typing out HTML but within a JavaScript string...ugly! Luckily Basil Goldman has created a jQuery plugin that allows you to create elements using MooTools-like syntax. Standard jQuery Element Creation Looks exactly like writing out...

  • RealTime Stock Quotes with MooTools Request.Stocks and YQL

    It goes without saying but MooTools' inheritance pattern allows for creation of small, simple classes that possess immense power.  One example of that power is a class that inherits from Request, Request.JSON, and Request.JSONP:  Request.Stocks.  Created by Enrique Erne, this great MooTools class acts as...


  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!