O'Reilly

Prevent Robot Indexing with Response Headers

By on  

Every so often you have parts of your website that would be better off not indexed by search engines.  API calls, search result pages, PDF documents -- all examples of responses which may not have value outside of the current user.  No we all know we can signal to the search engines not to index pages using a META tag, but oftentimes service calls and documents don't get the luxury of a META tag.  Luckily you can add a header to prevent these responses from being indexed.

The header name is X-Robots-Tag should be easy to add using the server-side language you prefer.  For example, adding this header with PHP may look like:

header('X-Robots-Tag: noindex');

If you're using a Django-based python site, the could would look like:

response['X-Robots-Tag'] = 'noindex'

This header can also be set within your .htaccess or httpd configuration files:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex"
</Files>

The truth is that there's no guarantee that something your server serves wont be indexed by a search engine, but small tweaks like this can ensure your search engine standing can improve and that users don't find their way to "dead" parts of your site via search engines.

Track.js Error Reporting

Recent Features

Incredible Demos

  • Create a CSS Cube

    CSS cubes really showcase what CSS has become over the years, evolving from simple color and dimension directives to a language capable of creating deep, creative visuals.  Add animation and you've got something really neat.  Unfortunately each CSS cube tutorial I've read is a bit...

  • Prevent Page Zooming in Mobile Browsers

    Ever since I got my iPhone, I've been more agreeable in going places that my fiancee wants to go. It's not because I have any interest in checking out women's shoes, looking at flowers, or that type of stuff -- it's because my iPhone lets...

Discussion

  1. Chris

    I have a big problem with spam registration on an ExpressionEngine site I help manage. Could this help? I have no development experience, fyi…

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!

Recently on David Walsh Blog

  • GSAP + SVG For Power Users: Motion Along A Path

    Now that the GreenSock API is picking up steam, there are many tutorials and Getting Started guides out there to provide good introductions to the library, not to mention GreenSock’s own Forum and Documentation. This article isn’t intended for beginners, but rather a...

  • Copy a Directory from Command Line

    Copying a directory for the sake of backup is something I do often, especially when I'm trying to figure out why something isn't working when I use an external library.  I'll copy the directory structure as a backup, mess around with the original source until I find a solution,...

  • Hotjar &#8211; All-in-one Analytics and Feedback

    Website analytics are a massive business -- the more data you can collect with regard to your users' behaviors on your site, the more you can increase and maximise conversion...and increased conversion is always good.  Sometimes increase conversion means more money, improved user experience, viewer retention,...

  • Crafting a 3D React Carousel

    There is something in me that is amazed but beautiful 3D interfaces. And it doesn’t matter whether they’re functional like Gyroscope features menu, technology demonstrators like the amazing periodic table demo from famous or they’re artistic representation pushing the limits of...

  • Collaborative Editing in JavaScript: An Intro to Operational Transformation

    I've set out to build a robust collaborative code editor for the web. It’s called Codr, and it lets developers work together in real time - like Google Docs for code. For web developers, Codr doubles as a shared reactive work surface where every change is...