Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    Regular Expressions for the Rest of Us

    Sooner or later you'll run across a regular expression. With their cryptic syntax, confusing documentation and massive learning curve, most developers settle for copying and pasting them from StackOverflow and hoping they work. But what if you could decode regular expressions and harness their power? In...

  • By
    Conquering Impostor Syndrome

    Two years ago I documented my struggles with Imposter Syndrome and the response was immense.  I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions.  I've even caught myself reading the post...

Incredible Demos

  • By
    MooTools Clipboard Plugin

    The ability to place content into a user's clipboard can be extremely convenient for the user. Instead of clicking and dragging down what could be a lengthy document, the user can copy the contents of a specific area by a single click of a mouse.

  • By
    Create GitHub-Style Buttons with CSS and jQuery, MooTools, or Dojo JavaScript

    I'm what you would consider a bit of a GitHub fanboy. We all know that GitHub is the perfect place to store repositories of open source code, but I think my love of GitHub goes beyond that. GitHub seems to understand that most...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!