Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    9 Mind-Blowing WebGL Demos

    As much as developers now loathe Flash, we're still playing a bit of catch up to natively duplicate the animation capabilities that Adobe's old technology provided us.  Of course we have canvas, an awesome technology, one which I highlighted 9 mind-blowing demos.  Another technology available...

  • By
    Animated 3D Flipping Menu with CSS

    CSS animations aren't just for basic fades or sliding elements anymore -- CSS animations are capable of much more.  I've showed you how you can create an exploding logo (applied with JavaScript, but all animation is CSS), an animated Photo Stack, a sweet...

Incredible Demos

  • By
    jQuery Wookmark

    The first thing that hits you when you visit Pinterest is "Whoa, the columns are the same width and the photos are cut to fit just the way they should."  Basic web users probably think nothing of it but as a developer, I can appreciate the...

  • By
    HTML5 Context Menus

    One of the hidden gems within the HTML5 spec is context menus. The HTML5 context menu spec allows developers to create custom context menus for given blocks within simple menu and menuitem elements. The menu information lives right within the page so...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!