Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    Page Visibility API

    One event that's always been lacking within the document is a signal for when the user is looking at a given tab, or another tab. When does the user switch off our site to look at something else? When do they come back?

  • By
    JavaScript Promise API

    While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...

Incredible Demos

  • By
    Build a Slick and Simple MooTools Accordion

    Last week I covered a smooth, subtle MooTools effect called Kwicks. Another great MooTools creation is the Accordion, which acts like...wait for it...an accordion! Now I've never been a huge Weird Al fan so this is as close to playing an accordion as...

  • By
    MooTools FontChecker Plugin

    There's a very interesting piece of code on Google Code called FontAvailable which does a jQuery-based JavaScript check on a string to check whether or not your system has a specific font based upon its output width. I've ported this functionality to MooTools. The MooTools...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!