Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    5 HTML5 APIs You Didn’t Know Existed

    When you say or read "HTML5", you half expect exotic dancers and unicorns to walk into the room to the tune of "I'm Sexy and I Know It."  Can you blame us though?  We watched the fundamental APIs stagnate for so long that a basic feature...

  • By
    Being a Dev Dad

    I get asked loads of questions every day but I'm always surprised that they're rarely questions about code or even tech -- many of the questions I get are more about non-dev stuff like what my office is like, what software I use, and oftentimes...

Incredible Demos

  • By
    Fancy Navigation with MooTools JavaScript

    Navigation menus are traditionally boring, right? Most of the time the navigation menu consists of some imagery with a corresponding mouseover image. Where's the originality? I've created a fancy navigation menu that highlights navigation items and creates a chain effect. The XHTML Just some simple...

  • By
    Duplicate the jQuery Homepage Tooltips Using Dojo

    The jQuery homepage has a pretty suave tooltip-like effect as seen below: Here's how to accomplish this same effect using Dojo. The XHTML The above HTML was taken directly from the jQuery homepage -- no changes. The CSS The above CSS has been slightly modified to match the CSS rules already...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!