Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    Vibration API

    Many of the new APIs provided to us by browser vendors are more targeted toward the mobile user than the desktop user.  One of those simple APIs the Vibration API.  The Vibration API allows developers to direct the device, using JavaScript, to vibrate in...

  • By
    How I Stopped WordPress Comment Spam

    I love almost every part of being a tech blogger:  learning, preaching, bantering, researching.  The one part about blogging that I absolutely loathe:  dealing with SPAM comments.  For the past two years, my blog has registered 8,000+ SPAM comments per day.  PER DAY.  Bloating my database...

Incredible Demos

  • By
    Scroll IFRAMEs on iOS

    For the longest time, developers were frustrated by elements with overflow not being scrollable within the page of iOS Safari.  For my blog it was particularly frustrating because I display my demos in sandboxed IFRAMEs on top of the article itself, so as to not affect my site's...

  • By
    PHP Woot Checker – Tech, Wine, and Shirt Woot

    If you haven't heard of Woot.com, you've been living under a rock. For those who have been under the proverbial rock, here's the plot: Every day, Woot sells one product. Once the item is sold out, no more items are available for purchase. You don't know how many...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!