Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    Convert XML to JSON with JavaScript

    If you follow me on Twitter, you know that I've been working on a super top secret mobile application using Appcelerator Titanium.  The experience has been great:  using JavaScript to create easy to write, easy to test, native mobile apps has been fun.  My...

  • By
    7 Essential JavaScript Functions

    I remember the early days of JavaScript where you needed a simple function for just about everything because the browser vendors implemented features differently, and not just edge features, basic features, like addEventListener and attachEvent.  Times have changed but there are still a few functions each developer should...

Incredible Demos

  • By
    WebKit Marquee CSS:  Bringin’ Sexy Back

    We all joke about the days of Web yesteryear.  You remember them:  stupid animated GIFs (flames and "coming soon" images, most notably), lame counters, guestbooks, applets, etc.  Another "feature" we thought we had gotten rid of was the marquee.  The marquee was a rudimentary, javascript-like...

  • By
    MooTools 1.2 Image Protector: dwProtector

    Image protection is a hot topic on the net these days, and why shouldn't it be? If you spent two hours designing an awesome graphic, would you want it ripped of in matter of seconds? Hell no! That's why I've created an image...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!