Python Decorator for Preventing Robot Indexing

By  on  

Much of my time at Mozilla has been spent catching up to the rest of the MDN team with respect to python.  The new MDN backend, codenamed Kuma, is entirely Django-based and has been a joy to learn.  My latest python adventures have been focused on increasing MDN's SEO score, a task which includes telling Google not to index given pages.  In doing so, I created my first view decorator:  a decorator that sends a response header which prevents a robot from indexing the given page.

The Python

The first step is importing the decorator dependencies:

from functools import wraps

The next step is creating the decorator definition:

def prevent_indexing(view_func):
    """Decorator to prevent a page from being indexable by robots"""
    @wraps(view_func)
    def _added_header(request, *args, **kwargs):
        response = view_func(request, *args, **kwargs)
        response['X-Robots-Tag'] = 'noindex'
        return response
    return _added_header

The original view is processed so that the response can be received;  simple add a key of X-Robots-Tag to the response, value set to noindex, and the decorator is complete!

To use the decorator, simply add it to a given view:

@login_required
@prevent_indexing
def new_document(request):
	""" Does whatever 'new_document' should do; page should not be indexed """

Voila -- the X-Robots-Tag header will be sent so that robots wont index the page!  Decorators allow for loads of additional functionality for any view, and are easily reusable;  I'm glad to add them to my arsenal!

Recent Features

  • By
    Responsive Images: The Ultimate Guide

    Chances are that any Web designers using our Ghostlab browser testing app, which allows seamless testing across all devices simultaneously, will have worked with responsive design in some shape or form. And as today's websites and devices become ever more varied, a plethora of responsive images...

  • By
    Camera and Video Control with HTML5

    Client-side APIs on mobile and desktop devices are quickly providing the same APIs.  Of course our mobile devices got access to some of these APIs first, but those APIs are slowly making their way to the desktop.  One of those APIs is the getUserMedia API...

Incredible Demos

  • By
    Optimize Your Links For Print Using CSS — Show The URL

    When moving around from page to page in your trusty browser, you get the benefit of hovering over links and viewing the link's target URL in the status bar. When it comes to page printouts, however, this obviously isn't an option. Most website printouts...

  • By
    MooTools ASCII Art

    I didn't realize that I truly was a nerd until I could admit to myself that ASCII art was better than the pieces Picasso, Monet, or Van Gogh could create.  ASCII art is unmatched in its beauty, simplicity, and ... OK, well, I'm being ridiculous;  ASCII...

Discussion

    Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!