Defer ALL The Things

By  on  

James Socol is my manager at Mozilla, and he's an epic boss. Usually your manager is someone who struggles to understand the gist of a webapp's code; hell man, James Socol wrote the code. James has been an awesome help in my Python development, but here he steps into the JavaScript world and talks Deferreds.

JavaScript was one of the first languages I ever learned, but as I drifted more toward Python development, I got increasingly out-of-touch with the best ways of doing things.

Then a friend asked me to help on a project. The server was in Node.js and the datastore was MongoDB. Async, webscale, buzzwords, we had it all!

Callback Hell

One of the downsides of everything being asynchronous is that it's very easy to fall into the trap of writing everything as an anonymous callback. You can generally tell this happened by looking at the end of the file.

                // OH GOD WHY

This just hurts me. It's from the end of a view that got objects from two collections. As a Python developer, I can do the same thing in four lines of code!

def home(request):
    foos = Foo.objects.all().orderby('-created')[0:5]
    bars = Bar.objects.all().orderby('-created')[0:5]
    return render(request, 'home.html', {'foos': foos, 'bars': bars})

No, it's not asynchronous (well...), but I think I'd rather spend the CPU cycles and save my brain.

I actually delayed coming back to JavaScript because so much of it looks like this.


Why do I hate this so much?

When callbacks are nested like this, they usually rely on lexical closure for access to variables, e.g.:

app.get('/', function(req, res) {
    // Open the connection., db) {
        // Get one collection.
        db.collection('users', function(err, usersColl) {
            // Search the first collection.
            usersColl.find({}, {'limit': 3}, function(err, usersCursor) {
                // Convert the result into an array.
                usersCursor.toArray(function(err, users) {
                    // Get the second collection.
                    db.collection('articles', function(err, artColl) {
                        // Search the second collection.
                        artColl.find({}, {'limit': 3}, function(err, artCursor) {
                            // Convert the result into an array.
                            artCursor.toArray(function(err, articles) {
                                // Now we have two arrays, users and articles, in scope.
                                // Render the home page.
                                res.render('home.ejs', {'users': users, 'articles': articles});

The innermost function only has access to the array users because it's closed over several other functions.

There are at least 3 things wrong with this:

  1. Both queries are asynchronous, but we do them in serial instead of parallel. That's a waste of time and burns most of our "async" advantages right off the bat.
  2. Because these inner functions depend on lexical closure, they're impossible to test in smaller fragments and difficult to refactor to become more testable.
  3. Rendering a template in the middle of a database query is no better than a database query in the middle of a template:
<h1><?= mysql_query($my, "SELECT title FROM posts WHERE..."); ?></h1>

And finally, #4, OH GOD THE NESTING.

But then!

Months later, at a hack day, researching something completely different, I stumbled across jQuery.Deferred() and it was like I had come out of the cave and seen the light.

I had seen projects like tame.js, which I like in principle, but I find overly complicated in practice. I thought those were the options: callback hell or precompilers that spit out mounds of impenetrable code.

All of a sudden my functions could be functions again! They could take arguments instead of relying on scope. They could be run in parallel. They could be nice and flat!

var mongodb = require('mongodb');
var express = require('express');
var Deferred = require('Deferred');

var app = express.createServer();
var db = new mongodb.Db('mysite',
                        new mongodb.Server('localhost', 27027, {}),
                        {native_parser: true});

var Deferred = require('Deferred');

// Only do this once. Keep it somewhere else.
function getArray(db, coll, search, options) {
    var def = Deferred();, db) {
        if (err) def.reject(err);
        db.collection(coll, function(err, collection) {
            if (err) def.reject(err);
            collection.find(search, options, function(err, cursor) {
                if (err) def.reject(err);
                cursor.toArray(function(err, arr) {
                    if (err) def.reject(err);
    return def.promise();

function home(req, res) {
    var uDef = getArray(db, 'users', {}, {'limit': 3});
    var aDef = getArray(db, 'articles', {}, {'limit': 4});
    var lookup = Deferred.when(uDef, aDef);
    lookup.done(function(users, articles)) {
        res.render('home.ejs', {'users': users, 'articles': articles});
    }).fail(function(err) {
        res.render('error.ejs', {'error': err})

app.get('/', home);

Ah, That's Better

A lot of library code, like Dojo and jQuery, takes advantage of Promises and Deferreds to make things cleaner. I hope that more and more userland code will do the same.

Finding Deferreds helped me learn to love JavaScript again. It's such a fun, flexible, interesting language, so I'm glad complex code can be easy to test and fun to read.

James Socol

About James Socol

James Socol is a Python, JavaScript, and web developer and Mozilla's Community Platforms Manager. He is an active member of the Mozilla, Python, and Django communities and maintains several widely-used open source projects.

Recent Features

  • By
    Vibration API

    Many of the new APIs provided to us by browser vendors are more targeted toward the mobile user than the desktop user.  One of those simple APIs the Vibration API.  The Vibration API allows developers to direct the device, using JavaScript, to vibrate in...

  • By
    9 Mind-Blowing Canvas Demos

    The <canvas> element has been a revelation for the visual experts among our ranks.  Canvas provides the means for incredible and efficient animations with the added bonus of no Flash; these developers can flash their awesome JavaScript skills instead.  Here are nine unbelievable canvas demos that...

Incredible Demos

  • By
    jQuery Chosen Plugin

    Without a doubt, my least favorite form element is the SELECT element.  The element is almost unstylable, looks different across platforms, has had inconsistent value access, and disaster that is the result of multiple=true is, well, a disaster.  Needless to say, whenever a developer goes...

  • By
    Animated AJAX Record Deletion Using Dojo

    I'm a huge fan of WordPress' method of individual article deletion. You click the delete link, the menu item animates red, and the item disappears. Here's how to achieve that functionality with Dojo JavaScript. The PHP - Content & Header The following snippet goes at the...


  1. Matthew Iversen

    Hi there, in your last code block, under the // Only do this once. Keep it somewhere else. comment, you have a reptitive nested code block that looked very familiar to me.

    Perhaps watching this video will help you reduce it to something nicer? The video solves patterns that look exactly like that, which apparently involves creating monads! Very interesting.

  2. A good read to accompany this post is this:

    “ You’re Missing the Point of Promises”, by the creator of Q, a CommonJS compatible promise implementation (sounds more complicated than it is). It’s a good introduction in what (and what not) promises are supposed to be.

  3. Nice post, although with jQuery deferred you now have to wrap lots of standard node.js calls in your own functions that resolve to deffereds. I currently prefer keeping my node code a bit simpler with a control flow library (sooooo many out there). I personally love Step (

  4. Hi! Great demonstration. I’m using this similar techniqe with success:


Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!