Defer ALL The Things

By on  

James Socol is my manager at Mozilla, and he's an epic boss. Usually your manager is someone who struggles to understand the gist of a webapp's code; hell man, James Socol wrote the code. James has been an awesome help in my Python development, but here he steps into the JavaScript world and talks Deferreds.

JavaScript was one of the first languages I ever learned, but as I drifted more toward Python development, I got increasingly out-of-touch with the best ways of doing things.

Then a friend asked me to help on a project. The server was in Node.js and the datastore was MongoDB. Async, webscale, buzzwords, we had it all!

Callback Hell

One of the downsides of everything being asynchronous is that it's very easy to fall into the trap of writing everything as an anonymous callback. You can generally tell this happened by looking at the end of the file.

                // OH GOD WHY

This just hurts me. It's from the end of a view that got objects from two collections. As a Python developer, I can do the same thing in four lines of code!

def home(request):
    foos = Foo.objects.all().orderby('-created')[0:5]
    bars = Bar.objects.all().orderby('-created')[0:5]
    return render(request, 'home.html', {'foos': foos, 'bars': bars})

No, it's not asynchronous (well...), but I think I'd rather spend the CPU cycles and save my brain.

I actually delayed coming back to JavaScript because so much of it looks like this.


Why do I hate this so much?

When callbacks are nested like this, they usually rely on lexical closure for access to variables, e.g.:

app.get('/', function(req, res) {
    // Open the connection.
    db.open(function(err, db) {
        // Get one collection.
        db.collection('users', function(err, usersColl) {
            // Search the first collection.
            usersColl.find({}, {'limit': 3}, function(err, usersCursor) {
                // Convert the result into an array.
                usersCursor.toArray(function(err, users) {
                    // Get the second collection.
                    db.collection('articles', function(err, artColl) {
                        // Search the second collection.
                        artColl.find({}, {'limit': 3}, function(err, artCursor) {
                            // Convert the result into an array.
                            artCursor.toArray(function(err, articles) {
                                // Now we have two arrays, users and articles, in scope.
                                // Render the home page.
                                res.render('home.ejs', {'users': users, 'articles': articles});

The innermost function only has access to the array users because it's closed over several other functions.

There are at least 3 things wrong with this:

  1. Both queries are asynchronous, but we do them in serial instead of parallel. That's a waste of time and burns most of our "async" advantages right off the bat.
  2. Because these inner functions depend on lexical closure, they're impossible to test in smaller fragments and difficult to refactor to become more testable.
  3. Rendering a template in the middle of a database query is no better than a database query in the middle of a template:
<h1><?= mysql_query($my, "SELECT title FROM posts WHERE..."); ?></h1>

And finally, #4, OH GOD THE NESTING.

But then!

Months later, at a hack day, researching something completely different, I stumbled across jQuery.Deferred() and it was like I had come out of the cave and seen the light.

I had seen projects like tame.js, which I like in principle, but I find overly complicated in practice. I thought those were the options: callback hell or precompilers that spit out mounds of impenetrable code.

All of a sudden my functions could be functions again! They could take arguments instead of relying on scope. They could be run in parallel. They could be nice and flat!

var mongodb = require('mongodb');
var express = require('express');
var Deferred = require('Deferred');

var app = express.createServer();
var db = new mongodb.Db('mysite',
                        new mongodb.Server('localhost', 27027, {}),
                        {native_parser: true});

var Deferred = require('Deferred');

// Only do this once. Keep it somewhere else.
function getArray(db, coll, search, options) {
    var def = Deferred();
    db.open(function(err, db) {
        if (err) def.reject(err);
        db.collection(coll, function(err, collection) {
            if (err) def.reject(err);
            collection.find(search, options, function(err, cursor) {
                if (err) def.reject(err);
                cursor.toArray(function(err, arr) {
                    if (err) def.reject(err);
    return def.promise();

function home(req, res) {
    var uDef = getArray(db, 'users', {}, {'limit': 3});
    var aDef = getArray(db, 'articles', {}, {'limit': 4});
    var lookup = Deferred.when(uDef, aDef);
    lookup.done(function(users, articles)) {
        res.render('home.ejs', {'users': users, 'articles': articles});
    }).fail(function(err) {
        res.render('error.ejs', {'error': err})

app.get('/', home);

Ah, That's Better

A lot of library code, like Dojo and jQuery, takes advantage of Promises and Deferreds to make things cleaner. I hope that more and more userland code will do the same.

Finding Deferreds helped me learn to love JavaScript again. It's such a fun, flexible, interesting language, so I'm glad complex code can be easy to test and fun to read.

James Socol

About James Socol

James Socol is a Python, JavaScript, and web developer and Mozilla's Community Platforms Manager. He is an active member of the Mozilla, Python, and Django communities and maintains several widely-used open source projects.

O'Reilly Velocity Conference
Save 20% with discount code AFF20

Recent Features

  • Facebook Open Graph META Tags

    It's no secret that Facebook has become a major traffic driver for all types of websites.  Nowadays even large corporations steer consumers toward their Facebook pages instead of the corporate websites directly.  And of course there are Facebook "Like" and "Recommend" widgets on every website.  One...

  • Vibration API

    Many of the new APIs provided to us by browser vendors are more targeted toward the mobile user than the desktop user.  One of those simple APIs the Vibration API.  The Vibration API allows developers to direct the device, using JavaScript, to vibrate in...

Incredible Demos

  • jQuery Wookmark

    The first thing that hits you when you visit Pinterest is "Whoa, the columns are the same width and the photos are cut to fit just the way they should."  Basic web users probably think nothing of it but as a developer, I can appreciate the...

  • Retrieve Your Gmail Emails Using PHP and IMAP

    Grabbing emails from your Gmail account using PHP is probably easier than you think. Armed with PHP and its IMAP extension, you can retrieve emails from your Gmail account in no time! Just for fun, I'll be using the MooTools Fx.Accordion plugin...


  1. Matthew Iversen

    Hi there, in your last code block, under the // Only do this once. Keep it somewhere else. comment, you have a reptitive nested code block that looked very familiar to me.

    Perhaps watching this video will help you reduce it to something nicer? The video solves patterns that look exactly like that, which apparently involves creating monads! Very interesting.


  2. A good read to accompany this post is this:


    “ You’re Missing the Point of Promises”, by the creator of Q, a CommonJS compatible promise implementation (sounds more complicated than it is). It’s a good introduction in what (and what not) promises are supposed to be.

  3. Nice post, although with jQuery deferred you now have to wrap lots of standard node.js calls in your own functions that resolve to deffereds. I currently prefer keeping my node code a bit simpler with a control flow library (sooooo many out there). I personally love Step (https://github.com/creationix/step)

  4. Hi! Great demonstration. I’m using this similar techniqe with success:


Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!

Recently on David Walsh Blog

  • Prevent Chrome from Translating a Page

    A while back I shared my favorite Google Chrome extension:  Google Art Project.  I've enjoyed seeing beautiful art when I open a new tab -- it's brought genuine happiness to my day, however small that happiness may be.  About a week ago, however, the art presented had...

  • Create Any Type Of Website With These Multi-Purpose Themes

    We have selected what we believe are the very best multipurpose WordPress themes on the market today. Our list contains a number of best sellers, several newcomers that are proving to be highly popular, and a few themes that are ideal for creating the types of...

  • An Introduction to Static Site Generators

    Static site generators seem to have been becoming more and more popular recently, but they’re not one of those ephemeral novelty things that grow in popularity as quickly as they fall into oblivion shortly after. For over a decade, many different projects — 394 of...

  • Automated Tests for Visual Responsive Layouts

    Today it's all about testing. In 2015, many developers knows about TDD and I personally think that testing is one of the key for quality products. But what about testing in a Front-end environment? How do you guys write your tests for a responsive page or...

  • Getting Dicey With Flexbox

    What if you could build complex CSS layouts in minutes? Flexbox is a new CSS layout spec that makes it easy to construct dynamic layouts. With flexbox, vertical centering, same-height columns, reordering, and direction agnosticism are a piece of cake. There's a popular myth floating around that...