Treehouse

Defer ALL The Things

By on  

James Socol is my manager at Mozilla, and he's an epic boss. Usually your manager is someone who struggles to understand the gist of a webapp's code; hell man, James Socol wrote the code. James has been an awesome help in my Python development, but here he steps into the JavaScript world and talks Deferreds.

JavaScript was one of the first languages I ever learned, but as I drifted more toward Python development, I got increasingly out-of-touch with the best ways of doing things.

Then a friend asked me to help on a project. The server was in Node.js and the datastore was MongoDB. Async, webscale, buzzwords, we had it all!

Callback Hell

One of the downsides of everything being asynchronous is that it's very easy to fall into the trap of writing everything as an anonymous callback. You can generally tell this happened by looking at the end of the file.

                            });
                        });
                    });
                // OH GOD WHY
                });
            });
        });
    // CLOSE EVERYTHING
    });
});

This just hurts me. It's from the end of a view that got objects from two collections. As a Python developer, I can do the same thing in four lines of code!

def home(request):
    foos = Foo.objects.all().orderby('-created')[0:5]
    bars = Bar.objects.all().orderby('-created')[0:5]
    return render(request, 'home.html', {'foos': foos, 'bars': bars})

No, it's not asynchronous (well...), but I think I'd rather spend the CPU cycles and save my brain.

I actually delayed coming back to JavaScript because so much of it looks like this.

Why?

Why do I hate this so much?

When callbacks are nested like this, they usually rely on lexical closure for access to variables, e.g.:

app.get('/', function(req, res) {
    // Open the connection.
    db.open(function(err, db) {
        // Get one collection.
        db.collection('users', function(err, usersColl) {
            // Search the first collection.
            usersColl.find({}, {'limit': 3}, function(err, usersCursor) {
                // Convert the result into an array.
                usersCursor.toArray(function(err, users) {
                    // Get the second collection.
                    db.collection('articles', function(err, artColl) {
                        // Search the second collection.
                        artColl.find({}, {'limit': 3}, function(err, artCursor) {
                            // Convert the result into an array.
                            artCursor.toArray(function(err, articles) {
                                // Now we have two arrays, users and articles, in scope.
                                // Render the home page.
                                res.render('home.ejs', {'users': users, 'articles': articles});

The innermost function only has access to the array users because it's closed over several other functions.

There are at least 3 things wrong with this:

  1. Both queries are asynchronous, but we do them in serial instead of parallel. That's a waste of time and burns most of our "async" advantages right off the bat.
  2. Because these inner functions depend on lexical closure, they're impossible to test in smaller fragments and difficult to refactor to become more testable.
  3. Rendering a template in the middle of a database query is no better than a database query in the middle of a template:
<h1><?= mysql_query($my, "SELECT title FROM posts WHERE..."); ?></h1>

And finally, #4, OH GOD THE NESTING.

But then!

Months later, at a hack day, researching something completely different, I stumbled across jQuery.Deferred() and it was like I had come out of the cave and seen the light.

I had seen projects like tame.js, which I like in principle, but I find overly complicated in practice. I thought those were the options: callback hell or precompilers that spit out mounds of impenetrable code.

All of a sudden my functions could be functions again! They could take arguments instead of relying on scope. They could be run in parallel. They could be nice and flat!

var mongodb = require('mongodb');
var express = require('express');
var Deferred = require('Deferred');

var app = express.createServer();
var db = new mongodb.Db('mysite',
                        new mongodb.Server('localhost', 27027, {}),
                        {native_parser: true});

var Deferred = require('Deferred');

// Only do this once. Keep it somewhere else.
function getArray(db, coll, search, options) {
    var def = Deferred();
    db.open(function(err, db) {
        if (err) def.reject(err);
        db.collection(coll, function(err, collection) {
            if (err) def.reject(err);
            collection.find(search, options, function(err, cursor) {
                if (err) def.reject(err);
                cursor.toArray(function(err, arr) {
                    if (err) def.reject(err);
                    def.resolve(arr);
                });
            });
        });
    });
    return def.promise();
}

function home(req, res) {
    var uDef = getArray(db, 'users', {}, {'limit': 3});
    var aDef = getArray(db, 'articles', {}, {'limit': 4});
    var lookup = Deferred.when(uDef, aDef);
    lookup.done(function(users, articles)) {
        res.render('home.ejs', {'users': users, 'articles': articles});
    }).fail(function(err) {
        res.render('error.ejs', {'error': err})
    });
}

app.get('/', home);

Ah, That's Better

A lot of library code, like Dojo and jQuery, takes advantage of Promises and Deferreds to make things cleaner. I hope that more and more userland code will do the same.

Finding Deferreds helped me learn to love JavaScript again. It's such a fun, flexible, interesting language, so I'm glad complex code can be easy to test and fun to read.

James Socol

About James Socol

James Socol is a Python, JavaScript, and web developer and Mozilla's Community Platforms Manager. He is an active member of the Mozilla, Python, and Django communities and maintains several widely-used open source projects.

ydkjs-4.png

Recent Features

  • An Interview with Eric&nbsp;Meyer

    Your early CSS books were instrumental in pushing my love for front end technologies. What was it about CSS that you fell in love with and drove you to write about it? At first blush, it was the simplicity of it as compared to the table-and-spacer...

  • Create Namespaced Classes with&nbsp;MooTools

    MooTools has always gotten a bit of grief for not inherently using and standardizing namespaced-based JavaScript classes like the Dojo Toolkit does.  Many developers create their classes as globals which is generally frowned up.  I mostly disagree with that stance, but each to their own.  In any event,...

Incredible Demos

Discussion

  1. Matthew Iversen

    Hi there, in your last code block, under the // Only do this once. Keep it somewhere else. comment, you have a reptitive nested code block that looked very familiar to me.

    Perhaps watching this video will help you reduce it to something nicer? The video solves patterns that look exactly like that, which apparently involves creating monads! Very interesting.

    http://www.youtube.com/watch?v=46Z7Hq4fhN0

  2. A good read to accompany this post is this:

    https://gist.github.com/3889970

    “ You’re Missing the Point of Promises”, by the creator of Q, a CommonJS compatible promise implementation (sounds more complicated than it is). It’s a good introduction in what (and what not) promises are supposed to be.

  3. Nice post, although with jQuery deferred you now have to wrap lots of standard node.js calls in your own functions that resolve to deffereds. I currently prefer keeping my node code a bit simpler with a control flow library (sooooo many out there). I personally love Step (https://github.com/creationix/step)

  4. Hi! Great demonstration. I’m using this similar techniqe with success:
    https://github.com/caolan/async#waterfall

    Example:
    https://github.com/zsitro/backbone-facebook-login/blob/master/index.html#L135

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!