From Webcam to Animated GIF: the Secret Behind chat.meatspac.es!

My team mate Edna Piranha is not only an awesome hacker; she's also a fantastic philosopher! Communication and online interactions is a subject that has kept her mind busy for a long time, and it has also resulted in a bunch of interesting experimental projects that have fostered an unexpected community and tons of spontaneous collaboration, plus have helped unearth a browser bug or two!

We could spend hours just going through her list of projects and getting amazed by all the ways in which she's approaching the same aspect (human interaction) from different angles, both philosophical and technical, but this isn't Edna Piranha's Fan Club Blog, and David only asked me to write about Animated GIFs in the widely successful Meatspace Chat, so let's focus on that.

It all started about a year ago. Edna had just built a decentralised microblogging engine with Redis, and was trying to find a use case for a database she had just heard about, called LevelDB.

She showed me a real time chat app she had hacked in a couple of hours, using LevelDB as the temporary, ephemeral storage. Anyone could sign in using Persona, and start sending messages to the one chat room. The avatar associated to your Persona account would be shown along with the message you sent, and messages would be deleted after a few minutes.

By that time I had been working on rtcamera, a camera app that could generate animated GIFs using your webcam as input, and somehow our thought trails converged: wouldn't it be super cool to use the webcam's input instead of an static avatar?

It was easy to implement this using the two libraries that I had extracted out from rtcamera: gumHelper and Animated_GIF, and the rest is history!

But for those of you who don't know about history: we kept the chat in private for a while because Edna was going to present it at RealtimeConf. And then... it just exploded! People started coming to the site in flocks and being both puzzled by the unexpected cheerness and the general Back to the True Web raw and honest spirit: no sign up forms, no name to fill in, no identity to build and maintain; just a text input and your face to show the world what you were up to in that very moment. If you haven't been to Meatspaces Chat yet, I recommend you go there now to familiarise yourself with how it looks and works before I get into technical details. You can also watch Edna's keynote at jQuery Con San Diego, where she talks about all this.

To the juicy technical details!

Are you all intrigued now? Cool!

But before we start deep diving into the code, let me add a little warning: Meatspaces chat is constantly being improved by the amazing community, so I will be referring to lines using a specific commit hash too. If you go directly to the project page and access the master branch, both the code and line numbers might differ from what this article says.

And we are really ready to go!

Accessing the camera

Everything starts with requesting access to the user's camera. We are using the gumHelper library for this. No, it has nothing to do with dental hygiene; it actually means "getUserMediaHelper", where getUserMedia is the part of the WebRTC API that allows us to obtain a live media stream containing live audio or video which we can then use in our websites. In this case we're only interested in video, as GIFs are (sadly) silent.

If you're running this on a laptop or a desktop--i.e. a full blown computer-- we'll access the webcam. If you're running this on a phone, it will not only ask you for permission to use the camera, but also show you a drop down so you can select which camera to use, if applicable (some devices only have a back camera).

We'll attempt to start streaming by calling gumHelper.startVideoStreaming:

gumHelper.startVideoStreaming(function (err, stream, videoElement, videoWidth, videoHeight) {
    // ...
}, { /* options */ });

startVideoStreaming takes a callback and an optional options object as parameters. In fairly standard node.js style, the callback function first parameter is err, which we check first. If it is truthy, we just give up on accessing the video. In earlier versions of the site, your messages would be accompanied by a giant meat cube avatar if video wasn't enabled for whatever the reason, but it was changed to disallow sending messages to prevent trolls from posting.

Supposing the stream was successfully started, the next step is to use the videoElement returned by gumHelper. This is just a plain HTML5 <video> element that we will place in the page to serve as preview, so the user can ensure they are in the frame when they press ENTER.

Capturing frames

The other thing we're doing is creating a VideoShooter instance. This is a little class that attaches to an existing video element and will start generating a GIF whenever we press ENTER, using frames from that video element:

videoShooter = new VideoShooter(videoElement, gifWidth, gifHeight, videoWidth, videoHeight, cropDimens);

The function to get a video capture is VideoShooter.getShot, which accepts a few parameters: callback (called to return the encoded GIF), numFrames (to specify how many frames to capture), interval (for setting the interval between capturing frames) and progressCallback (which is used to show a sort of progress indicator overlay over the video preview).

Internally, what getShot does is creating an instance of Animated_GIF and then periodically tells it to capture a frame as many times as requested, using Animated_GIF's addFrame method.

How often the frames are captured (and therefore how smooth the animation will be) depends on the interval parameter. The more frames and the more frequently they are captured, the better and less jerky the GIF will look, but it will also be bigger. We played a bit with the parameters and decided to settle on two second GIFs (10 frames shot every 0.2 seconds make 2 seconds). Hence the "lemma" of the site: "your two seconds of fame".

Animating the GIF

Each time we add a frame to the Animated_GIF instance, we pass videoElement as source parameter. It is then copied into an internal canvas to extract the image data and store it on a list of frames, taking advantage of the drawImage function that allows you to render HTML elements into CanvasRenderingContext2D objects.

Once the ten frames have been captured, the VideoShooter instance will call the getBase64GIF method from Animated_GIF.

This part is probably the most involved of all in the whole process, since we are ultimately generating binary data in JavaScript. Fortunately, it is all abstracted enough that we only need to call the method and wait for it to be generated on the background using Web Workers.

We use Web Workers because rendering is quite an intensive process and can easily block the main thread, making the whole app unresponsive--that's something we don't want to happen!

The callback function is invoked and sent the rendered GIF when it's ready. Since it is a Base64 string we can just include it without further processing on the submission object that is then posted to the server.

And that's how your funny faces get captured and travel down the wire to people all over the world. Or almost!

GIFWall

I thought that maybe perusing the entire codebase of Meatspaces Chat would be a bit too much if you're only interested in the GIF side of things, so I build this little demo app that periodically captures GIFs using your webcam and adds them to the page.

It also uses gumHelper, Animated_GIF and a simplified version of the VideoShooter module.

To demonstrate how easy it is to capture data from the webcam and turn it into a GIF with the right libraries to abstract the tedium, here is the main code from GIFwall:

var main = document.querySelector('main');
var mosaicContainer = document.getElementById('mosaic');
var videoWidth= 0, videoHeight = 0;
var videoElement;
var shooter;
var imagesPerRow = 5;
var maxImages = 20;

window.addEventListener('resize', onResize);

GumHelper.startVideoStreaming(function(error, stream, videoEl, width, height) {
    if(error) {
        alert('Cannot open the camera. Sad times: ' + error.message);
        return;
    }

    videoElement = videoEl;
    videoElement.width = width / 4;
    videoElement.height = height / 4;
    videoWidth = width;
    videoHeight = height;

    main.appendChild(videoElement);

    shooter = new VideoShooter(videoElement);

    onResize();

    startCapturing();

});

function startCapturing() {

    shooter.getShot(onFrameCaptured, 10, 0.2, function onProgress(progress) {
        // Not doing anything in the callback,
        // but you could animate a progress bar or similar using the `progress` value
    });

}

function onFrameCaptured(pictureData) {
    var img = document.createElement('img');
    img.src = pictureData;

    var imageSize = getImageSize();

    img.style.width = imageSize[0] + 'px';
    img.style.height = imageSize[1] + 'px';

    mosaicContainer.insertBefore(img, mosaicContainer.firstChild);

    if(mosaicContainer.childElementCount > maxImages) {
        mosaicContainer.removeChild(mosaicContainer.lastChild); 
    }

    setTimeout(startCapturing, 10);
}

function getImageSize() {
    var windowWidth = window.innerWidth;
    var imageWidth = Math.round(windowWidth / imagesPerRow);
    var imageHeight = (imageWidth / videoWidth) * videoHeight;

    return [ imageWidth, imageHeight ];
}

function onResize(e) {

    // Don't do anything until we have a video element from which to derive sizes
    if(!videoElement) {
        return;
    }

    var imageSize = getImageSize();
    var imageWidth = imageSize[0] + 'px';
    var imageHeight = imageSize[1] + 'px';

    for(var i = 0; i < mosaicContainer.childElementCount; i++) {
        var img = mosaicContainer.children[i];
        img.style.width = imageWidth;
        img.style.height = imageHeight;
    }

    videoElement.style.width = imageWidth;
    videoElement.style.height = imageHeight;

}

This is essentially Meatspace Chat, but without chatting and without sending the data to other connected people. Some homework for the reader could be to show a progress bar or other fancy similar effect while GIFs are being encoded, or even improve this so that the captured GIFs are actually sent to other users via real peer to peer connections over WebRTC.

There are so many things you can do on the web nowadays! Isn't that exciting? Now go get the sources, play with the code and have fun, and don't forget to share your work so we can all learn and have fun too! :-)

About Soledad Penadés

Sole wants the web to be awesome, fun and a platform for creative expression. She likes to play with Web APIs, write code that generates often realtime graphics and music, and share her findings so that more people can become creators rather than consumers.

soledadpenades.comsupersolePosts