Streaming Data with Fetch() and NDJSON

By  on  

*"If you stream it, you can do it" -- Walt Disney[^1] *

Streams are trickling into the scene as we search for ways to improve performance. What if instead of waiting for our entire ajax response to complete, we could start showing the data as it arrives?

Streams allow us to do this. They are a data source that can be created and processed incrementally. This means as chunks of data become available, we are able to do work on them right away.

Using the Fetch API with a data format called NDJSON that breaks larger JSON objects into smaller JSON objects delimitated by newline characters, we are able to receive a stream of smaller chunks of JSON data as a stream. As our NDJSON data is streaming in, we can start processing and rendering right away. This makes users happy because they are able to see things sooner and developers happy because it increases overall performance. If you tie all of this in with service workers, then you can really see the improvements in performance.

incremental NDJSON rending vs. JSON object

Example Usage

As seen below, you can use fetch() with an endpoint that sends NDJSON to start manipulating and rendering that data line-by-line as you receive it.

streaming json process

Sounds like a win-win, but what is the catch? Well, there are packages out there on npm such as ndjson that serializes JSON to NDJSON, but we don’t have an easy way to deserialize NDJSON to a stream of JavaScript objects… until now!

Introducing can-ndjson-stream

can-ndjson-stream is a simple JavaScript module that does the heavy lifting of serializing your NDJSON stream into a ReadableStream of JavaScript objects. Use this just as you would JSON.parse with a JSON object.

ndjson streaming db row

Follow these simple steps to use the can-ndjson-stream module.

//1. require the can-ndjson-stream module
import ndjsonStream from 'can-ndjson-stream';

//2. pass the NDJSON stream response body to the ndjsonStream function. Typically, this would be used with the fetch() method.

const readableStream = ndjsonStream(response.body);  
//3. use readableStream, which is now a ReadableStream of JS objects, however you like. The ReadableStream API exposes a .getReader() and .cancel() method.

//https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream

Getting Started

1. Download the package from npm and save it to your project.

npm i can-ndjson-stream --save

2. Import the module as you would a typical npm module at the top of each file.

Require var ndjsonStream = require('can-ndjson-stream');

-- OR --

ES6 import import ndjsonStream from 'can-ndjson-stream';

3. Parse your response.body using the ndjsonStream function and do work on the result.

Once you parse your response.body, you can interact with your ReadableStream by read'ing it like this: exampleStream.getReader().read(). That will return a promise that resolves to one line of your NDJSON.

Using Async/Await

import ndjsonStream from 'can-ndjson-stream';

const fetchNdjson = async () => {
  const response = await fetch("/api");
  const exampleReader = ndjsonStream(response.body).getReader();

  let result;
  while (!result || !result.done) {
    result = await exampleReader.read();
    console.log(result.done, result.value); //result.value is one line of your NDJSON data
  }
}

Using promises

import ndjsonStream from 'can-ndjson-stream';

fetch('/api')  // make a fetch request to a NDJSON stream service
.then((response) => {
	return ndjsonStream(response.body); //ndjsonStream parses the response.body

}).then((exampleStream) => {
	let read;
	exampleStream.getReader().read().then(read = (result) => {
		if (result.done) return;
		console.log(result.value);

		exampleStream.getReader().read().then(read); //recurse through the stream
	});
});

4. [Optional] Create a simple API to serve sample NDJSON data.

Follow this tutorial on the Bitovi blog that gets you started or take a look at the demo in the can-ndjson-stream repo.

What next?

If you enjoyed this article, tweet to us how you plan to use this module! Also check out the rest of the CanJS library. If you need any help, please don’t be afraid to leave a comment below or ask questions in the CanJS Gitter or forums!

Inspirational Quotes about Streams Through History[^1]

If not us, who? If not now, when? -- John F. Kennedy

Hope is a waking stream. -- Aristotle

Life is trying things to see if they work. -- Ray Bradbury

[^1]: may or may not be accurate.

[^2]: NDJSON stands for newline delimited JSON. Each newline character in a file separates that line into individual JSON objects enabling us to stream many smaller chunks of JSON rather than one large JSON object.

Bianca Gandolfo

About Bianca Gandolfo

Bianca is a JavaScript enthusiast currently working on the open source team at Bitovi. She likes to pretend she is coding away on the more important bits of the DoneJS framework in some hacker basement, but most of the time she is just writing documentation and fixing bugs near the beach. When she's not doing that, she practices getting sunburnt while wiping out repeatedly on a surfboard and writes JS curriculum for coding bootcamps around the world. Check out her courses on Data Structures and Algorithms in JS and JS Foundations for Functional Programming on Frontend Masters.

Recent Features

  • By
    Create a CSS Cube

    CSS cubes really showcase what CSS has become over the years, evolving from simple color and dimension directives to a language capable of creating deep, creative visuals.  Add animation and you've got something really neat.  Unfortunately each CSS cube tutorial I've read is a bit...

  • By
    Responsive and Infinitely Scalable JS Animations

    Back in late 2012 it was not easy to find open source projects using requestAnimationFrame() - this is the hook that allows Javascript code to synchronize with a web browser's native paint loop. Animations using this method can run at 60 fps and deliver fantastic...

Incredible Demos

  • By
    CSS Triangles

    I was recently redesigning my website and wanted to create tooltips.  Making that was easy but I also wanted my tooltips to feature the a triangular pointer.  I'm a disaster when it comes to images and the prospect of needing to make an image for...

  • By
    Styling CSS Print Page Breaks

    It's important to construct your websites in a fashion that lends well to print. I use a page-break CSS class on my websites to tell the browser to insert a page break at strategic points on the page. During the development of my...

Discussion

  1. Hari

    Getting below error if i use the promises code at line exampleStream.getReader().read().then(read);

    Uncaught (in promise) TypeError: ReadableStreamReader constructor can only accept readable streams that are not yet locked to a reader

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!