stdlib: Create Scalable Node.js Microservices in a Flash

By Keith Horwood on October 26, 2016

Hey everyone - today I have the honor of walking you through using the brand new service registry for microservices, stdlib. You can also check out stdlib on GitHub, which is the open source project we'll be using to generate service scaffolding and take care of package management.

What is stdlib?

Easy! It's a new take on software registries and package management that you're probably familiar with, like NPM. Instead of focusing on local installations of software, stdlib allows you to write simple microservices with multiple functional HTTP endpoints that you access over-the-wire as remote procedure calls. You can then register these services on the stdlib central registry, at which point they'll be discoverable by others (if you choose to publish them) and also be completely functional web backends. It's actually completely cross-compatible with NPM, if you decide you want to publish your service for local installation as well.

The stdlib registry itself is free to use, for anybody. Instead of others downloading your code, your service itself remains closed-source, with only HTTP endpoints exposed. The actual contents downloadable only by you and your team. Your services can also be run for free, as of writing this article, for up to 500,000 seconds of computation time.

You can think about using stdlib a little bit like a mix between NPM and Heroku for microservices, focused on smaller, functional application parts. The potential is vast, you can build a service with any amount of complexity you'd like and never worry about scale or managing infrastructure.

Getting Started

To get started with stdlib, you'll first have to download the command line tools, available on NPM. You should first have Node 6.X installed, available here, at nodejs.org.

Once you've installed node (or if it's installed already), simply run:

$ npm install -g stdlib.com@1.0.0-dev5

This installs the Developer Preview for CLI tools. There may be newer versions available, so always check out stdlib on GitHub when building new microservices. :)

Initializing a Workspace

To create a stdlib workspace that contains all of your in-development functions, first create a directory you'll be developing in and then initialize stdlib.

$ mkdir stdlib
$ cd stdlib
$ stdlib init

When initializing a workspace, you'll be asked to enter the e-mail address you used to sign up for the registry. You can skip this step with --no-login, but it's not recommended - you won't be able to push code without it! If you don't have an account yet, you'll be able to create one.

Creating a Service

Creating a service is really simple. No need to create a new directory. You should do this from the top-level workspace directory.

$ stdlib create

You'll be asked to add a service name, and a default function name. The default function will be the "root" web mapping of your service, if no additional function information is given - though services can support more than one function.

Anatomy of a Service

Services have four main components;

package.json

package.json is your main service package. This is an NPM-compatible file that contains a "stdlib" field with the following properties:

"name": The name your service will be published to. Usually in the format <username>/<service>. In order to push your service to the stdlib registry, you'll need to have access to the specified <username>.
"defaultFunction": The default entry point to your service if no function is provided. Services are accessed by <username>/<service>/<function>, so specifying this field as "main" would make <username>/<service> map to <username>/<service>/main.
"timeout": The timeout, in milliseconds, for service execution. This is the upperbound of compute time applied to all functions in the service.
"publish": Whether or not to publish publicly to the central registry when your service is pushed to the cloud. A list of public services can be found at the stdlib search page.

env.json

A JSON file containing environment variables to be sent to the process.env variable. While working locally, services will load anything in "dev", and when you push a release to stdlib, services will use the values contained in "release". Any other environment names exist for staging purposes in the cloud.

function.json

You'll notice a folder, f/, in your main service directory. If you specified your default function name as main, you should see:

f/main/function.json
f/main/index.js

In your f/ directory. This is your first functional endpoint!

function.json is a JSON file with a few fields;

"name": The function name. Must match f/<function_path> exactly, or the registry will throw an error.
"description": A short description of the function.
"args": An array containing information about the arguments the function expects. More on this in a bit.
"kwargs": An object (key-value pairs) containing information about the keywords arguments the function expects. More on this in a bit, too.

index.js

This is paired with a function.json file in a function directory. It's a simple function of the format;

module.exports = (params, callback) => {
  callback(null, 'hello world');
};

Where params is an object containing the arguments (params.args) and keyword arguments (params.kwargs) passed to the function.

callback is a function callback that ends function execution, expecting an error argument (or null if no error) and a JSON-serializable result (or a Buffer for file processing).

Using Your Service Locally

The first service you created from stdlib create should have one function that returns "hello world". To run this function (and test it), first go to your service directory:

$ cd <username>/<service>

If your username is best_developer and you created a service named test, you would do:

$ cd best_developer/test

Now simply type:

$ f .
> "hello world"

Amazing! :) stdlib comes packaged with a command-line testing tool called f, that relies upon the f package from NPM / GitHub to do microservice testing. If you begin your function path with a ., it will search for a local function, otherwise it will run a live function, in the cloud.

Note that this execution is equivalent to;

$ f ./main
> "hello world"

If your "defaultFunction" is set to "main".

Registering Your Service in the Cloud

There are two ways to register your service. Either in a staging environment, or as a release. Staging environments are mutable and can be overwritten at will. Releases are immutable and can not be overwritten, but they can be torn down.

Register as a Staging Service

To push your service to the cloud in a staging environment, use:

$ stdlib up <environment>

Where <environment> is the name of your intended staging environment. This name maps directly to the process.env variables stored in env.json, so make sure you use the right name. Your local service runs in dev environment, so that's a good target to test in the cloud as well:

$ stdlib up dev

This will now register and compile your service. Once complete, you'll be able to run your service in the cloud using;

$ f <username>/<service>@dev
> "hello world"

Or, using the example above:

$ f best_developer/test@dev
> "hello world"

You can also access your service over HTTPS via curl or in your browser;

https://f.stdlib.com/best_developer/test@dev

Voila! :)

Register as a Release

Releases are immutable, so make sure you're confident in what you're pushing because though they can be torn down, versions can never be overwritten (or go "back" by semver standards). To release a service, use;

$ stdlib release

This will register your service with the version specified in package.json. If you would like your service publicly searchable on the registry, set "publish": true in the "stdlib" field of package.json.

You can view a list of published releases here.

Removing a Service

To remove a service, simply type;

$ stdlib down <environment>

$ stdlib down -r <version>

Note that stdlib rollback can also be used as a shortcut to remove the currently specified release if published by accident.

Restarting or Rebuilding a Service

If your service, once published to the cloud, for any reason stops working you can try restarting it with;

$ stdlib restart <environment> [-r <version>]

This shouldn't be necessary, but the option exists should you encounter any errors.

Additionally, you can also rebuild a service. This reinstalls package dependencies and uses the most up-to-date version of the stdlib microservice software. This may be encouraged as we roll out updates, for performance and security reasons.

$ stdlib rebuild <environment> [-r <version>]

Creating More Service Functions

To create additional endpoints to your service (more functions!), simply type:

$ stdlib f:create <function>

This will create a new "hello world" function given the newly specified name. Modify it to your heart's content! It is not a default function, so it will need to be accessed using (provided the name new-func);

$ f ./new-func

Arguments and Keyword Arguments

To pass different params.args and params.kwargs values to your function, use:

$ f . arg0 arg1 --kwarg0 value --kwarg2 "another value"

When accessing services via HTTP GET, all query parameters are converted to keyword arguments.

https://f.stdlib.com/user_name/service@version?name=Ed%20Harris

Would set params.kwargs.name equal to the string "Ed Harris".

You can also pass any number of arguments or keyword arguments from another application using the f package, as described below.

Accessing Your Service From Other Applications

Any service that you put on the stdlib registry can be accessed via other applications using the HTTP endpoints for service, at:

https://f.stdlib.com/user_name/service@version/function

Node.js

To make this easier, using Node.js you can install the f library, available on GitHub. In the project you wish to access your service from, use:

$ npm install f --save

and then in your code...

const f = require('f');

f('user_name/service@version/function')(arg1, arg2, {kwarg1: 'value'}, (err, result) => {

  if (err) {
    // service errored
  }

  // do something with `result` here!

});

Web Browser

The f library can be used identically in the browser. It's part of the same GitHub repository and can be installed via Bower using poly/f.

That's it!

That's all you need to get started with stdlib. I definitely hope you enjoy using it as much as I've enjoyed building it. Stay tuned for more articles showing some sample functions and neat things you can do with microservices.

Good luck, have fun, and happy building!

FAQ

You made it this far, eh? Very thankful for David Walsh for hosting this and being as excited as I am for the future of the project. :)

Why the name stdlib?

It's a play on the C Standard Library, or #include <stdlib.h>. We were wondering what you'd call a registry for remote procedure calls on the web and stdlib.com just made the most sense!

Why args (arguments) and kwargs (keyword arguments)?

It's a specification from Python, but it works, makes sense, and maps to a ton of other languages we'll be building SDKs for. This way developers can write functions that only expect unnamed arguments, or can specify the names of the arguments they'd like to have passed in.

Why use stdlib? Why not, say, AWS Lambda?

stdlib service hosting is actually built on top of AWS Lambda. If that's what you prefer, by all means, we don't want to impede you. stdlib isn't a replacement for Lambda, it's a registry! It's a different offering altogether that augments the "server-less" model to ease your mind completely and give you workflows that your team can organize around so you don't have to build them yourself. stdlib provides easier version control, package management, team tooling and other nifty features.

Who created stdlib?

stdlib was created in a basement by a dog named Ruby. Some say you can still hear her panting when you register a service. All jokes aside, my name is Keith Horwood and I'm known for doing Node and open source stuff. (Ruby is my dog though, and she's beautiful.) I'm probably best known for authoring the popular open source API framework, Nodal. You can follow me on Twitter, @keithwhor but it would be better if you followed our whole company - @Polybit!

How can I contribute?

Submit a pull request or open an issue on the stdlib GitHub repository. If you want to help us create the future of "server-less" technology, you can also apply to work with us at Polybit. We're new, we're growing, and if you're enthusiastic with good ideas, we want you to join and tell us how to make everything better :).

About Keith Horwood

Keith is a self-taught developer from Toronto, Canada with a passion for learning and building. Formerly the Engineering Lead at Storefront, his most recent adventure is building Polybit, a service that allows to developers to more effectively build and deploy their application backends so they can focus on their product and their customers.