Do a Basic HTTP Request with Node.js

By  on  

I'm currently working on adding client-side testing to the Mozilla Developer Network (MDN).  I'm using Intern, a JavaScript-based WebDriver API created by my former employer SitePen.  MDN has used its own Persona login service for years, and since many key features of MDN require login (and subsequent account creation), it was important that I have a way to get test credentials.  There's a service that does provide those credentials, but that requires that I make a HTTP request from inside a test helper.

I usually use the popular request module, available on npm, but I wanted to avoid external dependencies outside of the testing library.  I had always heard that dealing with HTTP requests with the native Node.js API was a nightmare, but after some investigation, I found what I needed was actually incredibly easy.  Here's a reduced, extremely simple example of making a HTTP request with Node.js:

var http = require('http');

function getTestPersonaLoginCredentials(callback) {

    return http.get({
        host: 'personatestuser.org',
        path: '/email'
    }, function(response) {
        // Continuously update stream with data
        var body = '';
        response.on('data', function(d) {
            body += d;
        });
        response.on('end', function() {

            // Data reception is done, do whatever with it!
            var parsed = JSON.parse(body);
            callback({
                email: parsed.email,
                password: parsed.pass
            });
        });
    });

},

The only out of the ordinary part is the need to concatenate the incoming data stream, but apart from that, the process is actually pretty simple.  Of course you can add more checks for response status codes and the like, but there's a dead simple example of creating a HTTP request with the native Node.js code!

Recent Features

  • By
    JavaScript Promise API

    While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...

  • By
    Interview with a Pornhub Web Developer

    Regardless of your stance on pornography, it would be impossible to deny the massive impact the adult website industry has had on pushing the web forward. From pushing the browser's video limits to pushing ads through WebSocket so ad blockers don't detect them, you have...

Incredible Demos

  • By
    MooTools CountDown Plugin

    There are numerous websites around the internet, RapidShare for example, that make you wait an allotted amount of time before presenting you with your reward. Using MooTools, I've created a CountDown plugin that allows you to easily implement a similar system. The MooTools JavaScript The CountDown class...

  • By
    Create a Dynamic Table of Contents Using MooTools 1.2

    You've probably noticed that I shy away from writing really long articles. Here are a few reasons why: Most site visitors are coming from Google and just want a straight to the point, bail-me-out ASAP answer to a question. I've noticed that I have a hard time...

Discussion

  1. MaxArt

    Never got the fuss about the native HTTP client of node.js either. Sure, request is nicer, but the usage of the native class has always been quite straightforward to me.

    You know what’s pain? Using curl in php, for example. If you tried it, node.js is a piece of cake.

  2. Nice post, DW, always good to see you messing with node!

    Just wanted to suggest a few tweaks. Some of these are orthogonal to making the actual HTTP request, but given that people love copy/pasta I thought I’d make mention of them here.

    1) JSON.parse(body) can throw, which in node-land means your whole server dies (if uncaught). I’d definitely recommend wrapping that statement in a try/catch.
    2) Node’s callback conventions expect the first parameter to be an error, if one occurred, and the result should be passed as the second parameter. Not saying you *have* to follow that convention, but it’s probably a good idea for generic code examples.
    3) It’s also possible for an error to occur while making the request itself (e.g., errors in DNS resolution, HTTP parsing, making a connection, etc.). If unhandled, those potential errors will be thrown globally, and again risk crashing the server.
    4) Especially when parsing results as JSON, it’s a good idea to explicitly set the response encoding to utf8, otherwise you can run into subtle issues with multi-byte characters and their ilk.

    Not that I’d ever want to contribute to the node horror stories, but it’s important to stress that the http module is pretty low level, and the API almost assumes that end users will deal with a higher level abstractions over it (such as the request module you mentioned). It’s certainly not impossible to hand-sling your own HTTP requests with the http module, but it does assume a fair amount of rigor and familiarity with the API itself. Here’s my stab at a more robust / error-tolerant approach:

    https://gist.github.com/jmar777/87fc99cfe3ec27d88e6e

    • Thanks for the improvements!

    • alex

      Jeremy, thanks for taking your time share the improvements

    • This is lovely, thanks for share this.

  3. Great article! If you used concat-stream you could simplify this even more:

    var concat = require('concat-stream');
    
    function getTestPersonaLoginCredentials(callback) {
    
        return http.get({
            host: 'personatestuser.org',
            path: '/email'
        }, function(response) {
            response.pipe(concat(function(body) {
                // Data reception is done, do whatever with it!
                var parsed = JSON.parse(body);
                callback({
                    email: parsed.email,
                    password: parsed.pass
                });
            }))
        });
    
    }
    
  4. Very interesting article.
    I have a question about testing, I’d like to write a test for getTestPersonaLoginCredentials but, obviously, I don’t like to make real http call. Can I mockup http module? Can be used Sion for this?

    Thanks
    Fabrizio

  5. Howdy, nice example that I would like to convert to using node module request. Obviously, your example does not run at the node command-line since it is a function. Howto execute your example @ the node command-line?

  6. MikeC711

    I am a node.js newbie … have done apache httpClient in java. Seems simple, but I’m sure I’m missing something. I have the needed reqGet.end() (tried using simpler http.get method, but ran into this problem) … seems that the code AFTER reqGet.end() gets executed BEFORE the code inside the request. This caused me to go ahead and respond to my caller, before having written to the response stream. The http call succeeded (I can tell from the data base), but the client got nothing. Is there some sort of sync I need to do beyond the end() call? Thanks

  7. How are you executing the code? Can this be run from HTML front-end? If so, can someone share a working example?

  8. partycoder

    end only happens once. Therefore the event needs to be subscribed using once instead of on.

    Additionally, both the response event listeners both use data. This can is a circular reference and can cause a potential memory leak. It is strongly recommended that you null the body variable on end.

    • What does your ideal function look like then? Snippet?

  9. Dinesh

    Hi David,

    I tried to make https request. Can you show me the code how to make it?

    • I suppose you can use https module instead of http for that.

      var https = require("https"); // instead of http, use https
      
  10. I know this is 2 years old now but I thought I would share.

    I just started working with node today but have been doing web development for years. I took a guess and installed Jquery and tried their $.get and it works just as well as it does on the front end.

    This was in an atom electron app.

    var $ = require('jquery');
    
    $.get('http://jsonplaceholder.typicode.com/posts').then(cb);
    
  11. Rosen

    Thank you! I’m surprised how much searching I had to do to find an example of this — much appreciate you posting this!

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!