Do a Basic HTTP Request with Node.js
I'm currently working on adding client-side testing to the Mozilla Developer Network (MDN). I'm using Intern, a JavaScript-based WebDriver API created by my former employer SitePen. MDN has used its own Persona login service for years, and since many key features of MDN require login (and subsequent account creation), it was important that I have a way to get test credentials. There's a service that does provide those credentials, but that requires that I make a HTTP request from inside a test helper.
I usually use the popular request module, available on npm, but I wanted to avoid external dependencies outside of the testing library. I had always heard that dealing with HTTP requests with the native Node.js API was a nightmare, but after some investigation, I found what I needed was actually incredibly easy. Here's a reduced, extremely simple example of making a HTTP request with Node.js:
var http = require('http');
function getTestPersonaLoginCredentials(callback) {
return http.get({
host: 'personatestuser.org',
path: '/email'
}, function(response) {
// Continuously update stream with data
var body = '';
response.on('data', function(d) {
body += d;
});
response.on('end', function() {
// Data reception is done, do whatever with it!
var parsed = JSON.parse(body);
callback({
email: parsed.email,
password: parsed.pass
});
});
});
},
The only out of the ordinary part is the need to concatenate the incoming data stream, but apart from that, the process is actually pretty simple. Of course you can add more checks for response status codes and the like, but there's a dead simple example of creating a HTTP request with the native Node.js code!
Never got the fuss about the native HTTP client of node.js either. Sure, request is nicer, but the usage of the native class has always been quite straightforward to me.
You know what’s pain? Using curl in php, for example. If you tried it, node.js is a piece of cake.
Amen — PHP + cURL is a nightmare API!
Nice post, DW, always good to see you messing with node!
Just wanted to suggest a few tweaks. Some of these are orthogonal to making the actual HTTP request, but given that people love copy/pasta I thought I’d make mention of them here.
1)
JSON.parse(body)can throw, which in node-land means your whole server dies (if uncaught). I’d definitely recommend wrapping that statement in a try/catch.2) Node’s callback conventions expect the first parameter to be an error, if one occurred, and the result should be passed as the second parameter. Not saying you *have* to follow that convention, but it’s probably a good idea for generic code examples.
3) It’s also possible for an error to occur while making the request itself (e.g., errors in DNS resolution, HTTP parsing, making a connection, etc.). If unhandled, those potential errors will be thrown globally, and again risk crashing the server.
4) Especially when parsing results as JSON, it’s a good idea to explicitly set the response encoding to utf8, otherwise you can run into subtle issues with multi-byte characters and their ilk.
Not that I’d ever want to contribute to the node horror stories, but it’s important to stress that the
httpmodule is pretty low level, and the API almost assumes that end users will deal with a higher level abstractions over it (such as the request module you mentioned). It’s certainly not impossible to hand-sling your own HTTP requests with the http module, but it does assume a fair amount of rigor and familiarity with the API itself. Here’s my stab at a more robust / error-tolerant approach:https://gist.github.com/jmar777/87fc99cfe3ec27d88e6e
Thanks for the improvements!
Jeremy, thanks for taking your time share the improvements
This is lovely, thanks for share this.
Great article! If you used
concat-streamyou could simplify this even more:var concat = require('concat-stream'); function getTestPersonaLoginCredentials(callback) { return http.get({ host: 'personatestuser.org', path: '/email' }, function(response) { response.pipe(concat(function(body) { // Data reception is done, do whatever with it! var parsed = JSON.parse(body); callback({ email: parsed.email, password: parsed.pass }); })) }); }Very interesting article.
I have a question about testing, I’d like to write a test for getTestPersonaLoginCredentials but, obviously, I don’t like to make real http call. Can I mockup http module? Can be used Sion for this?
Thanks
Fabrizio
Howdy, nice example that I would like to convert to using node module request. Obviously, your example does not run at the node command-line since it is a function. Howto execute your example @ the node command-line?
I am a node.js newbie … have done apache httpClient in java. Seems simple, but I’m sure I’m missing something. I have the needed reqGet.end() (tried using simpler http.get method, but ran into this problem) … seems that the code AFTER reqGet.end() gets executed BEFORE the code inside the request. This caused me to go ahead and respond to my caller, before having written to the response stream. The http call succeeded (I can tell from the data base), but the client got nothing. Is there some sort of sync I need to do beyond the end() call? Thanks
How are you executing the code? Can this be run from HTML front-end? If so, can someone share a working example?
endonly happens once. Therefore the event needs to be subscribed usingonceinstead ofon.Additionally, both the response event listeners both use
data. This can is a circular reference and can cause a potential memory leak. It is strongly recommended that you null thebodyvariable on end.What does your ideal function look like then? Snippet?
Hi David,
I tried to make https request. Can you show me the code how to make it?
I suppose you can use
httpsmodule instead ofhttpfor that.var https = require("https"); // instead ofhttp, usehttpsI know this is 2 years old now but I thought I would share.
I just started working with node today but have been doing web development for years. I took a guess and installed Jquery and tried their $.get and it works just as well as it does on the front end.
This was in an atom electron app.
var $ = require('jquery'); $.get('http://jsonplaceholder.typicode.com/posts').then(cb);Thank you! I’m surprised how much searching I had to do to find an example of this — much appreciate you posting this!