HTTP Requests in Code

Dr. Greg Bernstein

Updated October 18th, 2021

HTTP Requests

Learning Objectives

Learn how to make HTTP requests from JavaScript code
Understand why you would make requests outside of a browser
Understand and use a Node.js library to make these requests

Node.js and Browser APIs

HTTP requests are subject to unpredictable and “relatively” long delays compared to local processing
To avoid blocking the node-fetch library for Node.js and the browser’s fetch API are asynchronous
Both of these APIs return JavaScript Promises

Applications of non-browser HTTP requests

Automating web queries, e.g., “screen scraping”, web site monitoring…
Testing our server implementations
Implementing multi-server architectures such as micro-services
Creating Proxies

Node.js HTTP Request Libraries

Example code RequestsNode.zip
node-fetch Provides an interface very similar to the browsers fetch API, but in Node.js. Very Popular. We will use this.
The original Request - Simplified HTTP client Was very popular but is now Deprecated along with its derivatives.

Node-Fetch Library

We’ll use node-fetch

Installation:

npm install --save node-fetch

Documentation

Since node-fetch is so similar to fetch we will use both documentation sets.

API

node-fetch API

fetch(url[, options]) – Perform an HTTP(S) fetch

url A string representing the URL for fetching
options Options for the HTTP(S) request
Returns: Promise
url should be an absolute URL, such as https://example.com/.

Options

{
    // These properties are part of the Fetch Standard
    method: 'GET',
    headers: {},        // request headers. format is the identical to that accepted by the Headers constructor (see below)
    body: null,         // request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream
    redirect: 'follow', // set to `manual` to extract redirect headers, `error` to reject redirect
    signal: null,       // pass an instance of AbortSignal to optionally abort requests
 
    // The following properties are node-fetch extensions
    follow: 20,         // maximum redirect count. 0 to not follow redirect
    timeout: 0,         // req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies). Signal is recommended instead.
    compress: true,     // support gzip/deflate content encoding. false to disable
    size: 0,            // maximum response body size in bytes. 0 to disable
    agent: null         // http(s).Agent instance or function that returns an instance (see below)
}

Request Example HTML Page 1

From simpleRequest.mjs

import fetch from 'node-fetch';
// Returns a promise that resolves to a response object
let myPromise = fetch("http://www.grotto-networking.com");

myPromise.then(function(res) { // Work with response
    console.log(`\nResponse from ${res.url}`);
    console.log(`Status: ${res.status}`);
    console.log(`Status code: ${res.statusText}`);
});

Getting the Body

From MDN Response/Body. These take “a Response stream and reads it to completion.”

Body.arrayBuffer() returns a promise that resolves with an ArrayBuffer.
Body.blob() returns a promise that resolves with a Blob.
Body.formData() returns a promise that resolves with a FormData object.
Body.json() returns a promise that resolves with the result of parsing the body text as JSON
Body.text() returns a promise that resolves with a string.

Request Example HTML Body

use response/body interface to get content. bodyRequest.mjs

import fetch from 'node-fetch';
// A promise for the response
let myRes = fetch("http://www.grotto-networking.com");
// A promise for the body
let myBody = myRes.then(function(res) { // Work with response
    console.log(`${res.statusText} response for ${res.url}`);
    // returns a promise for the body
    return res.text(); // or json(), or blob(), etc...
});

myBody.then(function(body) {
    console.log(`The length of the body ${body.length}`);
    console.log("Body contents:");
    console.log(body.slice(0, 100)); // look at the beginning
})

Example: output

OK response for http://www.grotto-networking.com/
The length of the body 10232
Body contents:
<!DOCTYPE html><html lang="en"><head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatib

Bad Promise Technique

Don’t nest code in .then()! ickBodyRequest.mjs

import fetch from 'node-fetch';

fetch("http://www.grotto-networking.com")
    .then(function(res) { // Work with response
        console.log(`${res.statusText} response for ${res.url}`);
        // Don't do this! Return the promise!
        res.text()
            .then(function(body) { // pyramid of doom starting
                console.log(`The length of the body ${body.length}`);
                console.log("Body contents:");
                console.log(body.slice(0, 100)); // look at the begining
            })
    });

Getting JSON Data

From jsonRequest.mjs

import fetch from 'node-fetch';

fetch("https://windsurf.grotto-networking.com/data/logs/windEvents2018.json")
    .then(function(res) { return res.json() }) // get body as JSON object
    .then(function(myObj) { // look at object
        console.log(`Number of Log entries: ${myObj.length}`);
        console.log(`Log entry 0: ${myObj[0].desc}`);
        console.log(myObj);
    });

All the Header Info

Using the MDN Headers API. studentRequests.mjs

import fetch from 'node-fetch';

fetch("http://www.grotto-networking.com")
    .then(function(res) { // Work with response
        lookAtResponse(res);
        return res.text(); // Processes body
    })
    .then(lookAtBody);

// Helper functions
function lookAtResponse(response) {
    console.log(`\nResponse from ${response.url}`);
    console.log(`Status: ${response.status}`);
    console.log(`Status code: ${response.statusText}`);
    console.log("Headers:");
    for (let key of response.headers.keys()) {
        console.log(`${key}:  ${response.headers.get(key)}`);
    }
}

function lookAtBody(body) {
    console.log("Body information")
    console.log('body size:', body.length);
}

Example Header Info Output

Response from http://www.grotto-networking.com/
Status: 200
Status code: OK
Headers:
connection:  close
content-encoding:  gzip
content-type:  text/html
date:  Thu, 08 Oct 2020 20:34:38 GMT
etag:  W/"27f8-5af34991971fc"
last-modified:  Sun, 13 Sep 2020 16:42:05 GMT
server:  nginx
transfer-encoding:  chunked
vary:  Accept-Encoding
Body information
body size: 10232

Using Promises for Control of Multiple Requests

Common Scenarios

Serial requests
Parallel requests
First return wins

Example 3 in order

threeInOrder.mjs

let site1 = {
  url: "https://www.grotto-networking.com",
  options: {method: "HEAD"}
};

let site2 = {
  url: "http://www.google.com",
  options: {method: "HEAD"}
};

let site3 = {
  url: "https://kitewest.com.au/",
  options: {method: "HEAD"}
};

let start = new Date();
fetch(site1.url, site1.options)
  .then(res => {
    // console.log(`Grotto status: ${JSON.stringify(res)}`);
    let time = (new Date() - start) / 1000;
    console.log(`Grotto status: ${res.statusText}, time: ${time}`);
    return fetch(site2.url, site2.options);
  })
  .then(res => {
    let time = (new Date() - start) / 1000;
    console.log(`Google status: ${res.statusText}, time: ${time}`);
    return fetch(site3.url, site3.options);
  })
  .then(res => {
    let time = (new Date() - start) / 1000;
    console.log(`Aus kiteboarding status: ${res.statusText}, time: ${time}`);
  });
console.log("Starting my web requests:");

Making Parallel Requests

Suppose we don’t care about the order in which the information comes back we just need to know when all our requests have been satisfied?

Example Node.js/Requests

threeInParallel.mjs

import fetch from 'node-fetch';
let site1 = {
  url: "https://www.grotto-networking.com",
  options: {method: "HEAD"}
};

let site2 = {
  url: "http://www.google.com",
  options: {method: "HEAD"}
};

let site3 = {
  url: "https://kitewest.com.au/",
  options: {method: "HEAD"}
};

let start = new Date();
let p1 = fetch(site1.url, site1.options).then(res => {
  // console.log(`Grotto status: ${JSON.stringify(res)}`);
  let time = (new Date() - start) / 1000;
  return console.log(`Grotto status: ${res.statusText}, time: ${time}`);
});

let p2 = fetch(site2.url, site2.options).then(res => {
  let time = (new Date() - start) / 1000;
  return console.log(`Google status: ${res.statusText}, time: ${time}`);
});

let p3 = fetch(site3.url, site3.options).then(res => {
  let time = (new Date() - start) / 1000;
  return console.log(
    `Aus kiteboarding status: ${res.statusText}, time: ${time}`
  );
});

console.log("Starting my web requests:");
Promise.all([p1, p2, p3]).then(x => {
  let time = (new Date() - start) / 1000;
  console.log(`All Finished, total time: ${time}`);
});

First Answer

Suppose we make a set of parallel requests but only want to wait till we get the first answer?

Racing Requests

threeInRace.mjs

import fetch from 'node-fetch';
let site1 = {
  url: "https://www.grotto-networking.com",
  options: {method: "HEAD"}
};

let site2 = {
  url: "http://www.google.com",
  options: {method: "HEAD"}
};

let site3 = {
  url: "https://kitewest.com.au/",
  options: {method: "HEAD"}
};

let start = new Date();
let p1 = fetch(site1.url, site1.options).then(res => {
  // console.log(`Grotto status: ${JSON.stringify(res)}`);
  let time = (new Date() - start) / 1000;
  return console.log(`Grotto status: ${res.statusText}, time: ${time}`);
});

let p2 = fetch(site2.url, site2.options).then(res => {
  let time = (new Date() - start) / 1000;
  return console.log(`Google status: ${res.statusText}, time: ${time}`);
});

let p3 = fetch(site3.url, site3.options).then(res => {
  let time = (new Date() - start) / 1000;
  return console.log(
    `Aus kiteboarding status: ${res.statusText}, time: ${time}`
  );
});

console.log("Starting my web requests:");
Promise.race([p1, p2, p3]).then(x => {
  console.log("was the winner! \n The rest: ");
});

Web requests in order `async/await`

From threeInOrderAsync.mjs in NodeRequests.zip

import fetch from 'node-fetch';
let site1 = {
  url: "https://www.grotto-networking.com",
  options: {method: "HEAD"}
};

let site2 = {
  url: "http://www.google.com",
  options: {method: "HEAD"}
};

let site3 = {
  url: "https://kitewest.com.au/",
  options: {method: "HEAD"}
};

let start = new Date();
async function inOrder() {
    let res = await fetch(site1.url, site1.options);
    let time = (new Date() - start)/1000;
    console.log(`Grotto status: ${res.statusText}, time: ${time}`);
    res = await fetch(site2.url, site2.options);
    time = (new Date() - start)/1000;
    console.log(`Google status: ${res.statusText}, time: ${time}`);
    res = await fetch(site3.url, site3.options);
    time = (new Date() - start)/1000;
    console.log(`Aus kiteboarding status: ${res.statusText}, time: ${time}`);
}
console.log("Starting my web requests:");
inOrder();

Cookies and Such

Why Care About Cookies

Testing proper cookie use
Testing interfaces that require a “login”

Documentation

node-fetch cookies Gives access to Set-Cookie header.
For help “parsing” cookies see NPM Cookie

Inspecting Cookies

cookieRequest.mjs

import fetch from 'node-fetch';
//let url = "http://localhost:3000";
//let url = "https://facebook.com";
let url = "https://google.com";
fetch(url)
    .then(function (res) { // Work with response
        console.log(`\nResponse from ${res.url}`);
        console.log(`Status: ${res.status}`);
        console.log(`Status code: ${res.statusText}`);
        console.log(res.headers.raw()['set-cookie']);
    });

Example Cookie Parsing

Parse the cookies into values and properties cookieParse.mjs

import fetch from 'node-fetch';
import cookie from 'cookie';

//let url = "http://localhost:3000";
//let url = "https://facebook.com";
let url = "https://google.com";
let myPromise = fetch(url)
    .then(function (res) { // Work with response
        console.log(`\nResponse from ${res.url}`);
        console.log(`Status: ${res.status}`);
        console.log(`Status code: ${res.statusText}`);
        res.headers.raw()['set-cookie'].forEach(function(ck){
            console.log(cookie.parse(ck)); // Process cookie string here
        });
    });

Example Output

$ node cookieParse.js

Response from https://www.google.com/
Status: 200
Status code: OK
{
  '1P_JAR': '2020-10-08-21',
  expires: 'Sat, 07-Nov-2020 21:52:51 GMT',
  path: '/',
  domain: '.google.com'
}
{
  NID: '204=S69NNakjJf7u5FcOIcA7RfuNaOeNeuHiO8f_BGKKpzov_XCIIghyy6VGiAykX7Je-A5P
fNVenQ27YIBgaBvVeJfyF4W2cPEU_oDL5m4h9MCWIVNnHYjUMzAJAZ5BsNNe-i53obL_4gnTNmyAmwTT
hjmI7FNSitn42RxnNW-Ant4',
  expires: 'Fri, 09-Apr-2021 21:52:51 GMT',
  path: '/',
  domain: '.google.com'
}

HTTP Requests in Code

HTTP Requests

Learning Objectives

Node.js and Browser APIs

Applications of non-browser HTTP requests

Node.js HTTP Request Libraries

Node-Fetch Library

Documentation

API

Options

Request Example HTML Page 1

Getting the Body

Request Example HTML Body

Example: output

Bad Promise Technique

Getting JSON Data

All the Header Info

Example Header Info Output

Using Promises for Control of Multiple Requests

Common Scenarios

Example 3 in order

Making Parallel Requests

Example Node.js/Requests

First Answer

Racing Requests

Web requests in order async/await

Cookies and Such

Why Care About Cookies

Documentation

Inspecting Cookies

Example Cookie Parsing

Example Output

Web requests in order `async/await`