JavaScript async generators

JavaScript logo Did you know that you can combine await with generators in a JavaScript for loop? The for await … of construction is very handy in this ES6-Promise-everywhere-JavaScript world. In this post, we'll go through a simple example consuming data from a paginated API like GitHub.

Coding

Say you want to consume data from a JSON REST API. As soon as there is more than a couple of objects to retrieve, the API will probably have some pagination mechanism. The thing is you want to abstract this at the fetcher level, processing objects one by one.

So, let's start with the processing loop. Basically, we wish to write it like this: 

  // generator search function
  function *search() {
    const results = request();
    yield* results;
  }

  // loop through the results
  for (const result of search()) {
    // do something with result.
  }

But request() is actually async, that's where the for await … of magic happen: 

  // async generator function
  async function *search() {
    const results = await request();
    yield* results;
  }

  // loop through the results
  for await (const result of search()) {
    // do something with result.
  }

Here is a working example using GitHub's API. It is a bit more complex because the search() function has a while looping through the result pages.

index.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

/*
 * A simple NodeJS example using async generators.
 */
'use strict';

const debug   = require('debug')('search');
const parse   = require('parse-link-header');
const request = require('request-promise-native');


// GitHub code search result generator.
async function *search(q) {
    const get = request.defaults({
        method: 'GET',
        json: true, // parse the result as JSON
        resolveWithFullResponse: true,
        headers: {
            // see https://developer.github.com/v3/#user-agent-required
            'User-Agent': 'js async generator demo',
        },
    });

    // setup the first URL to be requested. The followings requests will use
    // the rel="next" URL found in the response Link HTTP header, see
    // https://developer.github.com/v3/guides/traversing-with-pagination/
    let url = new URL('https://api.github.com/search/code');
    url.searchParams.set('q', q);

    // Main loop going through the paginated results.
    while (url) {
        debug(`requesting ${url}`);
        const rsp = await get({ url });
        if (rsp.statusCode !== 200 /* OK */) {
            throw new Error(`expected 200 OK but got ${rsp.statusCode}`);
        }

        debug(`yielding ${rsp.body.items.length} items`);
        yield* rsp.body.items;

        const link = parse(rsp.headers.link);
        if (link && link.next) {
            // Setup the next page URL for the next loop iteration.
            url = new URL(link.next.url);
        } else {
            // We've reached the last page, exit the loop.
            url = null;
        }
    }
}


// await may only be used from inside an async function.
async function main() {
    // You may change the search query, but beware of GitHub search rate limit
    // https://developer.github.com/v3/search/#rate-limit
    for await (const { score, name, repository } of search('OpenBSD+user:kAworu')) {
        console.log(`→ ${score.toFixed(2)}: ${name} (in ${repository.full_name})`);
    }
}


main();

Running

I made a simple project to run the code above: 

git clone https://github.com/kAworu/js-async-generator.git
cd js-async-generator.git
npm install

Now let's run the script with DEBUG=* node index.js so that we can see when search() actually perform the network requests: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

  search requesting https://api.github.com/search/code?q=OpenBSD%2Buser%3AkAworu +0ms
  search yielding 30 items +749ms
→ 15.32: openbsd-daily-on-linux.erb (in kAworu/kaworu.ch)
→ 13.97: daily (in kAworu/kaworu.ch)
→ 13.42: weekly (in kAworu/kaworu.ch)
→ 13.00: programming.erb (in kAworu/kaworu.ch)
→ 12.22: explicit_bzero.c (in kAworu/cryptopals)
→ 12.22: freezero.c (in kAworu/cryptopals)
→ 12.22: freezero.h (in kAworu/cryptopals)
→ 12.22: reallocarray.h (in kAworu/cryptopals)
→ 12.15: explicit_bzero.h (in kAworu/cryptopals)
→ 12.15: strlcat.h (in kAworu/cryptopals)
→ 12.15: strlcpy.h (in kAworu/cryptopals)
→ 12.15: timingsafe_bcmp.h (in kAworu/cryptopals)
→ 12.15: recallocarray.h (in kAworu/cryptopals)
→ 12.08: strlcat.h (in kAworu/tagutil)
→ 12.08: strlcpy.h (in kAworu/tagutil)
→ 11.48: monthly (in kAworu/kaworu.ch)
→ 11.43: timingsafe_bcmp.c (in kAworu/cryptopals)
→ 10.94: strlcat.c (in kAworu/cryptopals)
→ 10.94: strlcpy.c (in kAworu/cryptopals)
→ 10.94: reallocarray.c (in kAworu/cryptopals)
→ 10.83: strlcat.c (in kAworu/tagutil)
→ 10.83: strlcpy.c (in kAworu/tagutil)
→ 10.64: index.js (in kAworu/js-async-generator)
→ 10.60: t_toolkit.c (in kAworu/tagutil)
→ 10.56: recallocarray.c (in kAworu/cryptopals)
→ 10.44: strdup.c (in kAworu/tagutil)
→ 8.99: .zshrc (in kAworu/dotfiles)
→ 7.97: vimrc (in kAworu/dotfiles)
→ 7.93: perl.h (in kAworu/perl1)
→ 7.19: kaworu (in kAworu/dotfiles)
  search requesting https://api.github.com/search/code?q=OpenBSD%2Buser%3AkAworu&page=2 +4ms
  search yielding 2 items +427ms
→ 6.35: break_mac.c (in kAworu/cryptopals)
→ 5.23: srp.c (in kAworu/cryptopals)

As we can see, we awaited on two network requests: the first one got 30 entries and the second one got 2. The nice thing is that our processing code (i.e. the main() function) is completely unaware of this implementation detail.

That's it! I'll be happy to hear from you if this trick is useful, or more for await … of examples. Cheers.