How to scrape concurrently with a promise pool in Node.js

Updated 2026-06-25 · 6 min read

If you have a few thousand URLs to scrape, you have probably tried Promise.all(urls.map(fetch)) and watched it either get you rate limited within the first few hundred requests, or exhaust the machine's sockets and start throwing ECONNRESET. Firing every request at once is the obvious first move, and it is the one that breaks once the list is more than a handful of URLs, because nothing is holding the number of open connections down.

The mechanism is to keep a fixed number of fetches in flight at a time: hold a Set of N running promises, and each time one resolves, remove it from the set and start the next URL, so the open-socket count never climbs above N. It takes about 50 lines of plain Node.js, with nothing to install.

Key terms

• Concurrency. The number of requests in flight at the same instant, which is the lever a pool controls, separate from the total number of URLs.
• Promise pool. A loop that holds a bounded set of in-flight promises and starts the next task only when a running one settles, so the concurrency stays at a fixed ceiling.
• Promise.race. A method that resolves as soon as the first promise in an iterable settles, which is how the pool learns a slot has freed up without waiting for the whole batch.
• Settled. A promise that has either fulfilled or rejected, as opposed to still pending. The pool tracks settled-ness to know when to top up.

Here is what the script does:

• Walk the URL list with a cursor instead of mapping it all at once, so work starts only when a slot is free.
• Keep a Set of in-flight promises and top it up to the concurrency limit each time the loop comes around.
• Wait on Promise.race over the running set, so the loop wakes the moment any one fetch settles and can start the next URL.
• Capture each result or error against its input index, so one failing URL does not reject the whole run and the output stays in input order.
• Print a count of successes and failures at the end.

The complete script

// promise-pool.mjs

/* Run `worker` over every item in `items`, keeping at most `concurrency`
   calls in flight at once. Results come back in input order. A worker that
   throws is recorded as { ok: false, error } rather than rejecting the run,
   so one bad URL does not abort the batch. */
async function promisePool(items, concurrency, worker) {
  const results = new Array(items.length)
  const running = new Set()
  let cursor = 0

  while (cursor < items.length || running.size > 0) {
    /* Top the set up to the concurrency ceiling. Each task captures its own
       index so the result lands in the right slot, and removes itself from
       the running set as it settles so the next pass can refill that slot. */
    while (cursor < items.length && running.size < concurrency) {
      const index = cursor++
      const task = (async () => {
        try {
          results[index] = { ok: true, value: await worker(items[index], index) }
        } catch (error) {
          results[index] = { ok: false, error }
        }
      })()
      running.add(task)
      task.finally(() => running.delete(task))
    }

    /* Wait for the next task to settle before refilling. Without this the
       while-loop would spin. Promise.race wakes on the first settle. */
    if (running.size > 0) await Promise.race(running)
  }

  return results
}

const urls = Array.from({ length: 50 }, (_, i) => `https://httpbin.org/anything?n=${i}`)

const results = await promisePool(urls, 8, async (url) => {
  const response = await fetch(url, { headers: { 'User-Agent': 'Mozilla/5.0' } })
  if (!response.ok) throw new Error(`${response.status} for ${url}`)
  const body = await response.text()
  return { url, status: response.status, bytes: body.length }
})

const ok = results.filter(r => r.ok)
const failed = results.filter(r => !r.ok)
console.log(`Done. ${ok.length} ok, ${failed.length} failed`)
for (const r of failed) console.log(`[fail] ${r.error.message}`)

node promise-pool.mjs

What each step does

Walk the list with a cursor, not .map(). Mapping the whole array starts every request immediately, which is the failure mode this page exists to avoid. The cursor index advances only when the inner loop has room under the concurrency ceiling, so a request begins only when a slot is free.

Capture the index before the task starts. const index = cursor++ reads and increments in one step, so the async task closes over its own fixed index. Without this, every task would close over the same shared cursor and write to the wrong slot, which is the classic loop-variable-in-a-closure bug. The result lands in results[index], keeping the output in input order even though the requests finish out of order.

Add to the set, then remove on settle. running.add(task) puts the live promise in the pool, and task.finally(() => running.delete(task)) removes it the instant it settles. The Set is the live count of open requests, so running.size < concurrency is the gate that decides whether to start another.

Race the running set to refill. await Promise.race(running) resolves as soon as any one in-flight task settles, so the loop wakes, the just-settled task is already gone from the set via its finally, and the next pass tops the set back up to the ceiling. Racing the set is what makes the pool refill one slot at a time instead of waiting for the whole batch to drain.

Record errors instead of throwing. The worker is wrapped in a try/catch that writes { ok: false, error } to the slot, so a 500 or a dropped connection on one URL does not reject the pool and lose the other forty-nine results. The caller reads r.ok to split successes from failures at the end.

Gotchas

• Promise.all over the whole list opens every socket at once.

  • Issue: await Promise.all(urls.map(u => fetch(u))) starts all N requests in the same tick, so a 5,000-URL list opens 5,000 connections, which trips server rate limits and can exhaust the local file-descriptor budget with ECONNRESET or EMFILE.
  • Fix: gate the work through the pool so running.size never exceeds concurrency, which holds the open-socket count at the ceiling regardless of list length.

• A single rejected task aborts the whole batch.

  • Issue: Promise.all rejects on the first task that throws, so one URL returning a 500 discards every other result, including the ones that already succeeded.
  • Fix: wrap the worker in a try/catch that records { ok: false, error } per item, as the script does, so the pool finishes all items and you triage the failures after.

• The async task closes over the shared loop variable.

  • Issue: referencing cursor inside the task instead of a captured index means every task reads the value cursor has reached by the time it runs, so results overwrite each other and land in the wrong slots.
  • Fix: snapshot the index with const index = cursor++ before creating the task, so each task owns its own index for the life of the request.

• The while-loop spins at 100% CPU.

  • Issue: dropping the await Promise.race(running) leaves a while loop with no suspension point, so once the set is full the loop keeps re-checking the same condition and pins a core without making progress.
  • Fix: await Promise.race(running) when the set is non-empty, which parks the loop until the next task settles and a slot opens.

• Concurrency is not a rate limit.

  • Issue: a pool caps how many requests run at once, not how many per second, so a concurrency of 8 against an endpoint that answers in 50ms still sends roughly 160 requests a second and can trip a per-second quota.
  • Fix: add a token-bucket limiter inside the worker, or lower the concurrency; the rate side is covered in How to rate-limit requests with backoff in JavaScript.

• One slow URL stalls a slot, but not the pool.

  • Issue: a request to a hung endpoint holds its slot until it settles, so with no timeout one stuck URL permanently reduces the effective concurrency by one and a few of them can starve the pool.
  • Fix: give fetch an AbortSignal.timeout(ms) so a stalled request rejects and frees its slot, which the per-item try/catch then records as a failure.

Use this when

You have a known list of URLs or jobs to scrape and you want a fixed number running at once, in plain Node with nothing to install, while keeping the results in input order and the failures isolated per item.

Skip this when

You want the same pattern without writing it yourself, where the canonical p-limit wraps each call in a limiter and p-map maps an iterable at a concurrency cap; you need requests-per-second rather than a concurrency cap, where a token-bucket rate limiter is the right tool; the work must survive a process restart, where a Redis-backed queue like BullMQ persists the jobs; or the bottleneck is CPU-bound parsing rather than network waiting, where a worker-thread pool like piscina moves the work off the main loop.

Related guides

• How to rate-limit requests with backoff in JavaScript
• How to retry failed scrapes with exponential backoff
• How to scrape a sitemap and crawl every page
• How to resume a scrape after a crash
• How to deduplicate scraped records in JavaScript