How to wait for an element to load before scraping
If you're scraping a page whose content arrives over JavaScript, you have probably read the DOM a moment too early and pulled back an empty string, a null, or a loading spinner instead of the value you wanted. The element is in the markup you see in DevTools, but it is not there yet when your code looks, because the browser is still fetching and rendering it after the initial HTML loads.
The fix is to pause your scraper until the element actually exists in the live DOM, rather than guessing with a fixed sleep. We'll build a small script that opens the client-rendered page in headless Chrome, waits for the target element to actually appear before reading anything so the value is there when we look, reads the matched text in one pass, and catches a timeout so a missing element fails cleanly instead of crashing the run. It comes out to about 30 lines of Node.js and one open-source library, Puppeteer.
The complete script
// wait-for-element.mjs
import puppeteer from 'puppeteer'
const url = 'https://www.scrapethissite.com/pages/ajax-javascript/'
const selector = '.film-title'
const browser = await puppeteer.launch({ headless: true })
const page = await browser.newPage()
/* networkidle2 lets the first XHR batch settle, but it is not a guarantee the
element you want has rendered, so still wait for the selector explicitly. */
await page.goto(url, { waitUntil: 'networkidle2' })
try {
/* poll the live DOM until a node matching the selector is present AND visible.
visible: true rejects an element that exists but is display:none or zero-size.
the default timeout is 30s; 15s is enough for most renders and fails faster. */
await page.waitForSelector(selector, { visible: true, timeout: 15000 })
/* the element is now in the DOM. read every match in one pass. */
const titles = await page.$$eval(selector, els =>
els.map(el => el.textContent.trim())
)
console.log(`Found ${titles.length} titles`)
console.log(titles.slice(0, 5))
} catch (err) {
/* waitForSelector throws a TimeoutError if the node never appears. treat that
as "the content did not load", not as a crash, so the run reports cleanly. */
console.error(`Element "${selector}" did not appear in time: ${err.message}`)
} finally {
await browser.close()
}npm install puppeteer
node wait-for-element.mjsHow it works
Open the page with networkidle2. A page that builds itself from XHR has nothing useful in its first HTML response. Waiting for the network to go quiet means the page has had a chance to fire its data requests before the script moves on, so waitForSelector is not starting against a bare shell. It is a head start, not a finish line, which is why the explicit selector wait still follows.
Wait for the selector, with visible: true. page.waitForSelector(selector, { visible: true }) polls the page on each animation frame and after each DOM mutation, resolving the instant a matching node is present and rendered. Passing visible: true makes the wait reject a node that is in the DOM but display:none or zero-size, which is what catches spinners and pre-render placeholders. The timeout is in milliseconds and defaults to 30000; lowering it to 15000 fails a dead page sooner. If the element lives inside an <iframe>, this wait against the top document times out even though it is on screen, so get the frame with page.frames().find(f => f.url().includes('embed')) and call waitForSelector on the frame instead, as in How to scrape an iframe's contents in Puppeteer.
Read the element once the wait resolves. With the node guaranteed present, page.$$eval(selector, ...) runs a function over every match inside the page context and returns plain data. Reading text only after the wait resolves is the whole point: the value is there now, so textContent returns the rendered string rather than an empty node. When a node renders empty first and its value streams in a tick later, visible: true is satisfied while the text is still blank, so wait on the content rather than the node with page.waitForFunction(sel => document.querySelector(sel)?.textContent.trim().length > 0, {}, selector).
Handle the timeout instead of letting it throw. When the element never appears, waitForSelector rejects with a TimeoutError. Wrapping the wait in try/catch turns that into a logged message naming the selector, and the finally closes the browser on both the success and the failure path so a timeout does not leak a Chrome process. A wait can also run to its full timeout because the site served a challenge page or an empty stub to a headless client rather than because of timing, so log page.url() and a snippet of page.content() on timeout to confirm the real page rendered, and if it is a block, address detection first, as in How to patch headless Chrome to avoid detection.
Use this when
The value you need is rendered by client-side JavaScript after the initial HTML, so a one-shot read of the DOM returns nothing, and you want the scrape to start the instant the element is ready rather than after a guessed delay.
Skip this when
The element is already in the server's HTML response (a plain fetch and a parser like cheerio is lighter than a browser); the content loads only as you scroll it into view (use a lazy-load or infinite-scroll loop); the data arrives through an XHR you can read directly (intercept that response instead of waiting on the rendered node); or the element appears only after a click or form submit (drive the interaction first, then wait).