How to wait for an element to load before scraping
If a scraper reads an empty value from a JavaScript-rendered page, wait for the content you need. Puppeteer's waitForSelector can wait for an element to become visible, but a visible element can still contain no text. Use waitForFunction when the text itself determines whether the page is ready.
The practice page below loads film titles only after you choose a year. This script selects 2015, waits for nonempty titles and prints them. You need Node.js 22 or newer and Puppeteer.
The complete script
/* wait-for-element.mjs */
import puppeteer from 'puppeteer'
const url = 'https://www.scrapethissite.com/pages/ajax-javascript/'
const yearSelector = 'a.year-link[id="2015"]'
const selector = '.film-title'
const timeout = 15000
const browser = await puppeteer.launch({ headless: true })
let stage = 'opening the page'
try {
const page = await browser.newPage()
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 })
/* this page fetches its film data only after a year is clicked. */
stage = 'selecting a year'
await page.waitForSelector(yearSelector, { visible: true, timeout })
await page.click(yearSelector)
stage = 'waiting for film titles'
await page.waitForSelector(selector, { visible: true, timeout })
/* wait until the current matches contain text, then return that snapshot. */
const ready = await page.waitForFunction(
sel => {
const titles = Array.from(document.querySelectorAll(sel), el =>
el.textContent.trim()
)
return titles.length > 0 && titles.every(title => title.length > 0)
? titles
: false
},
{ timeout },
selector
)
const titles = await ready.jsonValue()
await ready.dispose()
console.log(`Found ${titles.length} titles`)
console.log(titles.slice(0, 5))
} catch (error) {
console.error(`${error.name} while ${stage}: ${error.message}`)
process.exitCode = 1
} finally {
await browser.close()
}npm install puppeteer
node wait-for-element.mjsHow it works
Trigger the request that loads the data. domcontentloaded lets the script proceed once the initial document has been parsed. On this page, film data arrives after the year link is clicked. Waiting longer without making that click does not start the request. For another site, identify whether navigation, a click or scrolling triggers the content you need.
Check visibility and text separately. waitForSelector waits for a matching element; visible: true adds a visibility requirement. It does not distinguish a populated result from a visible empty container or placeholder. The subsequent waitForFunction checks that there is at least one title and that the current matches all contain nonempty text. For another page, change the predicate to match its ready state; nonempty text could still be a loading message.
Read the snapshot that satisfied the condition. The predicate returns the title array when it is ready, and jsonValue() transfers that array to Node.js. The script prints the number of titles and the first five. This captures the populated rows present at that moment. If a different page adds rows in several batches, wait for its completion indicator or a known expected count before claiming to have the full result.
Report failures and close the browser. Each selector or text wait has its own 15-second limit; navigation has a separate 30-second limit. These are bounds chosen for this example, not a guarantee that another site will finish in that time. The error message preserves the error type and names the stage that failed, while process.exitCode = 1 tells the calling process that the scrape failed. Page creation, navigation and extraction are all inside try/finally, so they reach browser cleanup on success or a caught error. On a timeout, inspect the loaded page and selector before increasing the limit: a missing click, wrong selector or challenge page also prevents the expected result.
Use this when
The value appears after navigation or an interaction, and you can identify a selector and a content condition that make it ready to read.
Skip this when
If the value is in the initial HTML, use fetch and an HTML parser. If an accessible JSON response supplies the data, read that response directly. For content inside an iframe, wait in the frame rather than the top document. Infinite scrolling needs a loading loop and a stopping condition in addition to these waits.