How to scrape a page that requires scrolling to a specific element
If the page you're scraping returns the element you want but it comes back empty, you're probably reading it before the scroll that fills it has happened. The element sits below the fold, its rows or images are placeholders until something scrolls them near the viewport, and your script grabs the value too early and gets a blank string back. This is how lazy-load works on a page that hangs its content off scroll position.
The solution is to move the headless Chrome viewport to the element so the page fires the same scroll and intersection events a person would, then read the text once the lazy content has arrived. We'll build a small script that launches headless Chrome and opens the page, waits for the target element to exist in the DOM even though it starts below the fold, scrolls that exact element into the viewport so the page runs its own lazy-load, and only then reads the now-filled-in text. It takes about 35 lines of Node.js with one library, Puppeteer.
The complete script
// scroll-to-element.mjs
import puppeteer from 'puppeteer'
const url = 'https://news.ycombinator.com/'
const targetSelector = 'a.morelink' // the "More" link at the very bottom of the list
const browser = await puppeteer.launch({ headless: true })
const page = await browser.newPage()
await page.goto(url, { waitUntil: 'networkidle2' })
// wait for the element to exist in the DOM, even though it sits below the fold.
await page.waitForSelector(targetSelector, { timeout: 15000 })
// scroll the exact element into view from inside the page, then settle the layout.
await page.evaluate((selector) => {
const el = document.querySelector(selector)
el.scrollIntoView({ block: 'center', behavior: 'instant' })
}, targetSelector)
// give lazy content triggered by the scroll a moment to fetch and render.
await new Promise((resolve) => setTimeout(resolve, 1000))
// read the element now that it is in view and any lazy content has loaded.
const text = await page.$eval(targetSelector, (el) => el.textContent.trim())
console.log(text)
await browser.close()npm install puppeteer
node scroll-to-element.mjsHow it works
Launch headless and open the page. puppeteer.launch({ headless: true }) starts a headless Chrome instance, so client-side JavaScript runs and scroll events fire the way they do for a user. waitUntil: 'networkidle2' returns once the page has had two or fewer in-flight requests for 500ms, which clears the initial load before you start scrolling.
Wait for the element to attach. waitForSelector(targetSelector) polls the DOM until the node exists or the timeout trips. It does not require the element to be visible, only present, which is exactly right when the element is rendered below the fold from the start. Set an explicit timeout so a missing element fails fast instead of hanging on the 30-second default.
Scroll the element into view. Running el.scrollIntoView({ block: 'center' }) inside page.evaluate executes in the page's own context, so it moves Chrome's actual viewport; calling it any other way fails, since page.scrollIntoView does not exist and element.scrollIntoView() cannot run in Node where there is no DOM. Target the element by selector rather than guessing a pixel offset with window.scrollTo(0, 3000), which breaks whenever the page height shifts between runs and leaves the element off-screen so its scroll listener never fires. block: 'center' puts the element in the middle of the screen rather than flush against the top edge, which keeps it clear of sticky headers. Use behavior: 'instant' so the scroll completes before the next line runs.
Wait for lazy content, then read. The scroll is what tells the page to fetch the rows, images, or comments tied to that element, and that fetch is asynchronous. A short fixed pause covers most pages; for a precise wait, swap the setTimeout for a second waitForSelector on a child element that only appears after the load. On stepped infinite-scroll pages that load content in chunks per scroll, one scroll is not enough, so loop instead: scroll to the bottom, wait, check whether the target selector has appeared, and repeat until it does or a maximum iteration count is reached. page.$eval then runs in the page and returns the text.
Use this when
A page renders an element below the fold whose content only loads, or only becomes readable, once it scrolls near the viewport, and you need to read that specific element rather than the whole page.
Skip this when
The whole page is already present in the server HTML (use a plain fetch plus a parser); the page loads endlessly on scroll and you want every item (use a scroll-to-bottom loop for infinite scroll); the content appears behind a "Load more" button rather than on scroll (click the button instead); or the data arrives through a background XHR you can call directly (hit that endpoint and skip the browser).