Simplescraper

Multiple page and infinite scroll scraping ​

Most lists run across more than one page: a 'Next' button, a 'Load more' button, or more rows that appear as you scroll. Simplescraper finds how the next page loads and scrapes as many pages as you choose, in the extension and in the cloud.

Scrape more pages in the extension ​

  1. Choose your list and fields as usual (see Scraping and saving recipes). The 'Pages' step then asks 'Choose pages to scrape'.
  2. Click 'More pages'. Simplescraper looks for the control that loads the next page and selects it for you.
  3. Under 'How the next page loads', choose the type that matches the page:
    • Next page: a link or button opens the next page of the list.
    • Load more: a button adds more rows to this page.
    • Scroll: scrolling down loads more rows, with no button to choose.
  4. Set 'Pages to scrape' ('Load more clicks' or 'Scrolls' for the other two types) with the minus and plus buttons. The credit cost shows beside the count.
  5. Click 'Scrape 5 pages' (the button names the count you set).

Runs of up to five pages need no account and use no credits. Longer runs use 1 credit for each page after the first five, up to 1,000 pages per scrape. See Credits.

If the wrong button is selected ​

The selected button is outlined on the page, so you can see which one Simplescraper will click. It reads 'Automatically selected' when Simplescraper chose it and 'Manually selected' when you did. To choose a different one:

  • Pick another guess: hover a guess in the list to outline it on the page, then click it.
  • Select it on the page: click 'Select on page', then click the next page or load more button on the page itself. Press Esc to stop.
  • Type a selector: open 'More options' and enter it under 'Button selector'.

'More options' also sets 'Wait between pages', from 0 to 30 seconds, for sites that load slowly.

A faster option for numbered pages

If the page URLs follow a pattern such as 'page=2' or '/page/2', the cloud scrapes them in parallel, which is faster than clicking through. See Bulk scraping in the cloud.

Scrape more pages in the cloud ​

A recipe you save in the extension keeps its page settings, so its cloud runs page through the list the same way. To change them, open the recipe in the dashboard, edit it and use the 'Page navigation' section.

How infinite scroll runs ​

  • The number you set is a maximum. Each run loads the page, then scrolls down up to that many times. A recipe set to scroll 10 times loads the page and scrolls at most 10 times.
  • Scrolling stops on its own when the page stops loading. With the default settings, once two scrolls in a row reach the bottom and nothing new appears, the recipe stops and extracts, so a list that ends early finishes early.
  • Each scroll counts as a page. The first page load counts too, so a recipe set to scroll 10 times uses at most 11 pages. A run that stops early uses only the pages it made.
  • You never pick what to scroll. Simplescraper starts from the list you selected and scrolls whatever holds it: the whole page, or a panel with its own scroll bar, such as the results beside a map.

Change the scroll settings of a saved recipe ​

  1. Open the recipe in the dashboard and edit it.
  2. Under 'Page navigation', set 'Page type' to 'Scroll'.
  3. Enter 'Number of times to scroll down'. The most you can enter is 100.
  4. Open 'More scroll options' if the page needs them:
    • How far each scroll goes: 'End of page' scrolls to the end of the list and waits for more items, which suits most pages. 'One screen' moves one screen at a time, for lists that only show the items currently on screen.
    • Pause between scroll steps: the wait in seconds between steps down the page, from 0 to 5. Give slow sites longer.
    • When to collect items: 'When scrolling ends' collects every item once at the end. 'After each scroll' collects after each scroll, for feeds that remove posts once you scroll past them, such as X timelines. The same item can then appear more than once.
  5. Save the recipe.

A page that does not load more items as you scroll can still be scrolled once before extracting: with 'Page type' set to 'Single page' or 'Multiple pages', set 'Scroll down the page before extracting' to 'On'. Each page is then scrolled to the end once before extracting, for content that only appears once scrolled into view, lazy-loaded images for example.

For questions or help, reach out via chat.

Was this guide useful?