Simplescraper

Multiple page and infinite scroll scraping ​

Scrape through paginated results or infinite scroll feeds automatically.

You can scrape multiple pages and infinite scroll pages with Simplescraper. Before we dig into how, two important things to note:

  1. The crawler (bulk scraping) may be a better option: if the URLs of the pages that you want to scrape follow a structured pattern like 'page=1' or 'page/2' etc, it's faster to scrape concurrently via the crawler. See Crawling lists of URLs for instructions
  2. Pagination and infinite scroll should not be selected at the same time. If the page loads more content as you scroll down without any clicking required, use infinite scroll only. If a click of any kind is required, use pagination only

Pagination ​

To extract data across multiple pages, you must teach Simplescraper how to navigate:

  • Identify the element used to navigate to the next page (typically a 'next' button or an arrow), then click the pagination icon in the top menu to activate detection and then click on the navigation element. If done successfully you will see that the pagination icon has turned green. The css selector of the navigation element will also be visible when you hover over the pagination icon.
  • On some pages, selecting the navigation element will unintentionally load the next page. In this case press the SHIFT key when hovered over the navigation element instead of clicking to select it
  • When saving the recipe you will be presented with an option asking how many pages you wish to navigate.

Infinite scroll ​

  • If the page that you intend to scrape loads new content automatically as you scroll down and you wish to scrape this data, click the infinite scroll icon located in the extension's top menu
  • The icon will turn green to indicate that it's active
  • When saving the recipe you will be presented with an option asking how many times you wish the automated recipe to scroll

How infinite scroll runs ​

  • The number you set is a maximum. Each run loads the page, then scrolls down up to that many times. A recipe set to scroll 10 times loads the page and scrolls at most 10 times.
  • Scrolling stops on its own when the page stops loading. With the default settings, once two scrolls in a row reach the bottom and nothing new appears, the recipe stops and extracts, so a list that ends early finishes early.
  • Each scroll counts as a page. The first page load counts too, so a recipe set to scroll 10 times uses at most 11 pages. A run that stops early uses only the pages it made.
  • You never pick what to scroll. Simplescraper starts from the list you selected and scrolls whatever holds it: the whole page, or a panel with its own scroll bar, such as the results beside a map.

Change the scroll settings of a saved recipe ​

  1. Open the recipe in the dashboard and edit it.
  2. Under 'Page navigation', set 'Page type' to 'Scroll'.
  3. Enter 'Number of times to scroll down'. The most you can enter is 100.
  4. Open 'More scroll options' if the page needs them:
    • How far each scroll goes: 'End of page' scrolls to the end of the list and waits for more items, which suits most pages. 'One screen' moves one screen at a time, for lists that only show the items currently on screen.
    • Pause between scroll steps: the wait in seconds between steps down the page, from 0 to 5. Give slow sites longer.
    • When to collect items: 'When scrolling ends' collects every item once at the end. 'After each scroll' collects after each scroll, for feeds that remove posts once you scroll past them, such as X timelines. The same item can then appear more than once.
  5. Save the recipe.

A page that does not load more items as you scroll can still be scrolled once before extracting: with 'Page type' set to 'Single page' or 'Multiple pages', set 'Scroll down the page before extracting' to 'On'. Each page is then scrolled to the end once before extracting, for content that only appears once scrolled into view, lazy-loaded images for example.

For questions or help, reach out via chat.