Incremental scraping: one dataset that grows with every run
In cases where you need to scrape the same page repeatedly - for example job boards, price lists, news feeds, directories - the most useful format is one continuously updated, non-duplicated list of data rows rather than individual distinct snapshots. This is possible with incremental scraping.
When to use it
Use incremental scraping when a recipe runs repeatedly against the same page and the page adds content over time:
- Job boards, where each run should capture only new postings
- News or announcement pages, where new items appear alongside old ones
- Directories or lists you're assembling into one complete dataset
Don't use it when each run should stand alone - a full price snapshot you compare week to week, for example. Standard runs are the right tool there. The crawler doesn't need it either: it visits each URL once per run.
How it works
Incremental scraping keys every row on one property: the unique results key. Choose a value that identifies each row and isn't repeated elsewhere on the page - a URL, a product ID, an address.
On each run, Simplescraper compares every scraped row against the existing dataset. Rows with new keys are appended; rows with known keys are skipped. Nothing is saved twice, and existing rows are never modified.
Setting a property as the unique key.
Set it up
- Create a recipe with at least one property that holds unique values - for a job board, the posting's link.
- When saving or editing the recipe, click "Show advanced options" and find the Incremental scraping section (previously called "Unique results key").
- Select the key property from the dropdown and save.
Every run from this point appends to the recipe's dataset instead of creating separate results.
Data retention
| Standard recipes | Incremental recipes | |
|---|---|---|
| Each run produces | A separate result set | New rows appended to one dataset |
| Retention | Results available for 30 days | The dataset persists and grows |
- Turning the key off keeps everything accumulated - changing a setting never deletes data. New runs return to saving separate results.
- Turning it back on continues the same dataset where it left off.
- Enabling incremental scraping on an existing recipe doesn't import past snapshots; the dataset fills from the source page as runs continue.
Related
- Data retention - the full picture of how long each kind of data is stored
- Scheduling your webscraping - incremental scraping pairs naturally with a schedule
- Bulk scraping in the cloud - the crawler visits each URL once per run, no key needed
For questions or help, reach out via chat.