Incremental scraping: one dataset that grows with every run
In cases where you need to scrape the same page repeatedly - for example job boards, price lists, news feeds, directories - the most useful format is one continuously updated, non-duplicated list of data rows rather than individual distinct snapshots. This is possible with incremental scraping.
When to use it
Use incremental scraping when a recipe runs repeatedly against the same page and the page adds content over time:
- Job boards, where each run should capture only new postings
- News or announcement pages, where new items appear alongside old ones
- Directories or lists you're assembling into one complete dataset
When separate runs are better
Incremental scraping isn't required when each run should stand alone. The default separate-runs setting is better for snapshots you compare over time, such as weekly price checks. Bulk scraping doesn't require it either because each URL is scraped independently.
How it works
Incremental scraping keys every row on one property: the unique results key. Choose a value that identifies each row and isn't repeated elsewhere on the page - a URL, a product ID, an address.
On each run, Simplescraper compares every scraped row against the existing dataset. Rows with new keys are appended; rows with known keys are skipped. Nothing is saved twice, and existing rows are never modified.
Setting a property as the unique key.
Set it up
- Create a recipe with at least one property that holds unique values - for a job board, the posting's link.
- When saving or editing the recipe, click "Show advanced options" and find the Incremental scraping section (previously called "Unique results key").
- Select the key property from the dropdown and save.
Every run from this point appends to the recipe's dataset instead of creating separate results.
Data retention
| Standard recipes | Incremental recipes | |
|---|---|---|
| Each run produces | A separate result set | New rows appended to one dataset |
| Retention | Results available for 30 days | The dataset persists and grows |
- Turning the key off keeps everything accumulated - changing a setting never deletes data. New runs return to saving separate results.
- Turning it back on continues the same dataset where it left off.
- Enabling incremental scraping on an existing recipe doesn't import past snapshots; the dataset fills from the source page as runs continue.
Related
- Data retention - the full picture of how long each kind of data is stored
- Scheduling your webscraping - incremental scraping pairs naturally with a schedule
- Bulk scraping in the cloud - each URL is scraped independently, so no unique key is needed
For questions or help, reach out via chat.