Scrape the same page every day. Keep only what's new.
Most scrapers hand you a file after every run. Simplescraper keeps one dataset in your account that adds new rows, skips the ones you already have and stays beyond the 30-day window.
Unique results key
Pick a property that is unique on the page, such as a link, a product ID or an address. That property is the unique results key.
New rows only
Each run compares every row with the dataset. New keys are appended, known keys are skipped and existing rows are never changed.
Datasets that persist
A dataset persists and grows with every run, beyond the 30-day window that applies to the results of a single run.
Most scrapers hand you a file. We keep your data.
The dataset lives in your account, not in your downloads folder. There is no database to set up, no CSV files to merge and no repeats to clean out, and it is still there months after the first run.
- Raw HTML or a file per run
- You merge the files yourself
- Duplicates pile up
- Results disappear after a while
- Structured rows in one dataset
- New rows added on every run
- Repeats skipped on the way in
- Kept beyond the 30-day window
Pull the dataset from the API, send it to Google Sheets or Airtable, or let an AI assistant read it over MCP.
When to use incremental scraping
Separate runs are the default and suit snapshots you compare over time. Turn on incremental scraping when a page adds items and you want one complete list.
| Separate runs | One dataset | |
|---|---|---|
| Each run produces | A separate result set | New rows appended to one dataset |
| Kept for | 30 days | Persists and grows outside the 30-day window |
| Rows already seen | Saved again in every run | Skipped by the unique results key |
| Best for | Snapshots you compare, such as weekly price checks | Job boards, news pages and directories that add items over time |
| Changing your mind | Turn on incremental scraping at any time. Earlier snapshots are not imported. | Turn it off at any time. Everything collected stays and new runs save separate results again. |
How to turn on incremental scraping
Incremental scraping is a setting on any saved scraper. It pairs naturally with a schedule.
Pick the field that identifies a row
Save a scraper with at least one property that is unique per row, such as the posting's link.
Turn on incremental scraping
Open Show advanced options, find Incremental scraping and choose that property as the key.
Every run adds only new rows
Run it by hand or on a schedule. The dataset grows and the rows you already have are skipped.
FAQ
How does deduplication work?
Choose a unique field like URL or product ID as your deduplication key. Simplescraper automatically skips items you've already scraped, preventing duplicate data.
When should I use deduplication vs the crawler?
Use deduplication when repeatedly scraping pages that mix old and new content (like job boards or news feeds). Use the crawler for scraping many different URLs at once. Deduplication builds a permanent database over time; crawler is for one-time batch operations.
How long is my data kept?
Results from each run are kept for 30 days. Datasets built by incremental scraping persist and grow. Anything sent to Sheets, Airtable or a webhook is yours to keep.
Build a dataset without duplicates today.
Turn on incremental scraping for any scraper and start with 100 free credits.
Start free