# Simplescraper > No-code web scraping by dashboard and free Chrome extension, with AI extraction and website change monitoring. Turn any site into structured data or an API. Connect AI tools like Claude, Cursor, and ChatGPT to Simplescraper via MCP. Manage recipes and scrape websites directly from your AI assistant. Complete Simplescraper API reference: authentication, scraping endpoints, recipe management, Smart Extract, and webhook configuration. ## For AI agents and developers - [MCP server for Claude, Cursor, ChatGPT and other AI agents](https://simplescraper.io/docs/mcp-server): endpoint https://mcp.simplescraper.io/mcp. Connect AI tools like Claude, Cursor, and ChatGPT to Simplescraper via MCP. Manage recipes and scrape websites directly from your AI assistant. - [REST API](https://simplescraper.io/docs/api-guide): base URL https://api.simplescraper.io/v1. Complete Simplescraper API reference: authentication, scraping endpoints, recipe management, Smart Extract, and webhook configuration. - [Screenshot API](https://simplescraper.io/docs/screenshot-api): Capture full-page or viewport screenshots of any URL via the Simplescraper Screenshot API. Supports PNG, JPEG, and WebP with configurable viewport and page-load options. - [Google Search API](https://simplescraper.io/docs/google-search-api): Get Google search results as structured JSON from one API call: up to 100 organic results with position, title, URL and snippet, plus the answer box, knowledge graph, People also ask and related searches. 2 credits per page of ten results. - [URL Extractor API](https://simplescraper.io/docs/extract-urls-api): Extract all URLs from any website's sitemap via the Simplescraper URL Extractor API. Discover sitemaps automatically from common paths and robots.txt. - [Extract API v0 (deprecated)](https://simplescraper.io/docs/creating-an-api): Deprecated guide for the original Simplescraper API. See the updated API guide for current endpoint documentation and authentication details. - [Smart Extract](https://simplescraper.io/docs/smart-data-extract): Tell Simplescraper's AI what data you want and it finds and extracts it automatically. Returns reusable CSS selectors for reliable, repeated scraping. - [Enhance scraped data using AI](https://simplescraper.io/docs/enhance-scraped-data-using-ai): Turn scraped data into the information you need. Simplescraper AI Enhance summarizes text, pulls out details like emails, answers yes or no questions and sorts rows into categories. - [Extracting websites to Markdown](https://simplescraper.io/docs/extract-markdown): Convert any webpage or entire website into clean Markdown format with Simplescraper. Ideal for feeding web content into AI models like ChatGPT and Claude. ## Getting started - [Pricing](https://simplescraper.io/#pricing) - [Chrome extension](https://chrome.google.com/webstore/detail/lnddbhdmiciimpkbilgpklcglkdegdkg) - [All features](https://simplescraper.io/features) - [Quick Start and FAQ](https://simplescraper.io/docs): Get started with Simplescraper: extract data from any website as CSV, JSON or Markdown, and monitor web pages for changes with email and webhook alerts. - [Choosing how to scrape](https://simplescraper.io/docs/local-scraping-cloud-scraping-difference): Compare browser, cloud, and API scraping in Simplescraper. Learn which method suits your needs for speed, scale, scheduling, and automation. - [Credits](https://simplescraper.io/docs/credits): Understand how Simplescraper credits work: usage rates for browser scraping, cloud scraping and AI features. Credits renew monthly on paid plans. ## Features - [Perfectly Screenshot Any Website in Seconds](https://simplescraper.io/features/screenshot-api): Capture pixel-perfect screenshots via API or bulk-capture from the dashboard. Control viewport, format, full-page rendering, and more. - [AI-Powered Web Scraping](https://simplescraper.io/features/ai-extraction-and-enhance): Use AI to automatically extract structured data from any website and enhance it with summaries, translations, and analysis. No CSS selectors needed. - [Scrape Websites Behind a Login](https://simplescraper.io/features/scrape-behind-login): Extract data from sites requiring login with cookies or credentials. Works with LinkedIn, Facebook, Instagram, and any password-protected site. - [Bulk-scrape a list of URLs into structured data with no code](https://simplescraper.io/features/scrape-list-of-urls): Select the data you want, paste up to 10,000 URLs and get structured data for every page. Download CSV or JSON or send it anywhere with a webhook. No code. - [Website Change Monitoring](https://simplescraper.io/features/website-change-monitoring): Monitor a web page, an area or one element for changes and get an email, webhook or agent alert showing exactly what changed. - [Scheduled Web Scraping](https://simplescraper.io/features/schedule-and-monitor): Run scrapers every 30 minutes, hourly, daily or on a custom schedule in your timezone, and send every run to Google Sheets, a webhook or the API. - [Scrape Data Directly to Google Sheets](https://simplescraper.io/features/google-sheets-integration): Send scraped data automatically to Google Sheets. Real-time sync, scheduled updates, and seamless workflow integration. - [Turn Any Website Into Structured Data](https://simplescraper.io/features/extract-structured-data): Automatically detect and extract tables, lists, product info, and structured data from any website without writing selectors. - [Extract All Links from Any Website](https://simplescraper.io/features/url-extractor): Find and extract all URLs from websites. Perfect for building crawl lists, finding resources, or mapping site structures. - [Remove Duplicate Scraped Data Automatically](https://simplescraper.io/features/deduplication): Eliminate duplicate entries in your scraped data. Smart deduplication based on custom fields ensures clean datasets. - [Send Scraped Data Anywhere With Webhooks & Zapier](https://simplescraper.io/features/webhooks-integration): Send scraped data to any app using webhooks. Integrate with Zapier and Make integrations for automated workflows. - [AI Agent Automation (MCP)](https://simplescraper.io/features/ai-agent-automation-mcp-server): Connect Simplescraper to AI agents via MCP protocol. Enable Claude, Cursor, and ChatGPT to scrape live data for enhanced workflows. - [Turn Websites into Live APIs](https://simplescraper.io/features/realtime-api-endpoints): Create instant APIs from any website. Get fresh data on-demand with our powerful REST API endpoints. - [Scrape Multi-Page & Infinite Scroll Sites](https://simplescraper.io/features/pagination-and-infinite-scroll): Automatically handle pagination, load more buttons, and infinite scroll. Scrape entire catalogs and feeds effortlessly. ## Docs - [Quick Start and FAQ](https://simplescraper.io/docs): Get started with Simplescraper: extract data from any website as CSV, JSON or Markdown, and monitor web pages for changes with email and webhook alerts. - [Choosing how to scrape](https://simplescraper.io/docs/local-scraping-cloud-scraping-difference): Compare browser, cloud, and API scraping in Simplescraper. Learn which method suits your needs for speed, scale, scheduling, and automation. - [Credits](https://simplescraper.io/docs/credits): Understand how Simplescraper credits work: usage rates for browser scraping, cloud scraping and AI features. Credits renew monthly on paid plans. - [Intro to browser scraping](https://simplescraper.io/docs/browser-scraping-guide): Learn browser scraping with Simplescraper: get instant results from the Chrome extension side panel, scrape behind logins, and export data. - [Scraping and saving recipes](https://simplescraper.io/docs/how-scrape-save-recipes): Step-by-step guide to selecting data, configuring properties, and saving reusable scrape recipes with the Simplescraper Chrome extension. - [Expanding list selection](https://simplescraper.io/docs/expand-list-selection): Expand your data selection to include similar items across all sections of a page. Useful for grouped listings like job boards and product categories. - [AI Property / Column Naming](https://simplescraper.io/docs/ai-property-naming): Rename cryptic CSS-based property names to meaningful labels like job_title and price using Simplescraper's one-click AI Property Naming feature. - [Bulk scraping in the browser](https://simplescraper.io/docs/browser-batch-scraping): Scrape up to 10,000 URLs from your browser using a single recipe. Import URLs from previous scrapes or paste your own list. - [Scraping data behind a login](https://simplescraper.io/docs/scrape-behind-login): Scrape password-protected websites with Simplescraper using saved credentials or browser cookies. Supports multi-factor and SSO logins. - [Troubleshooting](https://simplescraper.io/docs/troubleshooting): Fix common Simplescraper issues: timeout errors, missing data, login failures, and incorrect results. Includes resolution steps for each scenario. - [Intro to cloud scraping](https://simplescraper.io/docs/dashboard-guide): Guide to the Simplescraper cloud dashboard: run recipes in the background, set schedules, bulk scrape URLs, and configure integrations. - [Multiple page and infinite scroll](https://simplescraper.io/docs/multiple-page-infinite-scroll): Configure Simplescraper to automatically paginate through multi-page results or scroll through infinite feed pages to capture all data. - [Bulk scraping in the cloud](https://simplescraper.io/docs/crawling-lists-of-urls): Scrape up to 10,000 URLs in parallel using Simplescraper's cloud bulk scraper. Generate URL lists, import from previous scrapes, and run in the background. - [Deep scraping URLs](https://simplescraper.io/docs/deep-scraping-urls): Use two linked recipes to scrape a list of URLs and then extract detailed data from each page. Ideal for job boards, product catalogs, and listings. - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping): Automate your scraping with Simplescraper's scheduler. Set intervals from every 2 minutes to specific days and times with timezone support. - [Incremental scraping](https://simplescraper.io/docs/incremental-scraping): Scrape incrementally: each run automatically appends what's new to one growing dataset, deduplicates against what you already have, and the data stays put. Set a unique key so Simplescraper only saves new data on each run. - [Data retention](https://simplescraper.io/docs/data-retention): How long scraped data is stored in Simplescraper: results from each run are available for 30 days, incremental datasets persist and grow, and data sent to integrations is yours to keep. - [Ready-made recipes](https://simplescraper.io/docs/ready-made-recipes): Browse Simplescraper's library of ready-made scrape recipes for Google, LinkedIn, Reddit, Instagram, Yelp, and other popular websites. - [Scrape recipe health](https://simplescraper.io/docs/recipe-health): Monitor the health of your scraping recipes with automated checks. Spot issues before they affect your data, and use Site Unblocker to get through bot protection. - [Monitor a web page for changes](https://simplescraper.io/docs/monitor-web-page-changes): Monitor any website for changes. Simplescraper checks the text or appearance of a web page on your schedule and alerts you by email, webhook or AI agent when it changes. - [Extract API v1](https://simplescraper.io/docs/api-guide): Complete Simplescraper API reference: authentication, scraping endpoints, recipe management, Smart Extract, and webhook configuration. - [Screenshot API](https://simplescraper.io/docs/screenshot-api): Capture full-page or viewport screenshots of any URL via the Simplescraper Screenshot API. Supports PNG, JPEG, and WebP with configurable viewport and page-load options. - [Google Search API](https://simplescraper.io/docs/google-search-api): Get Google search results as structured JSON from one API call: up to 100 organic results with position, title, URL and snippet, plus the answer box, knowledge graph, People also ask and related searches. 2 credits per page of ten results. - [URL Extractor API](https://simplescraper.io/docs/extract-urls-api): Extract all URLs from any website's sitemap via the Simplescraper URL Extractor API. Discover sitemaps automatically from common paths and robots.txt. - [Extract API v0 (deprecated)](https://simplescraper.io/docs/creating-an-api): Deprecated guide for the original Simplescraper API. See the updated API guide for current endpoint documentation and authentication details. - [Smart Extract](https://simplescraper.io/docs/smart-data-extract): Tell Simplescraper's AI what data you want and it finds and extracts it automatically. Returns reusable CSS selectors for reliable, repeated scraping. - [Enhance scraped data using AI](https://simplescraper.io/docs/enhance-scraped-data-using-ai): Turn scraped data into the information you need. Simplescraper AI Enhance summarizes text, pulls out details like emails, answers yes or no questions and sorts rows into categories. - [Extracting websites to Markdown](https://simplescraper.io/docs/extract-markdown): Convert any webpage or entire website into clean Markdown format with Simplescraper. Ideal for feeding web content into AI models like ChatGPT and Claude. - [MCP server](https://simplescraper.io/docs/mcp-server): Connect AI tools like Claude, Cursor, and ChatGPT to Simplescraper via MCP. Manage recipes and scrape websites directly from your AI assistant. - [Scraping data into Google Sheets](https://simplescraper.io/docs/saving-data-google-sheets): Connect Simplescraper to Google Sheets and automatically save scraped data to a spreadsheet every time a recipe runs. Append or replace modes available. - [Scraping data into Airtable](https://simplescraper.io/docs/scraping-data-into-airtable): Send scraped data directly to Airtable bases using Personal Access Tokens. Configure field mapping and auto-sync results from any Simplescraper recipe. - [Scraping data into Zapier](https://simplescraper.io/docs/scraping-data-into-zapier): Connect Simplescraper to Zapier via webhooks and send scraped data to 2,000+ apps. Step-by-step setup guide with webhook configuration. - [Scraping data into Make (Integromat)](https://simplescraper.io/docs/scraping-data-into-integromat): Send scraped website data to Make (formerly Integromat) scenarios via webhooks. Setup takes under a minute with Simplescraper's built-in integration. - [Scraping data into your email inbox](https://simplescraper.io/docs/scraping-data-into-email): Receive scraped data directly in your email inbox as a preview, CSV attachment, or both. Configure from the Send tab of any recipe. - [Sending data to a webhook](https://simplescraper.io/docs/sending-data-to-webhook): Send scraped data to any URL as a JSON webhook. Receive structured data in your own backend, automation tool, or any service that accepts HTTP POST requests. - [Adding a password](https://simplescraper.io/docs/adding-password-login): Add a password login to your Simplescraper account alongside Google authentication. Useful for sharing access or using alternative login methods. - [Your subscription](https://simplescraper.io/docs/managing-subscription): Manage your Simplescraper subscription: upgrade, downgrade, add credit topups, update payment details, or cancel from your account page. - [Delete account](https://simplescraper.io/docs/delete-account): Delete your Simplescraper account from the My account page. The action is irreversible and cancels any active subscription. ## Free tools - [Free URL Extractor — Extract All Links from Any Website Instantly](https://simplescraper.io/extracturls) - [Free Website to Markdown, JSON, CSV Converter for LLMs](https://simplescraper.io/extractmarkdown) ## Optional - [Blog](https://simplescraper.io/blog) - [Scrape templates](https://simplescraper.io/templates) - [Developer guides](https://simplescraper.io/learn) - [Privacy](https://simplescraper.io/docs/privacy) - [Terms](https://simplescraper.io/docs/terms-and-conditions) - [Full documentation as one file](https://simplescraper.io/llms-full.txt) --- # Feature pages ## Perfectly Screenshot Any Website in Seconds Capture pixel-perfect screenshots via API or bulk-capture from the dashboard. Control viewport, format, full-page rendering, and more. https://simplescraper.io/features/screenshot-api **What image formats does the Screenshot API support?** PNG, JPEG, and WebP. PNG gives lossless quality and supports transparency. JPEG is smaller with adjustable quality (1-100). WebP offers the best compression-to-quality ratio. Choose the format that fits your use case. **Can I capture screenshots without writing code?** Yes. Every scraping run saves a screenshot of each page it visits, which you can open from the run in your dashboard. To capture any address on demand, the Screenshot API takes one request with the page URL. **Can I capture full-page screenshots of long pages?** Yes. Set full_page to true and the API scrolls the entire page, including lazy-loaded content, and captures everything in a single image. Works on pages of any length. **How do I hide cookie popups and ads in screenshots?** Set hide_popups to true and the API automatically removes cookie consent banners, newsletter overlays, and common ad elements before capturing. For ads specifically, set hide_ads to true. **What viewport sizes are supported?** Any width and any height from 320px to 3,840px. Common presets include desktop (1920x1080), laptop (1280x800), tablet (768x1024), and mobile (375x812). You can also set device_scale_factor for retina captures at 1x, 1.5x, 2x, or 3x. **How long are hosted screenshot URLs available?** Hosted screenshot URLs (output=url) are available for 30 days. For permanent storage, use output=binary or output=base64 and store the images yourself. **How many screenshots can I capture in bulk?** There is no batch limit: send one request per page, as many side by side as you need. ## AI-Powered Web Scraping Use AI to automatically extract structured data from any website and enhance it with summaries, translations, and analysis. No CSS selectors needed. https://simplescraper.io/features/ai-extraction-and-enhance **How does AI web scraping work with Simplescraper?** Our AI web scraper lets you extract data using natural language - no coding needed. Just describe what you want like "get product names and prices" instead of writing code. The AI scraping tool automatically finds and extracts that data, even when websites change their layout. **Can I scrape any website with AI?** Yes! Our AI scrape tool works on any website - ecommerce sites, social media, job boards, news sites, and more. The AI web scraper handles JavaScript sites, dynamic content, and complex layouts automatically. Just tell it what data you need in plain English. **What can AI enhance do with my data?** After scraping, AI can summarize content, translate text, clean messy data, extract sentiment from reviews, or transform data any way you need. For example, turn social media feeds into summaries or job listings into structured JSON. **Do I need technical knowledge to use AI features?** No! Just describe what you want in plain English. Instead of learning CSS selectors or XPath, simply type what data you need and let AI handle the technical details. ## Scrape Websites Behind a Login Extract data from sites requiring login with cookies or credentials. Works with LinkedIn, Facebook, Instagram, and any password-protected site. https://simplescraper.io/features/scrape-behind-login **How do I scrape password-protected websites?** Two easy methods: 1) Save your username/password in our encrypted credentials manager - works across all scrapers for that site, or 2) Use the cookie method - save your cookies from our Chrome extension while logged in. **Which method is better - credentials or cookies?** Cookies are usually easier and more secure (no password sharing), but they can expire. Credentials last forever but require sharing your password (encrypted). For sites with 2FA or extra security, we recommend cookies. For long-term automated scraping, credentials work better. **Is my login information secure?** Yes. Your password is encrypted, securely stored and not visible to Simplescraper. With the cookie method, we never even see your password - just your browsing session after you've logged in. **What if my login stops working?** Cookies expire based on the website's settings (usually days to weeks). When they expire, just grab fresh cookies from your browser. For credentials, update the password in the Credentials Manager whenever you change it on the site. **Which websites can I scrape behind login?** Most password-protected sites work including business tools, member-only areas, and private dashboards. Sites with complex captchas may need additional setup. ## Bulk-scrape a list of URLs into structured data with no code Select the data you want, paste up to 10,000 URLs and get structured data for every page. Download CSV or JSON or send it anywhere with a webhook. No code. https://simplescraper.io/features/scrape-list-of-urls **How many URLs can I scrape at once?** Up to 10,000 per run. **Can I paste URLs from a spreadsheet?** Yes. Paste the column, upload a CSV, or use links you scraped with another scraper. **Do the pages need to be on the same site?** They need the same layout. One scraper covers every product page in a store. For different sites, save a scraper each. **Do I need to write selectors?** No. Simplescraper needs to know which data you want. Select it on the page with the extension or in the dashboard, with help from AI if you like. **What format do I get?** JSON and CSV. The data can also be sent anywhere on the web via webhook, API or an integration. **Is it free to try?** Yes. New accounts get 100 free credits. ## Website Change Monitoring Monitor a web page, an area or one element for changes and get an email, webhook or agent alert showing exactly what changed. https://simplescraper.io/features/website-change-monitoring **How often can a monitor check a page?** From every 5 minutes to once a day. **What can a monitor watch?** The whole page, an area you draw or one element you click. Choose text changes, appearance changes or both, and get an alert on any change or only when a keyword appears or disappears. **How do alerts arrive?** By email, by webhook to your own app or automation, or to your AI agent. Each alert shows exactly what was added and removed. **What happens if the page fails to load?** A blocked or half-loaded check is never reported as a change. Your baseline stays put until the page loads properly again. **How do I check a page outside the schedule?** Click 'Check now' on the monitor's page. ## Scheduled Web Scraping Run scrapers every 30 minutes, hourly, daily or on a custom schedule in your timezone, and send every run to Google Sheets, a webhook or the API. https://simplescraper.io/features/schedule-and-monitor **How often can I schedule automatic scraping?** Every 30 minutes, hourly, daily or weekly. Or set a custom schedule down to every few minutes, with specific times like "daily at 9:15 AM" or "every Monday, Wednesday and Friday at 12 PM and 6 PM". **How do I monitor scheduled scrapes?** View logs in your dashboard to see when scrapes ran, what data was extracted, and any errors. **What happens if a scheduled scrape fails?** A page that fails is retried once automatically. The run log shows what failed, so you can fix the recipe. **Can I pause or modify scheduled scrapes?** Yes, pause, edit, or delete schedules anytime from your dashboard. Changes take effect immediately. ## Scrape Data Directly to Google Sheets Send scraped data automatically to Google Sheets. Real-time sync, scheduled updates, and seamless workflow integration. https://simplescraper.io/features/google-sheets-integration **How do I connect Simplescraper to Google Sheets?** Click "Connect Google Sheets" in your scraper's Send tab. Authorize once, and every scrape automatically updates your spreadsheet. No manual copying needed. **Will new data replace my existing data?** You choose! Set it to append new rows (keeping old data) or replace everything for fresh data. Most users append to build historical records over time. **Can my team access the scraped data?** Yes! Share the Google Sheet with your team using normal Google permissions. Everyone with access sees the new rows as soon as each scheduled scrape lands. **Can I scrape to multiple sheets?** Absolutely! Create multiple scrapers, each connected to a different Google Sheet - or different tabs in the same Sheet. Perfect for organizing data by project, client, or data type. ## Turn Any Website Into Structured Data Automatically detect and extract tables, lists, product info, and structured data from any website without writing selectors. https://simplescraper.io/features/extract-structured-data **What types of data can Simplescraper extract?** Any visible website data: product details, prices, contact info, tables, lists, images, links, reviews, and more. If you can see it on the page, Simplescraper can extract it. **Do I need to know coding or CSS selectors?** No coding needed! Use our Chrome extension to point and click the data you want. Or try Smart Extract - open Get new data in the dashboard, paste a URL, and describe the data you want in plain English. **Can it handle JavaScript-heavy websites?** Yes! Simplescraper runs a full Chrome browser that loads JavaScript, waits for content, and handles dynamic sites. It even works with lazy-loaded content and data hidden behind buttons. **Can I extract URLs and then scrape those pages?** Yes! This two-step process is called deep scraping. First extract all product links from a category page, then automatically scrape detailed data from each product page. **How do I handle pagination or infinite scroll when extracting data?** Simplescraper can automatically click "Next" buttons, load more content, or scroll down to get all results. Just point to the button or set scroll distance, and it handles the rest. **How do I export the extracted data?** Export to CSV, JSON, Google Sheets, Airtable or directly to your database via webhooks. You can also use our API to access data programmatically. ## Extract All Links from Any Website Find and extract all URLs from websites. Perfect for building crawl lists, finding resources, or mapping site structures. https://simplescraper.io/features/url-extractor **How do I extract all links from a website?** Paste a website's address into the URL extractor and it lists every page it can find, from the site's sitemap. Filter the list by any part of the address, such as a domain or /blog/, then copy it or download it. Useful for building sitemaps or finding every product in a category. **How many links can the URL extractor find?** There is no cap unless you set one. A request runs for up to 60 seconds and returns every URL found by then. Through the API you get 50 requests a day on the free tier and 500 with a subscription. **Can I extract specific types of links only?** Yes. Filter the list by any part of the address, for example /blog/ for internal blog posts or /products/ for the product pages of one category. ## Remove Duplicate Scraped Data Automatically Eliminate duplicate entries in your scraped data. Smart deduplication based on custom fields ensures clean datasets. https://simplescraper.io/features/deduplication **How does deduplication work?** Choose a unique field like URL or product ID as your deduplication key. Simplescraper automatically skips items you've already scraped, preventing duplicate data. **When should I use deduplication vs the crawler?** Use deduplication when repeatedly scraping pages that mix old and new content (like job boards or news feeds). Use the crawler for scraping many different URLs at once. Deduplication builds a permanent database over time; crawler is for one-time batch operations. **How long is my data kept?** Results from each run are kept for 30 days. Datasets built by incremental scraping persist and grow. Anything sent to Sheets, Airtable or a webhook is yours to keep. ## Send Scraped Data Anywhere With Webhooks & Zapier Send scraped data to any app using webhooks. Integrate with Zapier and Make integrations for automated workflows. https://simplescraper.io/features/webhooks-integration **What are webhooks and how do they work?** Webhooks automatically send your scraped data to other apps instantly. Add a webhook URL, and every time you scrape, the data goes directly to your app, database, or automation tool. **Which apps can I connect with webhooks?** Connect to Zapier, Make (Integromat), n8n or any app that accepts webhooks. From there your data can reach Slack, Airtable, Google Sheets, CRMs and your own applications. **How do I set up a webhook?** Just paste your webhook URL in the Send tab of any scraper. That's it! Every scrape will automatically send JSON data to your endpoint. No coding required. ## AI Agent Automation (MCP) Connect Simplescraper to AI agents via MCP protocol. Enable Claude, Cursor, and ChatGPT to scrape live data for enhanced workflows. https://simplescraper.io/features/ai-agent-automation-mcp-server **What is the Simplescraper MCP server?** MCP (Model Context Protocol) lets AI assistants like Claude Desktop and Cursor directly control Simplescraper. Connect once, then use natural language to create scrapers, run scrapes, and analyze data. Just say "extract all products from this website" and your AI handles the rest. **How do I connect Claude or Cursor to Simplescraper?** For Claude Desktop: Go to Settings → Integrations → Add MCP Server, then enter our URL: https://mcp.simplescraper.io/mcp. For Cursor: Add the same URL to your mcp.json config file. After authenticating, you can control Simplescraper with natural commands. **What can AI agents do with Simplescraper MCP?** Most of what you do with scrapers in the dashboard. AI can list and search your scrapers, create new scrapers, update selectors, run scrapes, retrieve results, manage batch URLs, take screenshots, and analyze data. Example: "Compare my price tracker's latest results with last week's run and list the items that dropped over 10%." **Can AI agents use Smart Extract without CSS selectors?** Yes! Just tell your AI what to extract in plain English. Say "Extract product names, prices, and ratings from [URL]" and the AI uses Smart Extract to get the data without any technical setup. Perfect for quick data extraction tasks. ## Turn Websites into Live APIs Create instant APIs from any website. Get fresh data on-demand with our powerful REST API endpoints. https://simplescraper.io/features/realtime-api-endpoints **How do I turn any website into a structured data API?** Create a scraper (via dashboard or API), then call the /run endpoint to get structured JSON data anytime. Pass different URLs to the same scraper for dynamic scraping. Your API key is on each scraper's API tab. Every scraper becomes a reusable API endpoint that returns clean, structured data instantly. **What can I do with the Simplescraper API?** Run scrapes, create and update scrapers with CSS selectors, search your existing scrapers, manage batch URLs, and use AI Smart Extract. Most users start with the /run endpoint for real-time data. **Can I extract markdown content via the API?** Yes! Add extractMarkdown: true to your API request to get a clean markdown version of any webpage alongside your structured data. Perfect for feeding content to LLMs, creating documentation, or archiving readable versions of web pages. **Can I pass different URLs to the same scraper?** Yes! Include sourceUrl in your POST request to scrape any URL with the same scraper. Perfect for dynamic scraping like scraping multiple products from the same site. ## Scrape Multi-Page & Infinite Scroll Sites Automatically handle pagination, load more buttons, and infinite scroll. Scrape entire catalogs and feeds effortlessly. https://simplescraper.io/features/pagination-and-infinite-scroll **How do I scrape multiple pages with pagination?** Click the pagination icon in our Chrome extension, then click on the Next button or page number. When saving your scraper, specify how many pages to scrape and Simplescraper will automatically click through all pages. **Should I use pagination or batch scraping for multiple pages?** If URLs follow a pattern (like /page/1, /page/2), use batch scraping - it's much faster as pages scrape concurrently. Use pagination when you must click through pages sequentially or when URL patterns aren't predictable. See our batch scraping guide for URL generation tools. **What sites work best with infinite scroll scraping?** Perfect for modern feeds and timelines: Twitter/X feeds, product reviews, news sites, and social comments. Extract months of historical data or monitor new content as it appears. If it scrolls endlessly, we can scrape it. **How many pages or scrolls can I scrape in one go?** While technically unlimited, we recommend up to 100 scrolls for optimal performance - beyond this, scraping can slow down. For larger extractions, batch scraping is recommended. Contact us for high-volume solutions and workarounds. **What if clicking the Next button accidentally navigates away?** Hold the SHIFT key while hovering over the navigation element instead of clicking. This selects the element without triggering navigation. --- # Documentation ## Quick Start and FAQ https://simplescraper.io/docs # Getting started with Simplescraper Simplescraper is a data extraction and website monitoring platform for developers and no-code teams. It turns any website into structured data - CSV, JSON or Markdown - and watches web pages for changes. You can scrape in the browser for instant results, in the cloud for automation and scale, or via API for programmatic access. Website monitoring checks a page on the schedule you set and alerts you by email, webhook or AI agent when its text or appearance changes. Start with our [Choosing how to scrape](https://simplescraper.io/docs/local-scraping-cloud-scraping-difference) guide to extract data in the format and scale best suited for you, [Monitor a web page for changes](https://simplescraper.io/docs/monitor-web-page-changes) to watch a page, or choose a guide below. Quick Start [1 Install extension Add Simplescraper to Chrome](https://chrome.google.com/webstore/detail/lnddbhdmiciimpkbilgpklcglkdegdkg) [2 Select your data Click elements on any webpage](https://simplescraper.io/docs/how-scrape-save-recipes#scraping-data) [3 Download or save Export, or save as a recipe to reuse](https://simplescraper.io/docs/how-scrape-save-recipes#step-6-save-as-a-recipe-optional) ## Guides Browser Scraping [Intro to browser scraping Instant results, scrape right from the extension](https://simplescraper.io/docs/browser-scraping-guide) [Scraping and saving recipes Select elements, refine selectors, save for reuse](https://simplescraper.io/docs/how-scrape-save-recipes) [Expanding list selection Broaden your selection to capture more items](https://simplescraper.io/docs/expand-list-selection) [AI Property Naming Automatically name your scraped properties with AI](https://simplescraper.io/docs/ai-property-naming) [Bulk scraping in the browser Scrape multiple URLs in a single session](https://simplescraper.io/docs/browser-batch-scraping) [Scraping data behind a login Use cookies or credentials for protected content](https://simplescraper.io/docs/scrape-behind-login) Cloud Scraping [Intro to cloud scraping Automation, scale, and scheduling](https://simplescraper.io/docs/dashboard-guide) [Multiple page and infinite scroll Pagination and infinite scroll scraping](https://simplescraper.io/docs/multiple-page-infinite-scroll) [Bulk scraping in the cloud Crawl lists of URLs at scale](https://simplescraper.io/docs/crawling-lists-of-urls) [Deep scraping URLs Scrape a list then scrape each result](https://simplescraper.io/docs/deep-scraping-urls) [Scheduling your webscraping Set up automatic recurring scrapes](https://simplescraper.io/docs/scheduling-webscraping) [Incremental scraping Append what's new each run, skip duplicates](https://simplescraper.io/docs/incremental-scraping) [Ready-made recipes Pre-built recipes for popular websites](https://simplescraper.io/docs/ready-made-recipes) Website Monitoring [Monitor a web page for changes Get an alert when a page's text or appearance changes](https://simplescraper.io/docs/monitor-web-page-changes) API [Extract API v1 Programmatic data extraction for developers](https://simplescraper.io/docs/api-guide) [Screenshot API Capture screenshots of any webpage via API](https://simplescraper.io/docs/screenshot-api) [Google Search API Get Google results for any query via API](https://simplescraper.io/docs/google-search-api) AI Features [Smart Extract AI-powered extraction from any URL](https://simplescraper.io/docs/smart-data-extract) [Enhance scraped data using AI Transform and enrich your scraped data](https://simplescraper.io/docs/enhance-scraped-data-using-ai) [Extracting websites to Markdown Convert any webpage to clean Markdown](https://simplescraper.io/docs/extract-markdown) [MCP server Connect Simplescraper to AI agents and tools](https://simplescraper.io/docs/mcp-server) Integrations [Google Sheets Export scraped data directly to Sheets](https://simplescraper.io/docs/saving-data-google-sheets) [Airtable Send scraped data to Airtable bases](https://simplescraper.io/docs/scraping-data-into-airtable) [Zapier Automate workflows with Zapier integration](https://simplescraper.io/docs/scraping-data-into-zapier) [Make (Integromat) Connect to Make automation scenarios](https://simplescraper.io/docs/scraping-data-into-integromat) [Email Send scraped data to your inbox](https://simplescraper.io/docs/scraping-data-into-email) [Webhooks Push data to any endpoint in real time](https://simplescraper.io/docs/sending-data-to-webhook) Account [Adding a password Set up password login for your account](https://simplescraper.io/docs/adding-password-login) [Your subscription Manage your plan, billing, and credits](https://simplescraper.io/docs/managing-subscription) [Troubleshooting Common issues and how to fix them](https://simplescraper.io/docs/troubleshooting) ## FAQ ### How much data can I extract at a time? If you prefer to scrape via the [crawler](https://simplescraper.io/docs/crawling-lists-of-urls) for bulk scraping, you can scrape up to 10,000 URLs at a time per scrape recipe (there's no limit on how many recipes you can run at once). If scraping via [API requests](https://simplescraper.io/docs/api-guide), there are no concurrency limits - Simplescraper will scale with your requirements. ### Can you scrape the data for me? Simplescraper makes it easy for you to extract the data yourself, but if you'd rather not, you have options: - Use one of our [ready-made recipes](https://simplescraper.io/docs/ready-made-recipes) for popular websites - Have us build a custom solution and deliver the data to you Not sure which approach is best? [Get in touch](https://simplescraper.io/). ### **What are scrape recipes?** Scrape recipes are reusable data extraction patterns that tell Simplescraper exactly what to extract from a website. A recipe contains the URL of the target website and a set of CSS selectors that pinpoint the specific data you want to collect. Once created, recipes can be run repeatedly to extract fresh data, even as website content changes. You can create recipes using our browser extension, through the Smart Extract feature, or programmatically via our API. ### Can Simplescraper scrape behind a login? Yes - both via [browser scraping](https://simplescraper.io/docs/browser-scraping-guide) (using your existing logged-in session) and [cloud scraping](https://simplescraper.io/docs/scrape-behind-login) (using cookies or credentials). ### How do credits work? Credits allow you to scrape in the cloud. Scraping a single page with Javascript enabled uses 2 credits, and scraping without Javascript uses 1 credit. Because most modern websites require Javascript, it is enabled by default, but you can change this when creating/editing a recipe. You start with 100 free credits when you sign up so be sure to use them! On paid plans, credits renew each month and are valid for the subscription month only. You can cancel your subscription at any time from your [account settings](https://simplescraper.io/dashboard). ### How long are scrape results stored? Results from each run are available for 30 days. To keep data for longer, download it or send it to an integration like Google Sheets. Recipes using [incremental scraping](https://simplescraper.io/docs/incremental-scraping) work differently: each run appends to one dataset that stays beyond the 30-day window. Full details: [data retention](https://simplescraper.io/docs/data-retention). ### How to edit a recipe Click the 3 dots to the right of the recipe name on any recipe page to edit that recipe. You can also share, clone (duplicate) or delete a recipe from this menu. ![Edit a recipe](https://simplescraper.io/docs/assets/edit-recipe.-d2svS6B.png) ### How to scrape links and image URLs? When you select an element that is a hyperlink or an image, the link should be detected automatically and will appear in the scrape results. If for some reason the results do not contain links, try saving the recipe and running it in the cloud. ### How to scrape multiple pages and infinite scroll? You can scrape multiple pages and infinite scroll directly in the browser extension or via cloud scraping. See the [Multiple page and infinite scroll scraping](https://simplescraper.io/docs/multiple-page-infinite-scroll) guide for more info. ### How to first scrape a list of URLs and then scrape each of those individual URLs? Simplescraper makes this easy with bulk scraping, which can be accomplished both in the browser and the cloud. See [Crawling lists of URLs](https://simplescraper.io/docs/crawling-lists-of-urls) for instructions. ### How to manage my recipes? You can access all your scrape recipes from the [Simplescraper dashboard](https://simplescraper.io/dashboard) or by clicking the Simplescraper extension icon and selecting 'View recipes and results'. ### Does Simplescraper use proxies? Yes, Simplescraper auto-rotates IP addresses to ensure that we're able to scrape successfully. ### Does Simplescraper rate-limit requests? Yes, we scrape responsibly and an individual scrape job should be no different than a handful of people browsing a website. ## Choosing how to scrape https://simplescraper.io/docs/local-scraping-cloud-scraping-difference # Choosing How to Scrape Every scrape starts with the [Chrome extension](https://chrome.google.com/webstore/detail/lnddbhdmiciimpkbilgpklcglkdegdkg). Use it to select the data you want and save it as a reusable scrape recipe. From there, choose how to run it - in the browser, in the cloud, or via API. - **Browser scraping** runs and is managed in your browser via the extension. Results appear in seconds. - **Cloud scraping** runs on Simplescraper's servers and is managed in the [dashboard](https://simplescraper.io/dashboard). Use it for scheduling, scale, and background automation. - **API scraping** is cloud scraping triggered [programmatically](https://simplescraper.io/docs/api-guide). Ideal for developers and custom integrations. ### Compare capabilities | **Capability** | **Browser** | **Cloud** | **API** | | --- | --- | --- | --- | | Runs from | Extension | Web Dashboard | Your code or tools | | Scale | One page at a time | Unlimited concurrency | Unlimited concurrency | | Save scrape recipes | Yes | Yes | Yes | | Scrape in background | No | Yes | Yes | | Logged-in scraping | Yes | Yes (with cookies) | Yes (with cookies) | | Multi-page | Yes | Yes | Yes | | Scheduled runs | No | Yes | Via external triggers | | AI Enhance | Yes | Yes | Yes | ## Browser Scraping Scrape data directly from your browser with the extension. ### Ideal for - Instant results in your browser - Hands-on scraping with full control - Logged-in content via your session - Quick one-off extractions ### Guides - [Scraping and saving recipes with the extension](https://simplescraper.io/docs/how-scrape-save-recipes) - [Scraping data behind a login](https://simplescraper.io/docs/scrape-behind-login) - [Troubleshooting](https://simplescraper.io/docs/troubleshooting) ## Cloud Scraping Scrape using Simplescraper's servers, in the background, on schedule. ### Ideal for - Large-scale data collection - Automated monitoring and tracking - Scheduled recurring scrapes - Auto-feed to spreadsheets and tools ### Guides - [Multiple page and infinite scroll scraping](https://simplescraper.io/docs/multiple-page-infinite-scroll) - [Crawling lists of URLs (Bulk scraping)](https://simplescraper.io/docs/crawling-lists-of-urls) - [Deep scraping URLs](https://simplescraper.io/docs/deep-scraping-urls) - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping) - [Incremental scraping](https://simplescraper.io/docs/incremental-scraping) ## API Scraping Scrape programmatically from your app or service and retrieve results as JSON. ### Ideal for - Full programmatic control - run scrapes from your own code - External tools - trigger from Make, Zapier, or any automation platform - Custom integrations - feed data into your own apps or dashboards - High-volume pipelines - unlimited concurrency for large-scale operations ### Guides - [API documentation](https://simplescraper.io/docs/api-guide) ## Credits https://simplescraper.io/docs/credits # Credit usage Credits are consumed when you scrape multiple pages in the browser, scrape in the cloud, or use AI features. What a page costs depends on where it runs. ## Browser scraping | What you scrape | Credits | | --- | --- | | One single page | Free | | Each page of a multiple page scrape | 1 | ## Cloud scraping | What you scrape | Credits | | --- | --- | | One page with JavaScript enabled | 2 | | One page with JavaScript disabled | 1 | Many modern websites require JavaScript, so it's enabled by default. When creating or editing a recipe, you can adjust this setting. The 1 credit rate applies to single-page runs only, because a run that covers several pages loads a full browser for every page. ## AI features | What you process | Credits | | --- | --- | | 800 words, read or generated | 1 | ## Renewal On paid subscriptions, credits renew each month and are valid only for the subscription month. For questions or help, reach out via chat. ## Intro to browser scraping https://simplescraper.io/docs/browser-scraping-guide # Intro to Browser Scraping Browser scraping runs directly in your browser via the Simplescraper extension. Results appear in seconds, right in the side panel. This is the fastest way to scrape - select your data, click scrape, and export. No waiting for cloud servers, no credits used for single pages. ## What you can do with browser scraping - **Instant results** - data appears in seconds as you scrape - **Scrape behind a login** - use your existing browser session to access protected content - **Multi-page scraping** - paginate through results or scroll infinite feeds - **Export immediately** - download CSV, JSON, or copy to clipboard - **Save recipes** - save your scraping pattern to run again later (in browser, cloud, or via API) ## Getting started 1. [Install the extension](https://chrome.google.com/webstore/detail/lnddbhdmiciimpkbilgpklcglkdegdkg) from the Chrome Web Store 2. Navigate to the page you want to scrape 3. Click the Simplescraper extension icon to open the side panel 4. Select your data and click **"Start Scraping"** For a full walkthrough, see [Scraping and saving recipes with the extension](https://simplescraper.io/docs/how-scrape-save-recipes). ## When to use browser scraping Browser scraping is ideal when you need: - **Speed** - results appear instantly, no waiting for cloud processing - **Logged-in access** - scrape content behind a login using your active session - **Hands-on control** - watch the scrape happen and adjust as needed - **One-off exports** - grab the data and move on without saving a recipe For scheduled automation, large-scale scraping, or background operation, [cloud scraping](https://simplescraper.io/docs/dashboard-guide) is a better fit. ## Guides - [Scraping and saving recipes with the extension](https://simplescraper.io/docs/how-scrape-save-recipes) - [Bulk scraping in the browser](https://simplescraper.io/docs/browser-batch-scraping) - [Scraping data behind a login](https://simplescraper.io/docs/scrape-behind-login) - [Troubleshooting](https://simplescraper.io/docs/troubleshooting) ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Scraping and saving recipes https://simplescraper.io/docs/how-scrape-save-recipes # How to scrape and save recipes with the extension Meet Simplescraper 2.0: The new Simplescraper sidepanel extension makes web scraping more simple than ever. Follow the updated steps below to start scraping data quickly and easily. ## Install and pin the extension First, install the Simplescraper extension from the [Chrome Web Store](https://chrome.google.com/webstore/detail/simple-scraper-%E2%81%A0%E2%80%94-scrape/lnddbhdmiciimpkbilgpklcglkdegdkg?hl=en). To make it easily accessible, pin it to your toolbar by clicking the extension menu (puzzle piece icon) and clicking the pin next to Simplescraper: ## Scraping Data ### Step 1: Select scrape type and list First, navigate to the webpage you want to scrape. Then click the Simplescraper icon in your toolbar to open the side panel on the right. Choose your scrape type: - **Scrape Lists** (default) - For extracting repeating data like products, articles, search results etc - **Scrape Details** - For grabbing specific fields from a single page like profile pages, product details #### For Scrape lists option: 1. Click **"Detect Lists"** to auto-identify lists on the current page. Each list will be numbered and ranked by likelihood 2. Simplescraper will automatically select List 1 and show 'List 1 confirmed' 3. To choose a different list, click the numbered buttons to switch to that list. A green "Active" badge will appear to confirm your selection **Manual selection:** Click **"Manually choose"** if auto-detection doesn't find your list, then click on the list you intend to scrape. #### For Scrape Details option: Use Scrape Details when you want to extract specific fields from a single page (like a product page, profile, or article). 1. Select **"Scrape Details"** from the scrape type dropdown 2. Click on elements you want to extract - each click adds a property 3. Rename properties as needed in the Edit Properties tab Scrape Details recipes work well with [bulk scraping](https://simplescraper.io/docs/crawling-lists-of-urls) - create the recipe on one page, then run it across a list of similar URLs. ### Step 2: Configure page navigation (optional) If you need to scrape multiple pages follow these steps: 1. Select your navigation method from the dropdown: - **Next page button** - For clicking through numbered pages or "next" buttons - **Infinite scroll** - For pages that automatically load content as you scroll - **Load more button** - For clicking buttons that reveal additional content - **No navigation** - For scraping the current page only 2. Confirm the auto-detected selector or manually enter the CSS selector 3. Set max pages to scrape (leave blank for unlimited pages - simplescraper will stop when it finds no new data) 4. Adjust wait time between pages if needed (default is 2 seconds) *Note: Single page scraping is free. In the browser each additional page uses 1 credit, and in the cloud 2.* ### Step 3: Start scraping Click the **'Start Scraping'** button. Simplescraper will navigate through your pages, extracting data in real-time. When complete, you'll see your results in a sortable table format with extraction summary (e.g., "Extracted 150 items from 3 pages"). To run another scrape, click 'Try another Scrape'. ### Step 4: Edit your properties In 'Scrape Lists' mode, Simplescraper automatically selects data properties to extract. To customize these: 1. Click the 'Edit Properties' tab to see each property with its CSS selector and data preview 2. **To rename**: Hover over the property name, click the pen icon, enter the new name, and press Enter or click the green check 3. **To delete**: Click the X button to remove the property The edit properties tab also shows the data type (text, image, url) and number of list items (e.g., "100/100 rows") each property appears on. **Tip:** To rename every property at once, click **Rename with AI** - Simplescraper reads your data and suggests clear names automatically. See [AI Property Naming](https://simplescraper.io/docs/ai-property-naming). ### Step 5: Export your data Click the buttons below the table of scrape results to download the data in your preferred format. **Export options:** - **CSV** - For Excel, Google Sheets, or any spreadsheet app - **JSON** - Structured data for developers and APIs - **Copy** - Copy to your clipboard to paste into documents or AI tools You can also copy individual columns directly from the results table. ### Step 6: Save as a recipe (recommended) If you want to run this scrape task again, click **"Save Recipe"** to save your scraping pattern. Once saved, you can: - **Run it instantly** from the extension's "My Recipes" tab (see Step 7) - **Run it in the cloud** on schedule or at scale via the [Simplescraper dashboard](https://simplescraper.io/dashboard) - **Call it via API** for programmatic access Saving is optional - if you only need the data once, export it and you're done. ### Step 7: Run a saved recipe To run a recipe you've saved: 1. Click the Simplescraper extension icon to open the side panel 2. Click the **"My Recipes"** tab to view your saved recipes 3. Find your recipe (use search if you have many recipes) 4. Click on the recipe to open it 5. Click **"Run Recipe"** and Simplescraper will open a new tab and begin scraping based on your saved configuration #### Tips - To scrape multiple URLs at once, see [Bulk scraping](https://simplescraper.io/docs/crawling-lists-of-urls) - To run in the cloud, click **"Open in Dashboard"** in the top right, or visit [simplescraper.io/dashboard](https://simplescraper.io/dashboard) ## What's next? Once you've saved a recipe, you can also run it in the cloud for more power: - [Schedule recurring scrapes](https://simplescraper.io/docs/scheduling-webscraping) - run daily, hourly, or on custom schedules - [Scrape multiple pages](https://simplescraper.io/docs/multiple-page-infinite-scroll) - pagination, infinite scroll, and load more - [Bulk scrape URLs](https://simplescraper.io/docs/crawling-lists-of-urls) - scrape hundreds of URLs at once - [Send data to integrations](https://simplescraper.io/docs/saving-data-google-sheets) - Google Sheets, Airtable, webhooks - [Access via API](https://simplescraper.io/docs/api-guide) - trigger scrapes programmatically ## Need Help? Click the chat button in the bottom-right corner to get in touch ## Expanding list selection https://simplescraper.io/docs/expand-list-selection # Expanding list selection > Select all similar items on a page, even when they're grouped into separate sections. When Simplescraper detects lists, it tries to be precise — which sometimes means it only selects items from one section of the page. For example, a jobs page might group listings by department (Engineering, Marketing, Design). Simplescraper may initially detect only one department's jobs. If you want to scrape all items across all sections, use **Include Similar** to expand the selection. ## How to expand a list selection 1. Click **Detect Lists** as normal 2. Check the item count — if it seems low, you may be missing items from other sections 3. Click **\+ include similar** next to the list count 4. Simplescraper will expand the selection to include all matching items on the page The item count will update to reflect the full selection. To go back to the original selection, click **— single list**. These steps are for the Chrome extension. On the dashboard's Scrape page, a list that repeats under several headings shows 'Include similar lists' under the list chooser: tick it to take the items from every section, and untick it to keep one. When Simplescraper finds the full list on its own, the page opens with it already ticked. ## When to use this - Job boards with listings grouped by department or category - Product pages with items organized into sections - Any page where similar content is split into multiple groups ## Need help? Click the chat button in the bottom-right corner to get in touch. ## AI Property / Column Naming https://simplescraper.io/docs/ai-property-naming # AI Property Naming > Turn default property names like `sc-1x2y3z` into meaningful names like `job_title` and `price` in one click. When [selecting data with the extension](https://simplescraper.io/docs/how-scrape-save-recipes), Simplescraper automatically generates property names (each property is a column in your data). But depending on how structured the website is, these names are often not very useful. For example, selecting a list of jobs may result in names like `sc-3ox-f4` instead of useful names like `job_description`. Renaming them manually is time-consuming, especially when you have many columns. AI Property Naming solves this. It understands the data you're scraping and generates clear, useful names for you automatically. ## How to use AI Property Naming 1. Run a scrape on any page 2. Click the **Edit Properties** tab 3. Click **Rename with AI** 4. Review the suggested names — deselect any you want to keep as-is 5. Click **Rename Properties** to apply The AI looks at the page title, URL, and sample data to infer what each property contains. Names are formatted consistently using underscores (for example, `job_title`, `product_price`, `company_name`). ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Bulk scraping in the browser https://simplescraper.io/docs/browser-batch-scraping # Bulk scraping in the browser > Scrape up to 10,000 URLs using a single recipe, directly in your browser. Bulk scraping is ideal for: - **Scraping detail pages** - extract data from individual pages after collecting their URLs from a list - **Processing your own URL list** - paste URLs you've gathered from another source - **Scraping logged-in content at scale** - your browser session gives you access to protected pages For scheduled or background bulk scraping, see [Crawling lists of URLs (Cloud)](https://simplescraper.io/docs/crawling-lists-of-urls). ## Prerequisites Before you start, you'll need: 1. The [Simplescraper extension](https://chrome.google.com/webstore/detail/lnddbhdmiciimpkbilgpklcglkdegdkg) installed 2. A saved recipe that works on the pages you want to scrape 3. A list of URLs - either your own, or from [another Simplescraper recipe](https://simplescraper.io/docs/how-scrape-save-recipes) that scrapes a list of links. We'll cover both approaches below. ## Method 1: Import URLs from a previous scrape Automatic URL saving: Simplescraper automatically saves your 5 most recent scrape results. When importing URLs from a previous scrape, they're already available under Batch URLs — no need to upload anything. This method chains two recipes together - first scrape a list of URLs, then scrape each URL with your detail recipe. ### Step 1: Run your list recipe 1. Navigate to the page containing your list of items 2. Open the Simplescraper extension 3. Go to **My Recipes** and click on your list recipe 4. Click **Run Recipe** The scrape will extract all items from your list, including their URLs in the results. ### Step 2: Open your detail recipe Once the list scrape completes: 1. Go to **My Recipes** 2. Click on your detail page recipe ### Step 3: Import the URLs 1. Click the **Batch URLs** tab 2. Click **Import from Results** 3. Select the list recipe you just ran (it should be the most recent one) 4. Click **Next** 5. Choose the property containing the URLs (Simplescraper will auto-detect URL fields) 6. Click **Import URLs** Your URLs are now loaded into the Batch URLs tab. ### Step 4: Run the bulk scrape 1. Go to the **Run Recipe** tab 2. Check that **Use batch URLs** is selected (you'll see the count of URLs) 3. Click **Run Recipe** Simplescraper will navigate to each URL in sequence, extract the data, and display results as it goes. You can watch the progress and see data appear in real-time. ## Method 2: Paste or upload URLs If you already have a list of URLs from another source: ### Paste URLs directly 1. Open your recipe and go to the **Batch URLs** tab 2. Paste your URLs into the text area (one per line) ### Upload a CSV file 1. Go to the **Batch URLs** tab 2. Click **Upload CSV** 3. Select your file containing URLs ### Run the batch 1. Go to the **Run Recipe** tab 2. Ensure **Use batch URLs** is selected 3. Click **Run Recipe** ## Download results Once the scrape completes, download your results as **CSV**, **JSON**, or copy to clipboard. ## Related guides - [Scraping and saving recipes with the extension](https://simplescraper.io/docs/how-scrape-save-recipes) - [Crawling lists of URLs (Cloud)](https://simplescraper.io/docs/crawling-lists-of-urls) - [Deep scraping URLs](https://simplescraper.io/docs/deep-scraping-urls) ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Scraping data behind a login https://simplescraper.io/docs/scrape-behind-login # Scraping data behind a login > Access protected content by using your credentials or browser cookies. There are two methods that Simplescraper can use to scrape data located behind a login: credentials (username / password) and cookies. The cookie method is preferable; however, cookies may expire quickly or not work at all, in which case the credential method makes more sense. This guide covers how to use both methods. ## Credentials method The credentials method uses your username and password to login on your behalf and extract the data that you need. You can create a credential for a website and use it to login to all of your scrape recipes that share that domain. For example, if you have five LinkedIn recipes, you can create a single credential for LinkedIn that will work for all your LinkedIn recipes. Once a credential is added, it is available under 'Credentials' in the Manage Recipes section. Your password is encrypted, securely stored and not visible to Simplescraper. ### Add credentials while creating / editing recipe - Add a new recipe or edit an existing recipe ([see this guide for how to](https://simplescraper.io/docs/how-scrape-save-recipes/)) and scroll down to the Credentials section under Advanced Options - Click the 'add new credential +' button and add your credential details in the popup that appears. Provide a credential name that is clear and obvious, for example 'LinkedIn login'. Then enter the remaining URL, username and password details - Click save credentials and wait for the 'Credentials added successfully' confirmation text and then close the popup - The credential that you created will now be visible in the dropdown list - select it and then save your scrape recipe ### Add credentials via Credentials menu - In the sidebar of the Simplescraper [dashboard](https://simplescraper.io/dashboard) click Manage Recipes and then choose Credentials Manager and then click the green 'add credential +' button - In the popup that appears add your credential details. Provide a credential name that is clear and obvious, for example 'LinkedIn login'. Then enter the remaining URL, username and password details - Click save credentials and wait for the 'Credentials added successfully' confirmation text - Now when you add a new recipe or edit an existing recipe you will see the credential that you created in a dropdown list in the Credentials section ## Cookies method Simplescraper now allows you to add cookies via the extension and this is the preferred method. The previous method of using the EditThisCookie extension is still included below as it may be useful if adding cookies to an existing recipe. ### Adding Cookies via the Simplescraper extension (preferred method) - Open the Simplescraper extension on the page that you wish to scrape and click on the Cookies icon (it's located two positions to the right of the 'View Results' button). The icon will turn green, indicating that the cookies have been saved - Continue selecting the properties you wish to extract as usual - Click 'View Results', then choose to save the recipe - In the advanced settings, under the Cookies section, you will see that the cookies have been added - Save your recipe and run it. Simplescraper will login and your data will be returned ### Adding Cookies via the Copy Cookies extension *Scraping a webpage behind a login is achieved by using your login cookies for that site. Retrieving this cookie requires using a 3rd party extension called Copy Cookies. Please review their permissions and privacy policy [here](https://chromewebstore.google.com/detail/copy-cookies/jcbpglbplpblnagieibnemmkiamekcdg).* Here's how to scrape data that is behind a login using the Copy Cookies extension: - Install the [Copy Cookies](https://chromewebstore.google.com/detail/copy-cookies/jcbpglbplpblnagieibnemmkiamekcdg) extension - Navigate to the website that you wish to scrape and login to the site if you are not already logged in - Once logged in, click the Copy Cookies extension icon and click the copy button to copy the cookies to your clipboard - In the Simplescraper dashboard ([https://simplescraper.io/dashboard](https://simplescraper.io/dashboard)), navigate to your recipe, click edit beside the recipe name and then click 'show advanced options' - Under the cookies section paste the cookies from your clipboard into the text area (*ctrl + v* or *right-click > paste*) - Save your recipe and then run it. Simplescraper should login on your behalf and scrape the data that you need - Note: some websites don't use cookies for user authentication and so this method may not always work ## Troubleshooting https://simplescraper.io/docs/troubleshooting # Troubleshooting Sometimes extracting data doesn't go as smoothly as expected. Fortunately most issues can be easily fixed by following the suggestions below. If you've tried these suggestions and still haven't extracted the data that you need then reach out via chat. ## Timeout errors This happens when an element that you have selected to extract cannot be found on the page. **Resolution** - Check the screenshots (click 'view screenshot' on the logs page) to view the webpage and ensure that all looks as expected - Ensure the data that you are scraping is not behind a login. The screenshot should make this clear. If a login is required then follow [this guide](https://simplescraper.io/docs/scrape-behind-login/) to scrape data behind logins - Ensure that the page loaded correctly. If the screenshot is blank or looks incorrect then this should be clear. If so, contact support - If you are familiar with CSS selectors, copy one of the selectors that you have chosen (edit your recipe to see all of the selectors for a recipe) and search for that selector on the webpage using the Inspector. Ensure that the selector exists and if it does not then update the selector with the correct value - If you are scraping multiple pages and most of your pages scraped successfully with only one or two showing a timeout error then Simplescraper may have navigated beyond the last page or included a page that does not have elements. You can typically ignore the error in this scenario as long as data was returned ## Intro to cloud scraping https://simplescraper.io/docs/dashboard-guide # Intro to Cloud Scraping Cloud scraping enables always-on scraping via schedules, concurrent scrapes across many recipes, and background operation - your scrapes run whether you have the dashboard open or not. The Simplescraper dashboard at [simplescraper.io/dashboard](https://simplescraper.io/dashboard) is where you manage and run your recipes in the cloud. ## What you can do from the dashboard - **View all your recipes** - see every recipe you've saved from the extension - **Run recipes in the cloud** - scrape without keeping your browser open - **Run multiple recipes at once** - tens or hundreds of recipes can run simultaneously - **Set schedules** - automate recurring scrapes (hourly, daily, or custom) - **Bulk scrape URLs** - scrape many URLs with a single recipe - **View and export results** - download data or send it to integrations - **Configure integrations** - connect to Google Sheets, Airtable, webhooks, and more ## Getting to the dashboard You can access the dashboard in several ways: - Visit [simplescraper.io/dashboard](https://simplescraper.io/dashboard) directly - Click **"Open in Dashboard"** from any recipe in the extension - Click the Simplescraper extension icon and select **"View recipes and results"** ## Running a recipe in the cloud 1. Open a recipe from your dashboard 2. Click **"Run Recipe"** 3. Simplescraper will scrape in the background and return results when complete You can close the browser or navigate away — the scrape continues on Simplescraper's servers. ## When to use cloud scraping Cloud scraping is ideal when you need: - **Scheduled automation** - scrape daily, hourly, or on custom schedules without manual intervention - **Scale** - run many recipes concurrently - **Background operation** - scrapes run whether your browser is open or not - **Integrations** - send data directly to Google Sheets, Airtable, or webhooks For quick, one-off scrapes where you want immediate results, [browser scraping](https://simplescraper.io/docs/how-scrape-save-recipes) is often faster. ## Guides - [Multiple page and infinite scroll scraping](https://simplescraper.io/docs/multiple-page-infinite-scroll) - [Crawling lists of URLs (Bulk scraping)](https://simplescraper.io/docs/crawling-lists-of-urls) - [Deep scraping URLs](https://simplescraper.io/docs/deep-scraping-urls) - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping) - [Incremental scraping](https://simplescraper.io/docs/incremental-scraping) ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Multiple page and infinite scroll https://simplescraper.io/docs/multiple-page-infinite-scroll # Multiple page and infinite scroll scraping > Scrape through paginated results or infinite scroll feeds automatically. You can scrape multiple pages and infinite scroll pages with Simplescraper. Before we dig into how, two important things to note: 1. The crawler (bulk scraping) may be a better option: if the URLs of the pages that you want to scrape follow a structured pattern like 'page=1' or 'page/2' etc, it's faster to scrape concurrently via the crawler. See *[Crawling lists of URLs](https://simplescraper.io/docs/crawling-lists-of-urls/)* for instructions 2. Pagination and infinite scroll should not be selected at the same time. If the page loads more content as you scroll down without any clicking required, use infinite scroll only. If a click of any kind is required, use pagination only ## Pagination To extract data across multiple pages, you must teach Simplescraper how to navigate: - Identify the element used to navigate to the next page (typically a 'next' button or an arrow), then click the pagination icon in the top menu to activate detection and then click on the navigation element. If done successfully you will see that the pagination icon has turned green. The css selector of the navigation element will also be visible when you hover over the pagination icon. - **On some pages, selecting the navigation element will unintentionally load the next page. In this case press the SHIFT key when hovered over the navigation element instead of clicking to select it** - When saving the recipe you will be presented with an option asking how many pages you wish to navigate. ## Infinite scroll - If the page that you intend to scrape loads new content automatically as you scroll down and you wish to scrape this data, click the infinite scroll icon located in the extension's top menu - The icon will turn green to indicate that it's active - When saving the recipe you will be presented with an option asking how many times you wish the automated recipe to scroll ### How infinite scroll runs - **The number you set is a maximum.** Each run loads the page, then scrolls down up to that many times. A recipe set to scroll 10 times loads the page and scrolls at most 10 times. - **Scrolling stops on its own when the page stops loading.** With the default settings, once two scrolls in a row reach the bottom and nothing new appears, the recipe stops and extracts, so a list that ends early finishes early. - **Each scroll counts as a page.** The first page load counts too, so a recipe set to scroll 10 times uses at most 11 pages. A run that stops early uses only the pages it made. - **You never pick what to scroll.** Simplescraper starts from the list you selected and scrolls whatever holds it: the whole page, or a panel with its own scroll bar, such as the results beside a map. ### Change the scroll settings of a saved recipe 1. Open the recipe in the dashboard and edit it. 2. Under 'Page navigation', set 'Page type' to 'Scroll'. 3. Enter 'Number of times to scroll down'. The most you can enter is 100. 4. Open 'More scroll options' if the page needs them: - **How far each scroll goes:** 'End of page' scrolls to the end of the list and waits for more items, which suits most pages. 'One screen' moves one screen at a time, for lists that only show the items currently on screen. - **Pause between scroll steps:** the wait in seconds between steps down the page, from 0 to 5. Give slow sites longer. - **When to collect items:** 'When scrolling ends' collects every item once at the end. 'After each scroll' collects after each scroll, for feeds that remove posts once you scroll past them, such as X timelines. The same item can then appear more than once. 5. Save the recipe. A page that does not load more items as you scroll can still be scrolled once before extracting: with 'Page type' set to 'Single page' or 'Multiple pages', set 'Scroll down the page before extracting' to 'On'. Each page is then scrolled to the end once before extracting, for content that only appears once scrolled into view, lazy-loaded images for example. For questions or help, reach out via chat. ## Bulk scraping in the cloud https://simplescraper.io/docs/crawling-lists-of-urls # Bulk scraping in the cloud > Scrape up to 10,000 URLs in the background — no browser required. The crawler (bulk scraper) allows you scrape up to 10,000 URLs at a time with [Simplescraper](https://simplescraper.io). This method is recommended as it is faster than navigating individually through pages. To use the crawler, save a recipe as normal and then click the 'Crawl' tab and paste the URLs that you wish to scrape into the text area. Simplescraper will detect that there are URLs in the crawler and will scrape using these URLs instead of the original URL and pagination settings. If you wish to scrape multiple pages but only have the URL of the first page, click to the second page of the website and note the URL. There should be a structured pattern like 'page=2', 'p=2', 'page/2' etc. This value should be incremented for each page that you intend to scrape. For example if you're scraping 100 pages then the last URL will contain a value like 'page=100'. To help with this you can use the Generate URLs tool as outlined below. ## Generating URLs Simplescraper includes a URL generator which allows you to easily create a list of URLs for multiple pages on a website, which can then be scraped in one go. To use it, follow these steps: - Identify the base URL that you intend to replicate for multiple pages. In the video example above, we use [https://pubmed.ncbi.nlm.nih.gov/?term=zinc](https://pubmed.ncbi.nlm.nih.gov/?term=zinc) - Navigate to the second page of that URL so that we can determine the pagination structure of the URL. In the video example we click to the second page and the URL changes to [https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=2](https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=2). Notice that this page is represented as *page=2* in the URL. This tells us that the URL of each subsequent page will follow the format page=3, page=4 etc. Copy this URL - Navigate to the 'Crawl' tab of your scrape recipe and click the 'Generate URLs' button. Enter the URL of the second page and replace the page number with an \[x\]. So in our example [https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=2](https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=2) becomes [https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=\[x\]](https://pubmed.ncbi.nlm.nih.gov/?term=zinc&page=%5Bx%5D) - In the input fields enter the following: - Enter the 'start value' (which is usually 1 if you wish to begin scraping from the first page) - Enter the 'end value' representing the last page you wish to scrape (ensure the website has this many pages) - Enter the 'increment' value which is how much each page number should increase by (this is typically 1 but can be 10, 20, 100 etc depending on the website) - Click generate - Once the URLs are generated, copy the list and paste them into the input field on the 'Crawl' tab. You can now click 'Run recipe' to scrape each of the individual URLs If you'd like help with any step, contact us via chat. **Alternatively you can import a list of URLs that you scraped with a different recipe as explained in *[this Deep Scraping guide](https://simplescraper.io/docs/deep-scraping-urls/)*.** ## Deep scraping URLs https://simplescraper.io/docs/deep-scraping-urls # Deep scraping URLs > Scrape a list of links, then automatically extract data from each linked page. Deep Scraping allows you to not only scrape a list of links from a main webpage, but also collect data from the respective subpages. This is useful for extracting data from platforms like job boards or hotel listings, where each listing's details are on individual subpages. To accomplish this, you'll use two recipes: one recipe to scrape the links and another recipe to extract the data behind each link. ## How to Use Deep Scraping 1. **First Recipe:** [Create and save](https://simplescraper.io/docs/how-scrape-save-recipes/) a recipe that scrapes a complete list of links that you're interested in. For example, if you're scraping job listings, this recipe scrapes the link of each job post 2. **Second Recipe:** Create and save another recipe that will scrape the content behind a single link from this list. For instance, if you're scraping job listings, this second recipe would be set to extract job information (salary, job description, requirements etc) from a single job description page 3. **Run and Import:** Run the first recipe via cloud scraping to get a list of links. Import these links into your second recipe via the second recipe's 'Crawl' tab. Then, run the second recipe to scrape the content from each link > Note: When extracting a property that includes a link, Simplescraper will automatically save this link to a column named 'propertyname\_link'. ## Example: Shopify App Store Deep Scraping To scrape app links and their contact details from the Shopify App Store, you'll use two Simplescraper recipes: - First, create a recipe to scrape the list of app links available on `https://apps.shopify.com/` - Then create a second recipe focused on extracting contact details from a single app page - Save the first recipe as 'Shopify - app links' and the second as 'Shopify - contact info' The video below shows how to select elements for the first recipe. Once both recipes are saved, start by running 'Shopify - app links'. The result will be a column filled with the extracted links. Next, switch to the 'Shopify - contact info' recipe, navigate to the 'Crawl' tab, and choose 'Import URLs'. From the list, select 'Shopify - app links' and the property containing the links (in this example, 'App link'). Confirm the import, and you're set to begin deep scraping. Now all you need to do is click 'Run recipe' on the second recipe and the contact details for each of the apps will be scraped. The video below shows the steps to import the links from the first recipe into the second recipe. ## Automatically running an imported recipe When running a recipe that is importing links from another recipe you may wish to run the imported recipe first so that the most recent links are scraped. To do this automatically, navigate to the 'Crawl' tab of the recipe that is importing the links (the 'second recipe') and toggle the 'run imported recipe first' option to on. This ensures the link-collecting recipe runs automatically before scraping the individual pages, keeping your scraped data current. > Note: > > - The 'run imported recipe first' option only becomes visible on the crawl tab once a recipe has been imported > - 'run imported recipe first' is not applicable for requests made via the [API](https://simplescraper.io/docs/creating-an-api/). It's best suited for recipes that are scheduled or run manually from the dashboard. To recreate this via the API, call the first recipe and the second recipe sequentially ## Scheduling your webscraping https://simplescraper.io/docs/scheduling-webscraping # Scheduling your web scraping > Run your scrapes automatically — every few minutes, daily, or on specific days. *Video showing how to set a schedule for every 15 minutes on Monday, Wednesday and Friday* Simplescraper includes advanced scheduling options to enable you to scrape web data automatically whenever you choose. You can set schedules as general as 'every 2 minutes' or as specific as '12:30pm on weekdays', and everything in between. To set a schedule, create a new recipe or edit an existing recipe and scroll to the scheduling section under Advanced Options. The most common types of scheduling are available in the dropdown: *Every 30 minutes*, *Hourly*, *Daily* at a specific time, and *Weekly* on a specific day and time. Choosing *Weekly* shows a 'Day' and a 'Time' field: the recipe runs once a week on that day, on the hour you pick in your local timezone. For more advanced scheduling select *Custom* and four new options will appear: Minutes, Hours, Days and Timezone. - **Minutes**: the time past the hour that you wish to scrape. 0 represents on the hour while 15 represents 15 minutes past the hour and so on. You can choose multiple options, for example choosing 0, 15, 30 and 45 will schedule the scrape recipe to run every 15 minutes - **Hours**: The hours of the day that you wish to scrape. Choosing 0:00, 6:00, 12:00, 18:00 will scrape every 6 hours for example - **Days**: the days of the week that you wish to scrape. You can choose every day or multiple options like only scraping on weekdays for example - **Timezone**: Simplescraper uses your local timezone by default but you can change the timezone to be used if scraping at a specific time of the day As you customize your web scraping schedule, text will display the chosen schedule for additional confirmation. For questions or help with scheduling please reach out via chat. ## Incremental scraping https://simplescraper.io/docs/incremental-scraping # Incremental scraping: one dataset that grows with every run In cases where you need to scrape the same page repeatedly - for example job boards, price lists, news feeds, directories - the most useful format is one continuously updated, non-duplicated list of data rows rather than individual distinct snapshots. This is possible with incremental scraping. ## When to use it Use incremental scraping when a recipe runs repeatedly against the same page and the page adds content over time: - Job boards, where each run should capture only new postings - News or announcement pages, where new items appear alongside old ones - Directories or lists you're assembling into one complete dataset When separate runs are better: Incremental scraping isn't required when each run should stand alone. The default separate-runs setting is better for snapshots you compare over time, such as weekly price checks. Bulk scraping doesn't require it either because each URL is scraped independently. ## How it works Incremental scraping keys every row on one property: the unique results key. Choose a value that identifies each row and isn't repeated elsewhere on the page - a URL, a product ID, an address. On each run, Simplescraper compares every scraped row against the existing dataset. Rows with new keys are appended; rows with known keys are skipped. Nothing is saved twice, and existing rows are never modified. *Setting a property as the unique key.* ## Set it up 1. Create a recipe with at least one property that holds unique values - for a job board, the posting's link. 2. When saving or editing the recipe, click "Show advanced options" and find the Incremental scraping section (previously called "Unique results key"). 3. Select the key property from the dropdown and save. Every run from this point appends to the recipe's dataset instead of creating separate results. ## Data retention | | Standard recipes | Incremental recipes | | --- | --- | --- | | Each run produces | A separate result set | New rows appended to one dataset | | Retention | Results available for 30 days | The dataset persists and grows | - Turning the key off keeps everything accumulated - changing a setting never deletes data. New runs return to saving separate results. - Turning it back on continues the same dataset where it left off. - Enabling incremental scraping on an existing recipe doesn't import past snapshots; the dataset fills from the source page as runs continue. ## Related - [Data retention](https://simplescraper.io/docs/data-retention) - the full picture of how long each kind of data is stored - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping) - incremental scraping pairs naturally with a schedule - [Bulk scraping in the cloud](https://simplescraper.io/docs/crawling-lists-of-urls) - each URL is scraped independently, so no unique key is needed For questions or help, reach out via chat. ## Data retention https://simplescraper.io/docs/data-retention # Data retention: how long your results are stored Results from each scrape run are available in Simplescraper for 30 days. To keep data for longer, download it, send it to an integration as part of your workflow, or use incremental scraping - which isn't subject to the 30-day window. | Data | Retention | | --- | --- | | Scrape results (each run) | Available for 30 days | | Incremental datasets | Persist and grow - not subject to the 30-day window | | Recipes | Kept for the life of your account | | Screenshot API hosted images | 30 days | ## Keeping data longer - Download results as CSV or JSON from the dashboard at any time within the 30 days. - Send results to an integration - [Google Sheets](https://simplescraper.io/docs/saving-data-google-sheets), Airtable, Zapier, Make, or a webhook. Data delivered to your own tools is yours to keep indefinitely. - Use [incremental scraping](https://simplescraper.io/docs/incremental-scraping) when you're building one dataset from repeated runs of the same page. ## Related - [Incremental scraping](https://simplescraper.io/docs/incremental-scraping) - one growing, deduplicated dataset that stays beyond the 30-day window - [Saving data to Google Sheets](https://simplescraper.io/docs/saving-data-google-sheets) - deliver every run to a sheet automatically For questions or help, reach out via chat. ## Ready-made recipes https://simplescraper.io/docs/ready-made-recipes # Ready-made recipes Simplescraper has a growing list of ready-made recipes that allow you to scrape a variety of websites and apps without having to create recipes yourself. The readymade recipes include the following: | Recipe | | --- | | Angel.co jobs | | Coinmarketcap - coin page | | Crunchbase - search results | | Empire Flippers - listings | | Google search results | | Google - trends | | Indeed.com - job results | | Indiehackers - top posts | | Instagram - hashtag search | | LinkedIn - job search results | | Producthunt - voter extractor | | Reddit - comments | | Reddit - posts | | Shopify - apps | | Yelp - search results | You can find the recipe store [here](https://simplescraper.io/recipes) or in the sidebar of the app. **If you have suggestions for a ready-made recipe, let us know via chat.** ## Google search results The 'Google search results' recipe saves Google's results for any search term as rows you can run, schedule and export like any other recipe. To find it, choose 'Templates' on the dashboard's Home page and click 'Browse templates'; it is the first card. 1. Enter a 'Search term'. 2. Set 'Pages', from 1 to 10. Each page is ten results. 3. Pick a 'Country' to search from (United States unless you choose another) and a 'Date range' (any time unless you choose another). 4. Click 'Save and run', or 'Save recipe' to run it later. Each run returns one row per result, with these columns: - **position:** the result's rank, counted from 1 across every page without gaps. - **page:** the Google page the result came from. - **title:** the result's title. - **url:** the result's address. - **snippet:** the text Google shows under the title, when it shows some. - **date:** the date Google shows beside the result, when it shows one. A run costs 2 credits per Google page fetched, so a 3-page search costs 6 credits. A page that fails is not charged, and the run's rows keep every page that worked. To run the same search from code, with the answer box, knowledge graph, 'People also ask' and related searches as well, use the [Google Search API](https://simplescraper.io/docs/google-search-api). ## Scrape recipe health https://simplescraper.io/docs/recipe-health # Recipe Health Every recipe in your dashboard has a health indicator - a small pill next to the recipe name that shows whether your scrape is working or needs attention. Simplescraper runs health checks after each run - things like HTTP response codes, whether data was returned, and selector stability. If something looks off, the pill turns amber or red so you can catch problems early. ## Viewing recipe health 1. Open any recipe from your [dashboard](https://simplescraper.io/dashboard) 2. Look for the health pill next to the recipe name - it shows a colored dot and a status label 3. Click the pill to open the **Scrape Health** panel The pill shows one of three states: - **Green** - all checks passed - **Amber** - one or more warnings - **Red** - critical issues that are likely affecting your data ## What gets checked The health panel checks your most recent scrape results and logs. Each check shows a colored dot, a label, and a short message. | Check | What it looks for | | --- | --- | | Last Run | Did the most recent run return any data? Flags critical if zero rows came back | | Bot Protection | Was bot detection (like Cloudflare or Akamai) triggered on the target site? | | HTTP Response | Did the site return a 200 status code? For bulk scrapes, shows how many pages returned non-200 responses and which codes appeared | | CSS Selector Stability | Are your selectors likely to break? Some sites use randomized class names that change on each deploy | | Column Consistency | Do all columns have the same number of rows? A mismatch means one selector is matching fewer elements than the others | | Page Errors | Did any pages fail to load? Shows the error type if available | | Run History | Have the last two or more runs in a row returned zero data? One empty run can be a fluke, but repeated failures point to a real problem | ## Understanding the results **Green dot** - the check passed. **Amber dot** - a warning. The scrape may still be returning data, but something is worth looking into. For example: bot protection detected, some pages returned non-200 codes, or column counts don't match. **Red dot** - a critical issue. Your scrape is probably not returning the data you expect. Usually this means the last run returned zero rows, or several runs in a row have failed. **Gray dot** - the check couldn't run, usually because there aren't enough results yet (the recipe has never run, or only has one run in its history). Some findings include a **"view logs"** link that takes you to the run logs where you can see screenshots and error details. ## Site Unblocker Some websites actively block automated scraping. Site Unblocker solves this by testing thousands of proxies and building a custom pool for your domain, so your scrapes get through without you having to find and configure proxies yourself. Site Unblocker is available on paid plans. It appears as a row at the bottom of the health panel. ### Enabling Site Unblocker 1. Open the **Scrape Health** panel for a recipe that is being blocked 2. Click the **Enable** button on the Site Unblocker row 3. Simplescraper starts testing proxies against that site - you'll see a progress indicator while this runs ### Unblocker status | Status | What it means | | --- | --- | | Not enabled | The unblocker hasn't been turned on for this site | | Setting up / Unblocking | Simplescraper is testing proxies against your site. This can take a few minutes | | Unblocker active (green) | Working proxies found - your scrapes will use them automatically | | Unable to unblock (red) | All available proxies have been tested and none could access the site. Contact support for help | ## Tips - Run your recipe at least once before checking health - most checks need a completed run to report anything useful - If you see a **Column Consistency** warning, check whether one of your selectors targets an element that doesn't appear on every list item - **CSS Selector Stability** warnings flag randomized CSS selectors that are likely to change more frequently than normal. Try re-selecting those properties with more durable selectors - For bulk scrapes with a few non-200 pages, check whether those URLs are still valid. A handful of 404s in a large batch is normal ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Monitor a web page for changes https://simplescraper.io/docs/monitor-web-page-changes # Monitor a website for changes Simplescraper Website Monitoring detects changes to the text or appearance of any web page and alerts you whenever something changes. You can be notified by email, webhook, or by sending the change to an AI agent. Creating and managing monitors is done via the dashboard. **Use cases for web page monitoring:** - Competitor price changes and plans - New job postings on a careers page - Products coming back in stock, for restock alerts - Financial and stock information - News and announcements Scrape or monitor?: Use a monitor to know when a page changes and see what changed. If you need to extract data, use a scrape recipe instead. ## How it works You decide what to watch: the words on the page, how the page looks, or both, across the whole page or just one part of it. Then, on the schedule you choose, Simplescraper checks the page again, compares it with the previous version and sends you a change alert if the section you care about has changed. ## Set up a monitor ### Step 1: Choose the page 1. In the [Simplescraper dashboard](https://simplescraper.io/dashboard/monitor), click 'Monitor' under the 'Create' section of the sidebar. 2. Enter the URL of the page to watch. The name field will fill in from the address. We recommend changing it to something you will recognise in an alert. 3. Choose 'How often to check', from every 5 minutes to once a day. 4. Click 'Check page'. Simplescraper loads the page, which usually takes a few seconds, and shows it beside the settings. ### Step 2: Choose what to watch 1. Under 'Changes to monitor for', choose what counts as a change: - **Text changes:** the words on the page. - **Appearance changes:** how the page looks. This catches visual changes, such as a swapped image or a restyled button, even when the words stay the same. - **Text and appearance changes:** both. 2. Under 'What part of the page to monitor', choose how much of the page to watch: - **Whole page:** everything on the page. - **An area:** draw a box on the page, then click 'Confirm'. - **An element:** click one part of the page, such as a price or a heading, then click 'Confirm'. Watching one part of the page keeps changes elsewhere, such as ads or recommendations, from sending alerts. ### Step 3: Choose when to be alerted 1. Under 'Alert me when', choose when to get an alert: - **Anything changes:** any change to the part of the page you watch. - **Keyword added:** one of your keywords appears on the page. For a restock, 'in stock' alerts you when those words appear. - **Keyword removed:** one of your keywords disappears from the page. 2. For keyword monitoring, enter the words in 'Keywords', separated by commas. ### Step 4: Choose where alerts go and save 1. Under 'Where to send alerts', switch 'Email alerts' to On. Alerts go to your account email address. 2. To send changes to another app or an AI agent, paste a URL into 'Webhook URL'. 3. Check the 'Monitor summary', then click 'Save monitor'. The monitor starts watching when you click 'Save monitor'. If you leave the page before saving, the setup is discarded. That's it - your monitor is live. From now on, Simplescraper watches the page for you and lets you know as soon as a check finds a change. ## FAQ ### How much does website monitoring cost? Each check uses 2 credits. ### How do I check a page outside the schedule? Click 'Check now' on the monitor's page. ## Related - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping) - schedule a recipe when you need the page's data as rows - [Credits](https://simplescraper.io/docs/credits) - how credits are counted across Simplescraper For questions or help, reach out via chat. ## Extract API v1 https://simplescraper.io/docs/api-guide # Using the Simplescraper API The Simplescraper API allows you to extract structured data programmatically from web pages, as well as create and manage your scrape recipes. This guide covers how to effectively use the API for data extraction and recipe management. Tip: If you're just getting started with Simplescraper, our [FAQ](https://simplescraper.io/docs/) explains the concept of scrape recipes. ## Authentication All requests to the API require the inclusion of an API key. The API key should be sent in the `Authorization` header using the `Bearer` token format: ``` Authorization: Bearer your_api_key ``` **An API key is provided when you sign up to Simplescraper and can be found on your [Account Page](https://simplescraper.io/docs/managing-subscription) and on the API tab of each recipe that you create.** Code examples of how to include the API key in requests are provided in the sections below. ## Request structure #### POST and PUT Request structure For POST and PUT requests, the API key should be sent in the `Authorization` header using the `Bearer` token format and the Content-Type header must be set to application/json. Here's an example of a POST request to the `/recipes/:recipeId/run` endpoint: ``` const apikey = 'ap1k3y'; const recipeId = '12345'; const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run`; const requestBody = { sourceUrl: sourceUrl, // other optional properties can be included here extractMarkdown: false, runAsync: false, }; const response = fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }) // handle response... ``` Notes: - URL encoding of the sourceUrl is not necessary when sent in the request body, as JSON handles special characters automatically. - Set the Content-Type header to application/json for POST requests. #### GET Request structure For GET requests, the API key should be sent in the `Authorization` header using the `Bearer` token format. Other properties should also be passed in as query parameters, and encoded where necessary. As an example, here's a GET request to the `/results/:resultsID` endpoint: ``` const apikey = 'ap1k3y'; const resultsId = '12345'; const requestUrl = `https://api.simplescraper.io/v1/results/${resultsId}`; const response = fetch(requestUrl, { headers: { 'Authorization': `Bearer ${apikey}` } }); // handle response... ``` ## API Endpoints The base URL for all API requests is: [https://api.simplescraper.io/v1](https://api.simplescraper.io/v1), followed by the specific endpoint. Below is a table of all available endpoints. To use them, append these endpoints to the base URL and be sure to use the correct HTTP method (GET, POST, PUT, etc.) as shown for each endpoint. | endpoint | Method | description | | --- | --- | --- | | [/recipes/:recipeId/run](#post-recipe-recipeid-run) | POST | Run the specified scrape recipe and return results or status | | [/recipes](#post-recipes) | POST | Create a new recipe using provided details | | [/recipes](#get-recipes) | GET | List and search scrape recipes | | [/recipes/:recipeId](#get-recipes-recipeid) | GET | Get information about a recipe | | [/recipes/:recipeId](#put-recipes-recipeid) | PUT | Update an existing recipe with new details | | [/recipes/:recipeId/results-latest](#get-recipes-recipeid-results-latest) | GET | Retrieve the most recent result for a specific recipe | | [/results/:resultsId](#get-results-resultsid) | GET | View results or scrape progress for specified results ID. Status key indicates the progress of scrape task | | [/recipes/:recipeId/results-history](#get-recipes-recipeid-results-history) | GET | Retrieve results ids, completion dates, and pages scraped for a recipe's runs, newest first, 50 per page by default | | [/recipes/:recipeId/batch/urls](#post-recipes-recipeid-batch-urls) | POST | Update or replace batch scraper (crawler) URL list | | [/extract](#post-extract) | POST | Extract markdown, HTML, screenshots, or structured data from a single URL | | [/smart-extract](#post-smart-extract) | POST | Extract data and reusable CSS selectors from any website using AI | | /smart-extract/:extractId | GET | Poll status and retrieve results of a smart extract task | | /extract-urls | POST | Discover and extract all URLs from a website's sitemap. See [URL Extractor API](https://simplescraper.io/docs/extract-urls-api) | | /search | POST, GET | Search Google and return up to 100 ranked results plus the answer box, knowledge graph, 'People also ask' and related searches. See [Google Search API](https://simplescraper.io/docs/google-search-api) | For example, the full URL to run a scrape recipe looks like this: ``` https://api.simplescraper.io/v1/recipes/:recipeId/run ``` Note: Replace `:recipeId` and `:resultsId` with actual IDs when making requests. Detailed information about interacting with each of the main endpoints is explained below. ### POST /recipes/:recipeId/run Full endpoint: `https://api.simplescraper.io/v1/recipes/:recipeId/run` Send POST requests to this endpoint to initiate a scrape run for a recipe and return data. If the scrape time exceeds 90 seconds, a JSON object containing a results\_id and status of 'running' is returned which can then be polled at [https://api.simplescraper.io/v1/results/:resultsId](https://api.simplescraper.io/v1/results/:resultsId). ##### Request body properties The following properties should be sent in the request body as JSON: | Property | Required | Type | Description | | --- | --- | --- | --- | | sourceUrl | No | URL | The URL of the page to be scraped. This will update the current URL of the recipe. If not included, the existing recipe URL will be used. | | runAsync | No | Boolean | If true, returns a result ID immediately and runs the scrape task asynchronously. The result ID can then be used to poll the /results/:resultsID endpoint. | | extractMarkdown | No | Boolean | If true, a markdown version of the page will be extracted in addition to structured data (see [extract-markdown](https://simplescraper.io/docs/extract-markdown)) | | cursor | No | String | UID of the last item from previous page (provided in `next_cursor` field of previous response). Use for paginating through results. | | useCrawler | No | Boolean | If true, scrape URLs that have been added to the crawler via the dashboard or `/recipes/:recipeId/batch/urls` endpoint, instead of the source URL. Default false. | | urls | No | Array of URLs | POST body only. Scrape multiple pages through this recipe in a single call. Used instead of sourceUrl. Replaces the recipe's crawler URL list and runs asynchronously - a `results_id` is returned to poll at `/results/:resultsId`. | ##### Example POST request ``` async function runRecipe(apikey, recipeId, sourceUrl) { const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run`; const requestBody = { sourceUrl: sourceUrl, runAsync: false }; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // call function runRecipe('YOUR_API_KEY', 'YOUR_RECIPE_ID', 'https://example.com'); ``` ##### Scraping multiple URLs in one request Pass a urls array to scrape many pages through a single recipe in one call. The run always returns a `results_id` immediately and scrapes in the background - poll `/results/:resultsId` for the data. Passing `urls` replaces the recipe's saved crawler list, so the run scrapes exactly the URLs you send. You can send up to 5000 URLs per call; the number actually scraped is capped by your available credits (roughly 2 credits per URL), so a run with 50 credits scrapes about 25 URLs even if you send more. ``` const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run`; const requestBody = { urls: [ 'https://example.com/page-1', 'https://example.com/page-2', 'https://example.com/page-3' ] }; const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const { results_id } = await response.json(); // then poll https://api.simplescraper.io/v1/results/{results_id} for the scraped data ``` ##### > ##### Notes: > > GET requests to `/recipe/:recipeId/run` are also supported for systems that can't make POST requests, however POST is recommended whenever possible. When sending a GET request, pass properties as URL parameters and ensure proper URL encoding of the `sourceUrl`. > > > > ##### Example GET request > > ``` > async function runRecipe(apiKey, recipeId, sourceUrl) { > const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run?sourceUrl=${encodeURIComponent(sourceUrl)}`; > > try { > const response = fetch(url, { > headers: { > 'Authorization': `Bearer ${apikey}` > } > }); > const data = await response.json(); > console.log(data); > } catch (error) { > console.error('error:', error); > } > } > > // call function > runRecipe('YOUR_API_KEY', 'YOUR_RECIPE_ID', 'https://example.com'); > ``` ##### Response structure | Property | Example Value | Explanation | Options/Types | | --- | --- | --- | --- | | success | true | Whether the request was successful | Boolean | | name | "Example recipe" | Name of the recipe | String | | results\_id | "pAioZevQJaqpjod4EQkd" | Unique identifier for the results of the scrape task | String | | url | "[https://example.com](https://example.com)" | source URL of the recipe | String (valid URL) | | date\_completed | "2024-08-22T09:41:00.000Z" | When the scrape completed | String (ISO 8601 date format) | | status | "completed" | Current status of the scrape job | String: "completed", "failed", "running" | | data | \[...\] | Main payload of scraped data | Array of objects (structure depends on scrape target) | | screenshots | \[{ "url\_uid": 1, "url": "https://...", "screenshot": "https://" }\] | Screenshots of each page scraped | Array of objects | | errors | \[ { url: '', page\_message: 'cannot find element', response\_code: 200, screenshot: '' } \] | Error details for pages that did not return data successfully | Array of objects | ##### Example response (no timeout) ``` { "success": true, "name": "Example scrape recipe", "results_id": "pAioZevQJaqpjod4EQkd", "url": "https://example.com", "date_completed": "2024-08-22T09:41:00.000Z", "status": "completed", "data": [...], "screenshots": [ { "url_uid": 1, "url": "https://...", "screenshot": "https://..." } ], "errors": [ { "url_uid": 1, "url": "https://...", "page_message": "", "response_code": "", "screenshot": "https://..." } ] } ``` ##### Example response (timeout) ``` { "status": "running", "results_id": "r4t9iyofr234rtr9j", "message": "The task is still running. Please check status at https://api.simplescraper.io/v1/results/r4t9iyofr234rtr9j or await webhook notification if configured." } ``` * * * ### POST /recipes Full endpoint: `https://api.simplescraper.io/v1/recipes` Sending a POST request to this endpoint creates a new scrape recipe with the specified name, URL, and selectors. The created recipe's ID and details are returned. ##### Request body properties The following properties should be sent in the request body as JSON: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | Yes | String | The name of the recipe. | | url | Yes | URL | The URL of the page to be scraped. Must be a valid URL. | | selectors | Yes | Array | Array of selector objects that define what data to extract. Maximum 100 selectors. | Each selector object in the `selectors` array should have the following properties: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | Yes | String | Name of the data point to extract. | | selector | Yes | String | CSS selector string targeting the element to scrape. | | uid | No | String | Unique identifier for the selector. If not provided, one will be generated automatically. | ##### Example request ``` async function createRecipe(apiKey, name, url, selectors) { const requestUrl = 'https://api.simplescraper.io/v1/recipes'; try { const response = await fetch(requestUrl, { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ name: name, url: url, selectors: selectors }) }); const data = await response.json(); return data; } catch (error) { console.error('Error creating recipe:', error); } } // example const apiKey = 'YOUR_API_KEY'; const name = 'Product Prices Scraper'; const url = 'https://example.com/products'; const selectors = [ { name: 'Product Name', selector: '.product-title' }, { name: 'Price', selector: '.product-price' } ]; createRecipe(apiKey, name, url, selectors); ``` ##### Response structure | Property | Example Value | Explanation | | --- | --- | --- | | success | true | Whether the recipe was created successfully | | status | "completed" | Status of the operation | | recipe\_id | "rtJjthGverod4EQkt4t4d" | ID of the newly created recipe | | data | | Created recipe data | ##### Example response ``` { "success": true, "status": "completed", "recipe_id": "rtJjthGverod4EQkt4t4d", "data": { "recipe_id": "rtJjthGverod4EQkt4t4d", "name": "Product Prices Scraper", "url": "https://example.com/products", "date_created": "2025-05-12T09:41:00.000Z", "selectors": [ { "name": "Product Name", "selector": ".product-title", "uid": "a4b2-c3d9-e7f1-g5h6" }, { "name": "Price", "selector": ".product-price", "uid": "j8k7-l9m4-n2p5-q3r8" } ] } } ``` * * * ### GET /recipes Full endpoint: `https://api.simplescraper.io/v1/recipes` List and search scrape recipes with optional filtering and sorting. ##### Query Parameters | Parameter | Required | Type | Description | | --- | --- | --- | --- | | q | No | String | Search term to filter recipes by name or URL | | host | No | String | Filter recipes by specific host (e.g., "x.com") | | sort | No | String | Field to sort by (date\_created, date\_last\_ran, name). Default: date\_created | | order | No | String | Sort order (asc or desc). Default: desc | | limit | No | Number | Maximum number of recipes to return. Default: 50 | | offset | No | Number | Number of recipes to skip for pagination. Default: 0 | ##### Example request ``` async function listRecipes(apiKey, params = {}) { const queryString = new URLSearchParams(params).toString(); const requestUrl = `https://api.simplescraper.io/v1/recipes?${queryString}`; try { const response = await fetch(requestUrl, { method: 'GET', headers: { 'Authorization': `Bearer ${apiKey}` } }); const data = await response.json(); return data; } catch (error) { console.error('Error listing recipes:', error); } } // Usage examples // List all recipes listRecipes('YOUR_API_KEY'); // Search for Amazon recipes listRecipes('YOUR_API_KEY', { q: 'amazon' }); // Find recipes from a specific host, sorted by last run date listRecipes('YOUR_API_KEY', { host: 'amazon.com', sort: 'date_last_ran', order: 'desc' }); // Pagination example listRecipes('YOUR_API_KEY', { limit: 10, offset: 20 }); ``` ##### Response structure | Property | Description | | --- | --- | | success | Whether the request was successful | | status | Status of the operation | | total | Total number of recipes matching the query | | count | Number of recipes returned in this response | | offset | Current pagination offset | | limit | Current pagination limit | | data | Array of recipe objects | ##### Example response ``` { "success": true, "status": "completed", "total": 157, "count": 2, "offset": 0, "limit": 2, "data": [ { "recipe_id": "rtJjthGverod4EQkt4t4d", "name": "Twitter Profile Scraper", "url": "https://twitter.com/elonmusk", "host": "twitter.com", "date_created": "2025-05-12T00:41:00.000Z", "date_last_ran": "2025-05-14T00:22:45.000Z", }, { "recipe_id": "aB3cD4eFgHiJkLmNoPqR", "name": "Twitter Search Results", "url": "https://twitter.com/search?q=web%20scraping", "host": "twitter.com", "date_created": "2025-05-10T00:00:10.000Z", "date_last_ran": null, } ] } ``` * * * ### GET /recipes/:recipeId Full endpoint: `https://api.simplescraper.io/v1/recipes/:recipeId` Get detailed information about a specific scrape recipe. **Query Parameters** | Parameter | Required | Type | Description | | --- | --- | --- | --- | | recipeId | Yes | String | The ID of the recipe to fetch | **Example request** ``` async function getRecipe(apiKey, recipeId) { const requestUrl = `https://api.simplescraper.io/v1/recipes/${recipeId}`; try { const response = await fetch(requestUrl, { method: 'GET', headers: { 'Authorization': `Bearer ${apiKey}` } }); const data = await response.json(); return data; } catch (error) { console.error('Error fetching recipe:', error); } } // Usage example getRecipe('YOUR_API_KEY', 'rtJjthGverod4EQkt4t4d'); ``` **Response structure** | Property | Description | | --- | --- | | success | Whether the request was successful | | data | Recipe object with complete details | **Example response** ``` { "success": true, "data": { "recipe_id": "rtJjthGverod4EQkt4t4d", "name": "Twitter Profile Scraper", "url": "https://twitter.com/elonmusk", "host": "twitter.com", "date_created": "2025-05-12T09:41:00.000Z", "date_last_ran": "2025-05-14T01:22:45.000Z", "selectors": [ { "name": "Username", "selector": "div[data-testid='UserName'] span", "uid": "a4b2-c3d9-e7f1-g5h6" }, { "name": "Bio", "selector": "div[data-testid='UserDescription']", "uid": "j8k7-l9m4-n2p5-q3r8" }, ] } } ``` * * * ### PUT /recipes/:recipeId Full endpoint: `https://api.simplescraper.io/v1/recipes/:recipeId` Updates an existing recipe with new details. This endpoint can be used to update the recipe name and/or selectors. ##### Request body properties The following properties should be sent in the request body as JSON: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | No | String | The updated name of the recipe. | | url | No | String | The updated URL of the page to be scraped. | | selectors | No | Array | Array of CSS selector objects that define what data to extract. | | update\_mode | No | String | Mode for updating selectors: `replace` (default) or `merge`. When set to `merge`, existing selectors with the same name will be updated, and new ones will be added. | Each selector object in the `selectors` array should have the following properties: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | Yes | String | Name of the data point to extract. | | selector | Yes | String | CSS selector string targeting the element to scrape. | | uid | No | String | Unique identifier for the selector. If not provided for a new selector, one will be generated automatically. | ##### Example request ``` async function updateRecipe(apiKey, recipeId, updates, updateMode = 'replace') { const requestUrl = `https://api.simplescraper.io/v1/recipes/${recipeId}`; try { const response = await fetch(requestUrl, { method: 'PUT', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ ...updates, update_mode: updateMode }) }); const data = await response.json(); return data; } catch (error) { console.error('Error updating recipe:', error); } } // update name only updateRecipe( 'YOUR_API_KEY', 'rtJjthGverod4EQkt4t4d', { name: 'Updated Product Scraper' } ); // update selectors with merge mode updateRecipe( 'YOUR_API_KEY', 'rtJjthGverod4EQkt4t4d', { selectors: [ { name: 'Product Name', selector: '.new-product-title-class' }, { name: 'New Field', selector: '.new-field-class' } ] }, 'merge' ); ``` ##### Response structure | Property | Example Value | Explanation | | --- | --- | --- | | success | true | Whether the recipe was updated successfully | | status | "completed" | Status of the operation | | recipe\_id | "rtJjthGverod4EQkt4t4d" | ID of the updated recipe | | data | | Updated recipe data | ##### Example response ``` { "success": true, "status": "completed", "recipe_id": "rtJjthGverod4EQkt4t4d", "data": { "recipe_id": "rtJjthGverod4EQkt4t4d", "name": "Updated Product Scraper", "url": "https://example.com/products", "selectors": [ { "name": "Product Name", "selector": ".new-product-title-class", "uid": "a4b2-c3d9-e7f1-g5h6" }, { "name": "Price", "selector": ".product-price", "uid": "j8k7-l9m4-n2p5-q3r8" }, { "name": "New Field", "selector": ".new-field-class", "uid": "c5d4-e3f2-g1h0-i9j8" } ] } } ``` * * * ### GET /recipes/:recipeId/results-latest Full endpoint: `https://api.simplescraper.io/v1/recipes/:recipeId/results-latest` Retrieve the most recent scrape results for a specific recipe. **Path Parameters** | Parameter | Required | Type | Description | | --- | --- | --- | --- | | recipeId | Yes | String | The ID of the recipe to get results for | **Query Parameters** | Parameter | Required | Type | Description | | --- | --- | --- | --- | | limit | No | Number | Maximum number of results to return. Default: 2000 | | cursor | No | String | UID of the last item from previous page. Use for reliable pagination. | | orderBy | No | String | Field to order results by ('index' or 'timestamp'). Default: 'index' | | offset | No | Number | **Deprecated** - use `cursor` instead. | **Example request** ``` async function getRecipeLatestResults(apiKey, recipeId, params = {}) { const queryString = new URLSearchParams(params).toString(); const requestUrl = `https://api.simplescraper.io/v1/recipes/${recipeId}/results-latest${queryString ? '?' + queryString : ''}`; try { const response = await fetch(requestUrl, { method: 'GET', headers: { 'Authorization': `Bearer ${apiKey}` } }); const data = await response.json(); return data; } catch (error) { console.error('Error fetching latest results:', error); } } // usage examples // get all latest results for a recipe getRecipeLatestResults('YOUR_API_KEY', 'rtJjthGverod4EQkt4t4d'); // order by timestamp instead of index getRecipeLatestResults('YOUR_API_KEY', 'rtJjthGverod4EQkt4t4d', { orderBy: 'timestamp' }); ``` **Response structure** | Property | Description | | --- | --- | | success | Whether the request was successful | | results\_id | ID of the results set | | status | Status of the results set | | date\_completed | When the scrape was completed | | data | Array of scraped data items | | next\_cursor | UID of the last item. Use as `cursor` param to fetch next page. Null when no more results. | | markdown | Optional markdown representation (if available) | **Example response** ``` { "success": true, "results_id": "pAioZevQJaqpjod4EQkd", "date_completed": "2025-05-14T01:22:45.000Z", "data": [ { "uid": "doc1", "Username": "elonmusk", "Bio": "Mars & Cars", "Follower Count": "158.7M", "timestamp": 1692691000000 }, { "uid": "doc2", "Username": "x", "Bio": "Everything app", "Follower Count": "27.4M", "timestamp": 1692691000000 } ], "next_cursor": "doc2" } ``` ##### Pagination with cursor Use `cursor` to paginate through large result sets. The `next_cursor` in each response provides the value to use for the next page. **First page** (no cursor): ``` GET /recipes/:recipeId/results-latest?limit=10&orderBy=timestamp ``` **Second page** (use next\_cursor from previous response): ``` GET /recipes/:recipeId/results-latest?limit=10&orderBy=timestamp&cursor=doc10 ``` **Final page** (next\_cursor is null): ``` { "success": true, "data": [...], "next_cursor": null // No more results } ``` Note: Use the same `orderBy` value for all pages of a paginated request. * * * ### GET /recipes/:recipeId/results-history Full endpoint:: `https://api.simplescraper.io/v1/recipes/:recipeId/results-history` Get a summary of a recipe's scrape runs, newest first. Each entry contains the run's `results_id`, which can be passed to `GET /results/:resultsId` to fetch that run's full data. A request with no parameters returns the 50 most recent runs; page with `limit` and `cursor` to reach every stored run. ##### Query parameters | Parameter | Description | | --- | --- | | limit | Runs per page, from 1 to 200. Default 50 | | cursor | The `results_id` of the last entry on the previous page. The next page starts after it | A page with fewer entries than `limit` is the last page. A `cursor` that no longer matches a stored run returns `404`. ``` async function getAllRuns(apiKey, recipeId) { const runs = []; let cursor = null; for (;;) { const params = new URLSearchParams({ limit: '200' }); if (cursor) params.set('cursor', cursor); const response = await fetch(`https://api.simplescraper.io/v1/recipes/${recipeId}/results-history?${params}`, { headers: { 'Authorization': `Bearer ${apiKey}` } }); const page = await response.json(); runs.push(...page); if (page.length < 200) return runs; cursor = page[page.length - 1].results_id; } } ``` ##### Example request ``` async function getResultsHistory(apiKey, recipeId) { const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/results-history`; try { const response = await fetch(url, { headers: { 'Authorization': `Bearer ${apiKey}` } }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // usage getResultsHistory('YOUR_API_KEY', 'YOUR_RECIPE_ID'); ``` ##### Response structure The response is a JSON array (not wrapped in an object). Each entry has: | Property | Description | | --- | --- | | results\_id | ID of the run's results set. Use with `GET /results/:resultsId` | | date\_completed | When the run completed (ISO 8601) | | status | Status of the run | | pages\_scraped | Number of pages scraped in the run | ##### Example response ``` [ { "results_id": "pAioZevQJaqpjod4EQkd", "date_completed": "2025-05-14T01:22:45.000Z", "status": "completed", "pages_scraped": 3 }, { "results_id": "q9RtYuvBNwqpjod4EQkf", "date_completed": "2025-05-13T22:10:02.000Z", "status": "completed", "pages_scraped": 1 } ] ``` A recipe with no completed runs returns an empty array. * * * ### GET /results/:resultsID Full endpoint:: `https://api.simplescraper.io/v1/results/:resultsId` Get results for a particular scrape task based on the result ID. Check status property for value of `completed`to determine if task has finished. status of `running` indicates the task is still in progress. ##### Query Parameters | Parameter | Required | Description | | --- | --- | --- | | apikey | Yes | The API key for user authentication. | | limit | No | The maximum number of results to return. | | cursor | No | UID of the last item from previous page. Use for reliable pagination. | | orderBy | No | Field to order scrape results by ('index' or 'timestamp'). Default: 'index' | | offset | No | **Deprecated** - use `cursor` instead. | ##### Example request ``` async function getResults(apiKey, resultsId) { const url = `https://api.simplescraper.io/v1/results/${resultsId}`; try { const response = fetch(url, { headers: { 'Authorization': `Bearer ${apikey}` } }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // Usage getResults('YOUR_API_KEY', 'YOUR_RESULTS_ID'); ``` ##### Example response ``` { "results_id": "pAioZevQJaqpjod4EQkd", "date_completed": "2024-08-22T09:41:00.000Z", "status": "completed", "data": [...], "screenshots": [ { "url_uid": 1, "url": "https://...", "screenshot": "https://..." } ], "errors": [...], "next_cursor": "doc10" } ``` The `data`, `screenshots`, and `errors` structures match the `/recipes/:recipeId/run` response. Unlike that endpoint, this response does not include the recipe `name` or `url`. `next_cursor` is present only when more pages are available. * * * ### POST /recipes/:recipeId/batch/urls Full endpoint:: `https://api.simplescraper.io/v1/recipes/:recipeId/batch/urls` Update or replace the batch scraper (crawler) URLs. The endpoint allows adding new URLs or replacing existing ones in the batch collection. Use the `batch_mode` parameter to specify whether to `append` or `replace` URLs. The endpoint returns the result of the operation, including success status, a summary of new and existing URLs, and any invalid URLs with detailed error information. Note that by default, when running a recipe, the Simplescraper API does not use the batch scraper (crawler) unless the `useCrawler` flag is specified. See ['/recipes/:recipeId/run'](#post-recipe-recipeid-run) endpoint section for more details. ##### Parameters | Parameter | Required | Description | | --- | --- | --- | | apikey | Yes | The API key for user authentication. | | batch\_mode | No | Operation mode, either `append` (default) or `replace`. | | batch\_urls | Yes | An array of URLs to be processed in the batch operation. | ##### Example request ``` async function addBatchUrls(apiKey, recipeId, batchMode, urls) { const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/batch/urls`; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ batch_mode: batchMode, batch_urls: urls }) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // Usage addBatchUrls('YOUR_API_KEY', 'YOUR_RECIPE_ID', 'append', ['https://example.com/page1', 'https://example.com/page2']); ``` ##### Example response ``` { "success": true, // true, false 'partial' "summary": { "totalExisting": 150, "totalNew": 30, "totalErrors": 2 }, "data": { "newUrls": [ "https://example.com/page1", "https://example.com/page2" ], "errorDetails": [ { "url": "https://example.com/badurl", "message": "Invalid URL format", "type": "INVALID_URL_FORMAT" }, { "url": "https://example.com/anotherbadurl", "message": "Invalid URL format", "type": "INVALID_URL_FORMAT" } ] } } ``` ##### Notes - **batch\_mode**: - `append` mode will add new URLs to the existing list, with a cap of 5000 URLs. - `replace` mode will clear existing URLs and replace them with the provided list, enforcing the same limit. - **Validation**: URLs are validated for format, length, protocol, and TLD correctness. Invalid URLs are listed in the response's `errorDetails` array. - **Limit**: A maximum of 5000 URLs can be stored in a batch at any time. Attempting to exceed this limit will result in trimming of the excess URLs. * * * ### POST /extract Full endpoint: `https://api.simplescraper.io/v1/extract` Extract markdown, HTML, screenshots, or structured data from a single URL. This is a synchronous endpoint - results are returned immediately with no job queue or polling required. Specify what you want back using the boolean flags below. If no extraction flags are set, only page metadata (title, description) is returned. ##### Request body properties The following properties should be sent in the request body as JSON: | Property | Required | Type | Description | | --- | --- | --- | --- | | url | Yes | URL | The URL of the page to extract data from. | | markdown | No | Boolean | If true, extract a markdown version of the page content. Default false. | | html | No | Boolean | If true, include the raw HTML of the page. Default false. | | screenshot | No | Boolean | If true, take a screenshot of the page. Default false. | | selectors | No | Array | Array of selector objects defining what structured data to extract. Maximum 100 selectors. | | recipeId | No | String | ID of an existing recipe - loads selectors from that recipe. If both `recipeId` and `selectors` are provided, `selectors` takes precedence. | | proxy | No | String | Proxy type: `"standard"` (default), `"premium"`, `"residential"`, or `"none"`. | | timeout | No | Number | Maximum time in seconds to wait for the page. Default 30. | | cookies | No | Array | Array of cookie objects to set before loading the page. | | credentialId | No | String | ID of saved login credentials to use. | | waitForSelector | No | String | CSS selector to wait for before extracting. | Combining flags: `markdown`, `html`, and `screenshot` are independent flags - set any combination in a single request. For example, `{ "markdown": true, "html": true, "screenshot": true }` returns all three fields in the same response alongside any structured `data` from selectors. Each selector object in the `selectors` array should have the following properties: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | Yes | String | Name of the data point to extract. | | selector | Yes | String | CSS selector string targeting the element to scrape. | Each cookie object should have the following properties: | Property | Required | Type | Description | | --- | --- | --- | --- | | name | Yes | String | Cookie name. | | value | Yes | String | Cookie value. | | domain | Yes | String | Domain the cookie applies to. | ##### Example POST request ``` async function extractPage(apiKey, targetUrl) { const url = 'https://api.simplescraper.io/v1/extract'; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ url: targetUrl, markdown: true, // include markdown of page content screenshot: true, // take a screenshot selectors: [ // extract structured data { name: 'title', selector: 'h1' }, { name: 'price', selector: '.price' } ] }) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // call function extractPage('YOUR_API_KEY', 'https://example.com'); ``` ##### Example POST request (selectors from recipe) ``` async function extractFromRecipe(apiKey, targetUrl, recipeId) { const url = 'https://api.simplescraper.io/v1/extract'; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ url: targetUrl, recipeId: recipeId }) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // call function extractFromRecipe('YOUR_API_KEY', 'https://example.com/product', 'YOUR_RECIPE_ID'); ``` ##### > ##### Notes: > > > > GET requests to `/extract` are also supported. When sending a GET request, pass properties as URL query parameters. > > > > ##### Example GET request > > ``` > async function extractViaGet(apiKey, targetUrl) { > const params = new URLSearchParams({ > url: targetUrl, > markdown: 'true' > }); > const requestUrl = `https://api.simplescraper.io/v1/extract?${params}`; > > try { > const response = await fetch(requestUrl, { > headers: { > 'Authorization': `Bearer ${apiKey}` > } > }); > const data = await response.json(); > console.log(data); > } catch (error) { > console.error('error:', error); > } > } > > // call function > extractViaGet('YOUR_API_KEY', 'https://example.com'); > ``` ##### Response structure The following fields are always present in the response: | Property | Example Value | Explanation | Options/Types | | --- | --- | --- | --- | | id | "zO7AHflCUqoXuarfj3Hz" | Unique identifier for this extraction | String | | url | "[https://example.com](https://example.com)" | The URL that was extracted | String (valid URL) | | status | "completed" | Status of the extraction | String: "completed" | | date\_scraped | "2026-02-01T10:30:00.000Z" | When the extraction was performed | String (ISO 8601 date format) | | metadata | | Page metadata. Includes `word_count` when markdown is requested. | Object | The following fields are only present when the corresponding flag is set: | Property | Condition | Explanation | | --- | --- | --- | | markdown | `markdown: true` | Markdown representation of the page content | | html | `html: true` | Raw HTML of the page | | data | selectors provided | Array of extracted data objects | | screenshots | `screenshot: true` | Array of screenshot objects with `url` and `screenshot` URL | ##### Example response ``` { "id": "zO7AHflCUqoXuarfj3Hz", "url": "https://example.com", "status": "completed", "date_scraped": "2026-02-01T10:30:00.000Z", "metadata": { "title": "Example Domain", "description": null, "word_count": 42 // present when markdown: true }, "markdown": "# Example Domain\n\n...", // present when markdown: true "data": [ // present when selectors provided { "title": "Example Product", "price": "$9.99" } ], "screenshots": [ // present when screenshot: true { "url": "https://example.com", "screenshot": "https://storage.googleapis.com/..." } ] } ``` * * * ### POST /smart-extract Full endpoint: `https://api.simplescraper.io/v1/smart-extract` Smart Extract uses AI to accurately extract data and generate reusable CSS selectors from any website using only a list of the data points you need (data schema). Read more about this feature [here](https://simplescraper.io/docs/smart-data-extract). ##### Request body properties | Property | Required | Description | | --- | --- | --- | | url | Yes | The URL of the page to be scraped. | | schema | Yes | A comma-separated list of properties to extract. | | recipeId | No | ID of scrape recipe - if included will use cookies, URL and schema (property names) from this recipe | ##### Example request ``` async function runSmartExtract(apikey, url, schema) { const endpoint = 'https://api.simplescraper.io/v1/smart-extract'; const requestBody = { url: url, schema: schema }; try { const response = await fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${apikey}` }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } // usage let url = 'https://example.com/product-page'; let schema = 'product name, price, description, rating'; runSmartExtract('YOUR_API_KEY',url,schema); ``` ##### Response structure | Property | Example Value | Explanation | | --- | --- | --- | | url | "[https://example.com/product-page](https://example.com/product-page)" | The URL that was extracted | | extract\_uid | "12345" | Unique identifier for this smart extraction | | results\_uid | "54321" | Unique identifier for the results set | | date\_completed | "2024-01-01T00:00:00.000Z" | When the extraction completed (ISO 8601) | | data | \[...\] | Array of extracted data objects matching the schema | | selectors | \[...\] | Array of AI-generated CSS selector objects with name, selector, uid | | screenshots | \[...\] | Array of screenshot objects captured during extraction | | status | "completed" | Status of the extraction | If the extraction takes longer than 2 minutes, the response returns early with a `status` of `"running"` and an `id` that can be polled at `GET /smart-extract/:extractId` until the status reaches `"success"` or `"error"`. ##### Example response ``` { "url": "https://example.com/product-page", "extract_uid": "12345", "results_uid": "54321", "date_completed": "2024-01-01T00:00:00.000Z", "data": [ { "product_name": "Amazon apples", "price": "USD $1,400", "description": "A great selection of Gala, Fuji, Honeycrisp, Golden Delicious & more", "rating": "4.5" } ], "selectors": [ { "name": "product_name", "selector": "div.displayProduct", "uid": "e5de-60cd-4386-917b" }, { "name": "price", "selector": "div.displayListingPrice", "uid": "ea74-d519-4508-b5ca" }, { "name": "description", "selector": "div.description", "uid": "4cc4-d406-4dd1-887f" }, { "name": "rating", "selector": "div.feature-item:nth-child(2)", "uid": "c720-a90e-44dd-9eb4" } ], "screenshots": [ { "url": "https://example.com/product-page", "screenshot": "https://storage.googleapis.com/..." } ], "status": "completed" } ``` ##### Example response (timeout - still running) ``` { "id": "54321", "extract_uid": "54321", "status": "running", "message": "The task is still running. Poll GET https://api.simplescraper.io/v1/smart-extract/54321 to fetch the completed result." } ``` ##### Polling for smart extract results If the POST response returns `status: "running"`, poll the extraction status using the `extract_uid`: ``` GET https://api.simplescraper.io/v1/smart-extract/:extractId ``` The response structure matches the POST response above, except it does not include `results_uid`. Check the `status` field - `"success"` or `"completed"` means the extraction is done, `"running"` means it is still processing, and `"error"` indicates a failure. ## Request timeouts Requests to the `/recipes/:recipeId/run` endpoint will timeout after 90 seconds and a JSON object containing a `results_id` and status of `'running'` will be returned. The `results_id` can then be used to poll the `/results/:resultsId` endpoint for a status of 'completed'. In general, most scrape requests are completed within 90 seconds. ## Error Handling The API uses standard HTTP response codes to indicate the success or failure of an API request. In general: - Codes in the `2xx` range indicate success - Codes in the `4xx` range indicate an error that resulted from the provided information or the account (eg, a required parameter was missing, insufficient credits etc.) - Codes in the `5xx` range indicate an error with our servers In addition to the HTTP status code, all error responses include a JSON object in the response body with an `error` key containing a human-readable error message. ### Error Response Format All error responses have the following structure: ``` { "error": { "type": "api-key-not-included", "message": "API key was not included in the request." } } ``` One exception: `/extract-urls` returns errors as `{ "success": false, "error": "" }` with the message as a plain string, and uses HTTP 401 for an invalid API key. ### Error Codes | Error Type | HTTP Status Code | Error Message | | --- | --- | --- | | **Successful Call** | **200** | | | api-key-not-included | 403 | API key was not included in the request. | | out-of-credits | 402 | Credits expired. | | out-of-api-reads | 402 | API reads expired. | | results-not-found | 404 | Results not found. Ensure the results ID is correct and the recipe was run. | | recipe-not-found | 404 | The recipe was not found. Please make sure it exists and that the recipe ID is correct. | | user-not-found | 404 | User not found. | | request-timeout | 408 | The request timed out after 5 minutes. | | invalid-request | 400 | Invalid request format. | | invalid-value | 400 | Invalid value provided. | | invalid-url-format | 400 | The provided URL format is invalid. | | schema-and-url-required | 400 | Either a recipeId or both schema and url must be provided. | | rate-limit-exceeded | 429 | You have exceeded the rate limit. | | method-not-allowed | 405 | This method is not allowed for this endpoint. | | extraction-failed | 500 | The page could not be scraped. | | extraction-timeout | 408 | The extraction timed out. | | default | 500 | An unexpected error occurred. | For persistent errors or issues not covered here, please contact customer support for assistance. ## Code examples #### Calling multiple URLs and handling timeout/async ``` // function to initiate scraping for a single URL async function runScrapeForUrl(apiKey, recipeId, sourceUrl) { const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run`; const requestBody = { apikey: apiKey, sourceUrl: sourceUrl, runAsync: false }; try { const response = await fetch(url, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${apikey}` }, body: JSON.stringify(requestBody) }); const data = await response.json(); if (data.status === 'running') { return { status: 'running', resultsId: data.results_id }; } else if (data.status === 'error') { return { status: 'error', error: data.error }; } else { return data; } } catch (error) { return { status: 'error', error: error.message }; } } // main function to process multiple URLs async function main() { const apikey = 'your-api-key'; const recipeId = 'your-recipe-id'; const urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ]; for (const url of urls) { let result = await runScrapeForUrl(apikey, recipeId, url); // if scrape is running asynchronously, poll for results if (result.status === 'running') { result = await pollForResults(result.resultsId, apiKey); // example covered later } else if (result.status === 'error') { console.error(`Error scraping ${url}:`, result.error); } else if (result.status === 'completed') { console.log(`Successfully scraped ${url}`); // process successful result here } } } main(); ``` #### Polling for Results To handle tasks that are still processing, implement a polling mechanism on the client side. Use a sensible interval of a few seconds to avoid overloading the endpoint. ``` // calling the v1/results/:resultsId endpoint async function pollForResults(resultsId, apikey, maxAttempts = 10) { const url = `https://api.simplescraper.io/v1/results/${resultsId}`; for (let attempt = 0; attempt < maxAttempts; attempt++) { try { const response = await fetch(url, { headers: { 'Authorization': `Bearer ${apikey}` } }); const data = await response.json(); if (data.status === 'completed') { console.log('Scraping completed successfully:', data); return data; } console.log('Job still processing, retrying in 5 seconds...'); await new Promise(resolve => setTimeout(resolve, 5000)); } catch (error) { console.error('Error polling for result:', error); } } console.error('Max polling attempts reached. Please check the job status manually.'); return null; } // call function async function main() { const result = await pollForResults('your-results-id', 'your-api-key'); if (result) { console.log('Final result:', result); } else { console.log('Failed to retrieve results'); } } main(); ``` #### Handling Errors When working with our API, we recommend checking both the HTTP status code and the presence of an `error` key in the response body. Here's an example of how you might handle errors in your code: ``` async function makeApiRequest(apikey, endpoint) { try { const response = fetch('https://api.simplescraper.io/v1/${endpoint}', { headers: { 'Authorization': `Bearer ${apikey}` } }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.error || `HTTP error status: ${response.status}`); } const data = await response.json(); // process successful response } catch (error) { console.error('There was an error:', error.message); // handle the error } } // call function async function main() { try { const data = await makeSimplescraperRequest('your-api-key', 'recipes/123456/run'); console.log('API response:', data); } catch (error) { console.error('Error in main:', error.message); } } main(); ``` ## Screenshot API https://simplescraper.io/docs/screenshot-api # Screenshot API The Screenshot API captures pixel-perfect screenshots of any web page and returns the image as base64, a hosted URL, or raw binary. For structured data extraction that also includes a screenshot, see the [/extract endpoint](https://simplescraper.io/docs/api-guide#post-extract) instead. The Screenshot API is optimized specifically for image capture with full control over format, viewport, and rendering options. ## Authentication All requests require an API key in the `Authorization` header: ``` Authorization: Bearer your_api_key ``` Your API key is available on your [Account Page](https://simplescraper.io/docs/managing-subscription) and on the API tab of each recipe. See the [main API authentication docs](https://simplescraper.io/docs/api-guide#authentication) for details. ## Endpoints The base endpoint is `https://api.simplescraper.io/v1/screenshot`. You can use either POST (recommended, with a JSON body) or GET (with query string parameters). Both accept the same parameters and return the same response. | Method | Usage | | --- | --- | | [POST /v1/screenshot](#post-v1-screenshot) | JSON body - handles special characters in URLs without manual encoding | | [GET /v1/screenshot](#get-v1-screenshot) | Query params - useful for simple integrations and browser testing | ### Response formats The `output` parameter controls how the screenshot is delivered. The response always includes metadata except when using `binary`. | Value | Default | Response type | `screenshot` field | Storage | | --- | --- | --- | --- | --- | | `base64` | Yes | JSON | Base64-encoded image string | Not stored (unless `store: true`) | | `url` | No | JSON | Public hosted URL | Stored automatically, 30-day TTL | | `binary` | No | Raw image bytes | N/A (no JSON, `Content-Type: image/{format}`) | Not stored (unless `store: true`) | When `output` is `base64` or `binary`, you can set `store: true` to also save the image to cloud storage. The hosted URL will appear in the JSON response as `screenshot_url` (for `base64`), or you can make a separate request with `output: "url"` to get one. ### POST /v1/screenshot Full endpoint: `https://api.simplescraper.io/v1/screenshot` Send a POST request with a JSON body to capture a screenshot. This is the recommended method - it handles special characters in URLs without manual encoding. ##### Request body properties | Property | Required | Type | Default | Description | | --- | --- | --- | --- | --- | | url | Yes | String | \- | Target URL to capture | | output | No | String | `base64` | How the image is delivered: `base64`, `url`, or `binary` | | store | No | Boolean | false | Save the image to cloud storage. Automatically `true` when `output` is `url`. | | format | No | String | `png` | Image format: `png`, `jpeg`, or `webp` | | quality | No | Number | 80 | JPEG/WebP compression quality, 1-100 (ignored for PNG) | | full\_page | No | Boolean | false | Capture the full scrollable page instead of just the viewport | | viewport\_width | No | Number | 1920 | Browser viewport width in pixels, 320-3840 | | viewport\_height | No | Number | 1080 | Browser viewport height in pixels, 320-3840 | | device\_scale\_factor | No | Number | 1 | Pixel density multiplier: `1`, `1.5`, `2`, or `3` | | delay | No | Number | 0 | Seconds to wait after page load before capture, 0-15 | | wait\_for\_selector | No | String | null | CSS selector to wait for before capturing | | wait\_until | No | String | `load` | Page load event to wait for: `load`, `domcontentloaded`, `networkidle0`, `networkidle2` | | hide\_popups | No | Boolean | true | Hide cookie consent popups before capture | | hide\_ads | No | Boolean | true | Hide ad network content during page load | | hide\_background | No | Boolean | false | Make the page background transparent (PNG and WebP only, ignored for JPEG) | ##### Example: base64 response (default) ``` async function captureScreenshot(apikey, targetUrl) { const url = 'https://api.simplescraper.io/v1/screenshot'; const requestBody = { url: targetUrl, format: 'png', viewport_width: 1280, viewport_height: 800, device_scale_factor: 2 }; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data.screenshot); // base64-encoded image } catch (error) { console.error('error:', error); } } captureScreenshot('YOUR_API_KEY', 'https://example.com'); ``` ##### Response ``` { "id": "ss_abc123def456", "url": "https://example.com", "status": "completed", "date_captured": "2026-02-25T12:00:00.000Z", "screenshot": "iVBORw0KGgoAAAANSUhEUgAA...", "metadata": { "title": "Example Domain", "image_size_bytes": 234567, "capped": false } } ``` ##### Example: hosted URL response ``` const requestBody = { url: 'https://example.com', output: 'url', format: 'png', viewport_width: 1280, viewport_height: 800 }; const response = await fetch('https://api.simplescraper.io/v1/screenshot', { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data.screenshot); // hosted URL for the screenshot ``` ##### Response ``` { "id": "ss_abc123def456", "url": "https://example.com", "status": "completed", "date_captured": "2026-02-25T12:00:00.000Z", "screenshot": "https://screenshots.simplescraper.io/ss_abc123def456.png", "metadata": { "title": "Example Domain", "image_size_bytes": 234567, "capped": false } } ``` Hosted images are kept for 30 days. Setting `output: "url"` automatically sets `store: true`. ##### Example: binary response ``` const requestBody = { url: 'https://example.com', output: 'binary', format: 'jpeg', quality: 90 }; const response = await fetch('https://api.simplescraper.io/v1/screenshot', { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); // Response is raw image bytes, not JSON const imageBuffer = await response.arrayBuffer(); ``` The response body is the raw image with `Content-Type: image/jpeg` (or `image/png`, `image/webp` depending on `format`). There is no JSON wrapper. Metadata is not included. ##### Example: base64 with storage ``` const requestBody = { url: 'https://example.com', output: 'base64', store: true }; const response = await fetch('https://api.simplescraper.io/v1/screenshot', { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data.screenshot); // base64-encoded image console.log(data.screenshot_url); // hosted URL for the screenshot ``` ##### Response ``` { "id": "ss_abc123def456", "url": "https://example.com", "status": "completed", "date_captured": "2026-02-25T12:00:00.000Z", "screenshot": "iVBORw0KGgoAAAANSUhEUgAA...", "screenshot_url": "https://screenshots.simplescraper.io/ss_abc123def456.png", "metadata": { "title": "Example Domain", "image_size_bytes": 234567, "capped": false } } ``` ### GET /v1/screenshot Full endpoint: `https://api.simplescraper.io/v1/screenshot?url=https://example.com&format=png` GET requests accept the same parameters as query strings. This is useful for simple integrations, browser testing, or systems that can't send POST bodies. The parameter names and accepted values are identical to the [POST endpoint above](#post-v1-screenshot). URL-encode the `url` and `wait_for_selector` parameters. ##### Example GET request ``` async function captureScreenshot(apikey, targetUrl) { const params = new URLSearchParams({ url: targetUrl, format: 'png', viewport_width: '1280', viewport_height: '800', device_scale_factor: '2' }); const requestUrl = `https://api.simplescraper.io/v1/screenshot?${params}`; try { const response = await fetch(requestUrl, { headers: { 'Authorization': `Bearer ${apikey}` } }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } captureScreenshot('YOUR_API_KEY', 'https://example.com'); ``` ## Credits and pricing Each screenshot request costs **1 credit**. See the [Credits page](https://simplescraper.io/docs/credits) for credit bundles and your current balance. ## Error responses | Status code | Error | Message | | --- | --- | --- | | 400 | `invalid-url-format` | The URL isn't valid. | | 400 | `invalid-value` | A parameter is outside its accepted range or not a valid option. | | 401 | `unauthorized` | API key is missing or invalid. | | 402 | `insufficient-credits` | Not enough credits on your account. | | 408 | `screenshot-timeout` | The page took too long to load or render. | | 429 | `rate-limit-exceeded` | Too many requests - slow down and retry. | | 500 | `screenshot-failed` | The screenshot couldn't be captured. This can happen if the page blocks automated browsers or returns a non-200 response. | ## Notes **Full-page capping** - Very tall pages may be capped at the browser's maximum capture height. If that happens, `metadata.capped` will be `true` in the response. You'll still get everything up to the cap. **Pages that scroll inside a panel** - Some sites, often web apps, documentation sites and trust centers, keep the page itself one window tall and scroll their content inside a panel. With `full_page` set to `true`, the screenshot opens that panel to its full height, so you get the whole page rather than the first screen. Content the site only draws as you scroll, such as lazy-loaded images further down, can still appear as placeholders. **Transparent backgrounds** - Set `hide_background` to `true` to get a transparent background on PNG or WebP screenshots. Handy for pages with no background color set. Has no effect on JPEG since the format doesn't support transparency. **Image storage** - When `output` is `url` or `store` is `true`, the image is saved to cloud storage and kept for 30 days. **The `wait_until` parameter** - Controls when the page is considered "loaded": - `load` - Waits for the `load` event (all resources including images). This is the default. - `domcontentloaded` - Fires when HTML is parsed, before images and stylesheets finish. Faster but may miss visual content. - `networkidle0` - Waits until there are zero network connections for 500ms. Best for SPAs and pages that fetch data after load. - `networkidle2` - Waits until two or fewer network connections remain for 500ms. Good for pages with persistent connections like analytics or websockets. ## Google Search API https://simplescraper.io/docs/google-search-api # Google Search API When you need Google's results for a search term as data, the useful format is the ranked list itself: each result's position, title, address and snippet, ready for code, a spreadsheet or an AI agent. The Google Search API returns exactly that from one request, up to 100 results at a time, together with the answer box, knowledge graph, 'People also ask' questions and related searches whenever Google shows them. **Use cases for the Google Search API:** - Rank tracking: see where your pages and your competitors' pages appear for the terms you care about - Lead and source lists: collect the sites Google returns for a niche, then scrape each one - Research and monitoring: follow what Google shows for a topic in a given country or over the past day, week or month - AI agents and RAG pipelines: give a model fresh, ranked sources before it reads them Prefer no code?: The same search is available in the dashboard as the 'Google search results' ready-made recipe, which saves the results as rows you can schedule and export. See [Ready-made recipes](https://simplescraper.io/docs/ready-made-recipes). ## How it works You send a search term, and Simplescraper asks Google for as many pages of results as your `limit` needs, ten results per page. The results come back in one response, numbered 1, 2, 3 in the order Google ranked them, with a result that Google repeats on a later page kept only once. Each page of ten results costs 2 credits, and a page that fails is not charged. ## Authentication All requests require an API key in the `Authorization` header: ``` Authorization: Bearer your_api_key ``` Your API key is available on your [Account Page](https://simplescraper.io/docs/managing-subscription) and on the API tab of each recipe. See the [main API authentication docs](https://simplescraper.io/docs/api-guide#authentication) for details. ## Endpoints Full endpoint: `https://api.simplescraper.io/v1/search` | Method | Usage | | --- | --- | | [POST /v1/search](#post-v1-search) | JSON body. Recommended, because it needs no URL encoding | | [GET /v1/search](#get-v1-search) | Query string parameters. Useful for quick tests in a browser or a simple integration | Both methods accept the same parameters and return the same response. ### Request parameters | Parameter | Required | Type | Default | Description | | --- | --- | --- | --- | --- | | query | Yes | String | \- | The search term, 1 to 400 characters. On GET you can also send it as `q` | | country | No | String | `us` | The two-letter country code Google searches from, such as `us`, `gb` or `de` | | language | No | String | Google's choice for the country | The two-letter language code of Google's interface, optionally with a region: `en`, `fr`, `pt-br` | | limit | No | Number | 10 | How many results to return, a whole number from 1 to 100 | | date\_range | No | String | `any` | Only results from this period: `any`, `past_hour`, `past_day`, `past_week`, `past_month` or `past_year` | | include | No | Array of strings | All four | Which extras to return beside the results: `answer_box`, `knowledge_graph`, `people_also_ask`, `related_searches`. On GET, a comma-separated list. Send an empty array, or an empty string on GET, for results only | A parameter the API does not recognize is refused with a 400 error that names it, so a typo such as `limt` never goes unnoticed. ### POST /v1/search ``` async function searchGoogle(apikey, query) { const url = 'https://api.simplescraper.io/v1/search'; const requestBody = { query: query, country: 'us', limit: 20, date_range: 'past_month' }; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const result = await response.json(); console.log(result.data); // the organic results, ranked } catch (error) { console.error('error:', error); } } searchGoogle('YOUR_API_KEY', 'web monitoring'); ``` ### GET /v1/search ``` curl "https://api.simplescraper.io/v1/search?q=web%20monitoring&country=gb&limit=10&include=answer_box,related_searches" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ### Response ``` { "id": "Xk2pQ0aLmN8rT4vW1yZb", "query": "web monitoring", "status": "completed", "date_searched": "2026-10-14T12:00:00.000Z", "metadata": { "engine": "google", "country": "us", "language": null, "date_range": "past_month", "result_count": 20, "pages_requested": 2, "pages_fetched": 2 }, "data": [ { "position": 1, "page": 1, "title": "What is Web Monitoring?", "url": "https://example.com/web-monitoring", "snippet": "Web monitoring is the process of checking websites for availability, performance and changes...", "date": "3 days ago", "sitelinks": [{ "title": "Pricing", "url": "https://example.com/pricing" }] } ], "answer_box": { "title": "...", "answer": "...", "snippet": "...", "url": "https://..." }, "knowledge_graph": { "title": "...", "type": "...", "description": "...", "website": "https://...", "attributes": {} }, "people_also_ask": [{ "question": "What is a web monitor?", "snippet": "...", "title": "...", "url": "https://..." }], "related_searches": [{ "query": "web monitoring tools" }], "errors": [] } ``` | Field | Description | | --- | --- | | id | A unique id for this search | | query | The search term, as searched | | status | `completed` when every page Google was asked for answered, `partial` when some pages failed (see [Partial results](#partial-results)) | | date\_searched | When the search ran, as an ISO 8601 date | | metadata | The country, language and date range used, `result_count` (results in `data`), `pages_requested` and `pages_fetched` | | data | The organic results, in Google's order | | answer\_box, knowledge\_graph, people\_also\_ask, related\_searches | Each appears only when you asked for it in `include` and Google showed one. They come from the first page of results | | errors | One entry per page of results that failed; empty when every page answered | Each result in `data` has these fields: | Field | Description | | --- | --- | | position | Rank in the returned list, counted from 1 without gaps | | page | The Google page the result came from, 1 to 10 | | title | The result's title | | url | The result's address | | snippet | The text Google shows under the title, or `null` when it shows none | | date | The date Google shows beside the result. Present only when Google shows one | | sitelinks | The sub-links Google shows under the result, each with `title` and `url`. Present only when Google shows them | A search Google has no results for is a success: `status` is `completed` and `data` is an empty array. ### Partial results A search over several pages can lose a page to a temporary failure on Google's side. When at least one page answered, you still get the results that arrived, with `status: "partial"` and one entry in `errors` for each page that failed: ``` { "status": "partial", "metadata": { "result_count": 10, "pages_requested": 2, "pages_fetched": 1 }, "errors": [ { "page": 2, "page_message": "search-page-failed", "response_code": 500 } ] } ``` Only the pages that answered are charged. To fill the gap, send the same search again. ## Credits Each page of up to ten results costs 2 credits, taken from your account's credits: - `limit` 10 or less is one page and costs 2 credits. - `limit` 20 is two pages and costs 4 credits; `limit` 15 also needs two pages and costs 4. - `limit` 100 is ten pages and costs 20 credits. Before Google is asked, the API checks that your credits cover every page the search needs. If they do not, the search is refused with a 402 error and nothing is charged. After the search, you are charged only for the pages that returned results: a page that failed costs nothing, and when the first page has no results, no further page is fetched or charged. ## Error responses Errors return a JSON body with a `type` and a `message`, for example: ``` { "error": { "type": "invalid-value", "message": "Invalid value provided.", "field": "limit" } } ``` | Status code | Error | When it happens | | --- | --- | --- | | 400 | `invalid-value` | A parameter is missing, out of range or not recognized. `field` names the parameter | | 402 | `out-of-credits` | Your credits cannot cover the pages the search needs | | 403 | `api-key-not-included` | The request carries no API key | | 404 | `user-not-found` | The API key does not belong to an account | | 408 | `search-timeout` | Google did not answer in time. Try again | | 500 | `search-failed` | Google could not be searched. When the search service answered with an error, `response_code` carries its status. Try again | | 500 | `search-not-configured` | Search is temporarily unavailable on our side. Try again later or reach out via chat | A search that fails with any of these errors is not charged. ## Use it from an AI assistant The [Simplescraper MCP server](https://simplescraper.io/docs/mcp-server) offers the same search as the `google_search` tool, with `query`, `country`, `language`, `date_range` and a `limit` from 1 to 20. Ask your assistant something like *"Search Google for the top project management tools and summarize the first ten results"*, and it can then read any result page with the `extract_markdown` tool. ## FAQ ### How fast is a search? A search for 10 results typically answers in about two seconds, and 20 results in about two and a half. Larger searches fetch their later pages in parallel, so they take only a little longer. ### Can I get more than 100 results? One request returns up to 100 results, which is Google's first ten pages. There is no paging beyond that. ### Can I choose the country and language? Yes. `country` sets the Google market the search runs in, and `language` sets the interface language. Without `language`, Google uses its default for the country. ### Does a search show up in my usage? Yes. Each API search appears on the usage page under 'API', with the credits it cost. ## Related - [Ready-made recipes](https://simplescraper.io/docs/ready-made-recipes) - the 'Google search results' recipe runs the same search from the dashboard, on a schedule if you like - [Extracting websites to Markdown](https://simplescraper.io/docs/extract-markdown) - read the pages a search returns - [MCP server](https://simplescraper.io/docs/mcp-server) - search Google from Claude, ChatGPT or Cursor For questions or help, reach out via chat. ## URL Extractor API https://simplescraper.io/docs/extract-urls-api # URL Extractor API The URL Extractor API discovers and extracts all URLs from any website. Point it at a target domain and it handles sitemap discovery and URL aggregation. This endpoint does not deduct credits. ## Authentication All requests require an API key in the `Authorization` header: ``` Authorization: Bearer your_api_key ``` Your API key is available on your [Account Page](https://simplescraper.io/docs/managing-subscription) and on the API tab of each recipe. See the [main API authentication docs](https://simplescraper.io/docs/api-guide#authentication) for details. ## Endpoint Full endpoint: `https://api.simplescraper.io/v1/extract-urls` The endpoint accepts POST requests with a JSON body. ### Request body properties | Property | Required | Type | Description | | --- | --- | --- | --- | | domain | Yes | String | Full URL with protocol (e.g., `https://example.com`) | | urlLimit | No | Number | Maximum number of URLs to return. | | sitemapLimit | No | Number | For sitemap-index sites, maximum number of nested sitemaps to traverse. Useful for sampling large sites quickly. | ### Example request ``` async function extractUrls(apiKey, domain) { const url = 'https://api.simplescraper.io/v1/extract-urls'; const requestBody = { domain: domain, urlLimit: 100 }; try { const response = await fetch(url, { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }); const data = await response.json(); console.log(data); } catch (error) { console.error('error:', error); } } extractUrls('YOUR_API_KEY', 'https://example.com'); ``` ### Response structure | Property | Type | Description | | --- | --- | --- | | success | Boolean | Whether the request succeeded | | data | Object | Container for sitemap and URL results | | data.sitemapUrl | String or null | URL of the discovered sitemap. Null if no sitemap was found. | | data.urls | Array | List of URLs extracted from the sitemap(s) | | data.status | String | `"completed"` on success, `"error"` on failure | ### Example response ``` { "success": true, "data": { "sitemapUrl": "https://example.com/sitemap.xml", "urls": [ "https://example.com/page1", "https://example.com/page2", "https://example.com/page3" ], "status": "completed" } } ``` ## Error responses All errors return `{ success: false, error: "string" }`. | Status code | Error | Message | | --- | --- | --- | | 400 | Domain is required | The `domain` parameter is missing from the request body. | | 400 | Invalid domain: must be a valid web URL | The `domain` value is not a valid HTTP/HTTPS URL. | | 401 | Invalid API key | API key is missing or invalid. | | 403 | Origin not allowed | No valid API key and the request origin is not allowlisted. | | 429 | Rate limit exceeded | Daily request limit for your tier has been reached. The error message includes the reset time. See "Rate limits" below. | | 500 | Internal Server Error | An unexpected error occurred during sitemap discovery or extraction. | ## Rate limits Each response carries: - `X-RateLimit-Limit` - daily request cap for your tier - `X-RateLimit-Remaining` - requests left in the current 24-hour window - `X-RateLimit-Reset` - ISO timestamp when the window resets - `X-RateLimit-Concurrency` - max simultaneous in-flight requests Per-tier caps: | Tier | Daily | Concurrent | | --- | --- | --- | | Active subscription | 500 | 10 | | Free tier | 50 | 2 | Two ways to hit HTTP 429: - **Daily cap reached** - error message includes the reset time. Wait until then. - **Concurrency cap reached** - error message says `"Too many concurrent requests"`. Transient. Retry once an in-flight request completes. ## Notes **No credit cost:** This endpoint does not deduct credits from your account. **Sitemap indexes:** If the discovered sitemap is a sitemap index, the endpoint fetches nested sitemaps and aggregates all URLs. Use `sitemapLimit` to cap how many. **URL limits:** `urlLimit` caps the total URLs returned, applied after all sitemaps are processed. **No sitemap found:** If no sitemap can be discovered, the response returns HTTP 200 with `status: "no_sitemap_found"` and a `message` linking to [Deep scraping URLs](https://simplescraper.io/docs/deep-scraping-urls). **Partial results:** Requests run up to 60 seconds. If traversal hasn't finished, the response returns HTTP 200 with `status: "time_limit_reached"` and the URLs found so far. Most sites finish well under that. **MCP server access:** This endpoint is also available via the Simplescraper MCP server as the `extract_urls` tool. See the [MCP server docs](https://simplescraper.io/docs/mcp-server) for details. ## Extract API v0 (deprecated) https://simplescraper.io/docs/creating-an-api # Creating an API ## This guide is deprecated. Please read the new Simplescraper API guide here: [https://simplescraper.io/docs/api-guide](https://simplescraper.io/docs/api-guide). An API endpoint will be automatically created as soon as you run a recipe. Simply click on the API tab of your recipe to get the URL which you can query for extracted data. When you call this URL, you have the option to return data that was scraped previously or to scrape the target website in real-time (via the *&run\_now=true* parameter) and return live data. The API tab on your recipe page lists all parameters that you can pass in with the URL in order to modify the recipe, some of which are explained below. | Parameter | Purpose | Example | | --- | --- | --- | | source\_url | Change the source URL (page to be scraped) of the recipe. Useful for sites like job boards, accommodation websites etc. where different URLs can be passed in to return different results | &source\_url=[https://example.com](https://example.com) | | run\_now | Run the recipe before returning results, i.e. scrape the website now. Without this parameter the most recent results are returned from the database | &run\_now=true | | limit | Limit how many scrape results are returned in a single call to the API. Applies only to cached data already in database (so when *&run\_now=false*) | &limit=1000 | | run\_async | Run request asynchronously. Ideal for long running scrape tasks - allows you to trigger the recipe to run without risk of timeout. Requires that you handle results via webhook or other integration | &run\_async=true | Please note that URLs in the Crawler are ignored when triggering a recipe via the API. Requests via the API scrape a single URL at a time only. To scrape URLs in the Crawler without having to visit the dashboard, set a schedule and use a webhook or other integration to send the data to your desired destination. ## Smart Extract https://simplescraper.io/docs/smart-data-extract # How to use Smart Extract > Tell the AI what data you want — it finds and extracts it for you. Simplescraper Smart Extract uses AI to accurately extract data and generate reusable CSS selectors from any website. All that's required is a URL and a data schema that lists the properties you wish to extract. Smart Extract can be accessed via: - The [Simplescraper dashboard](https://simplescraper.io/dashboard/extract) - click 'Get new data' in the sidebar, then 'Scrape webpages'. Smart Extract powers the 'Structured data' output there - Programmatically via the [API](https://simplescraper.io/docs/api-guide#post-smart-extract) Smart Extract + CSS selectors: Smart Extract returns valid CSS selectors, making it easy to convert AI-powered extractions into regular scrape recipes. Start with AI to quickly identify and validate the data you want, then switch to standard scraping for speed, accuracy, and scale. This combines the flexibility of AI with the reliability of traditional scraping - meaning no hallucinations or context limits. ### Dashboard Usage - Open [Get new data](https://simplescraper.io/dashboard/extract) in the dashboard sidebar and select the 'Scrape webpages' tab - Under 'Website to scrape', choose 'Single URL' and enter the URL of the page you wish to extract data from. To run the same extraction across many pages, choose 'Multiple URLs' and paste one URL per line (up to 50 per run) - Keep 'Data required' set to 'Structured data' - Under 'Values to extract', enter a data schema (comma-separated list of properties you wish to extract) - Click 'Get data' - After a few seconds, your results appear as a table. Switch between the 'Data' and 'CSS selectors' views to see the extracted values and the selector behind each field - when Simplescraper finds the data in the page's network API, the selector view shows an API field path instead - Click 'Download data' to save the results as CSV - Click *'Save as scrape recipe'* to convert the extraction into a regular scrape recipe, allowing you to scrape at scale using Simplescraper The 'Scrape webpages' tab also offers two options that work differently from Smart Extract: the 'Markdown' output returns each page as clean markdown text instead of structured rows (see [Extracting websites to Markdown](https://simplescraper.io/docs/extract-markdown)), and 'From a recipe' re-runs one of your existing scrape recipes. ### API Usage - API docs can be found here: [https://simplescraper.io/docs/api-guide#post-smart-extract](https://simplescraper.io/docs/api-guide#post-smart-extract). ### Tips on writing your data schema - The schema provided should be a short, accurate list of each of the visible data points on the website that you wish to extract. - For example, if extracting data from a jobs board: *'Role, salary, location, job type, company, description, experience required'* is a good schema. - Including a hint of what data is being extracted can increase accuracy. - For example, instead of: *'name, location, price, size, bedrooms, bathrooms'*, including a reference to the type of data can improve results. Example: *'**property** name, location, price, size **(sqm)**, bedrooms, bathrooms'*. - A schema is not a prompt. - This works: "title, old price, current price, discount, review count, description, num capsules, rating". - This does not: "visit the website and extract everything on the page beginning with A". * * * ## Example schemas Use these as inspiration for your own extractions: | Site | Schema | | --- | --- | | **Car auctions** (carsandbids.com) | `car name, details, time remaining, bid price, location` | | **Product page** (nike.com) | `price, old price, name, num of colors` | | **Job board** (Microsoft careers) | `job title, location, remote possible, description` | | **Real estate** (realestate.com.au) | `price aud, price us, location, size (m2)` | | **Social profile** (x.com) | `name, @tag, tagline, joined date, link, number of posts, top tweet text` | ## Workflow example: Using Smart Extract to create a scrape recipe One powerful way to use Smart Extract is as a starting point for creating a regular scrape recipe. This gives you the flexibility of AI to quickly identify the data you want, and then the speed, accuracy, and reliability of standard scraping for production use. The idea: extract data and selectors → review → save as recipe → scale. ### Step 1: Extract data and selectors using Smart Extract Use Smart Extract to extract the data you want from a page. You'll get both the structured data *and* a list of CSS selectors for each field. ``` async function runSmartExtract(apiKey, url, schema) { const response = await fetch('https://api.simplescraper.io/v1/smart-extract', { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ url, schema }) }); const data = await response.json(); return data; } ``` **Example usage:** ``` const apiKey = 'YOUR_API_KEY'; const url = 'https://example.com/product-page'; const schema = 'product name, price, description, rating'; const result = await runSmartExtract(apiKey, url, schema); ``` ### Step 2: Review the results You'll receive two important things in the response: - `data`: an array of structured values (the extracted data) - `selectors`: a list of CSS selectors for each field you requested If the selectors look good, you're ready to convert them into a scrape recipe. ### Step 3: Create a recipe using the returned selectors Once you're happy with the fields and selectors, use them to create a new recipe. This lets you run scalable scrapes using Simplescraper's standard scraping engine. ``` async function createRecipe(apiKey, name, url, selectors) { const response = await fetch('https://api.simplescraper.io/v1/recipes', { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ name: name, url: url, selectors: selectors }) }); const recipe = await response.json(); console.log('Recipe created:', recipe); return recipe; } ``` **Example usage (continued):** ``` const recipeName = 'Product Page Scraper'; await createRecipe(apiKey, recipeName, url, result.selectors); ``` So start with Smart Extract to quickly find the right selectors, then save them as a recipe for fast, reliable scraping at scale. If the site changes later, you can run Smart Extract again and update the existing recipe using `PUT /recipes/:recipeId` - no need to manually update your recipes. > Notes: > > Smart Extract is in beta and may not be 100% accurate. If you encounter any issue or incorrect data please contact us via chat ## Enhance scraped data using AI https://simplescraper.io/docs/enhance-scraped-data-using-ai # Enhance scraped data with AI When you scrape data there is often another step: cleaning it up, or pulling out the part you need, such as an email buried in page text or a remote-work policy in a job description. AI Enhance columns do this for you automatically. AI Enhance can: - **Summarize** long text into a line or two - **Extract** one detail, such as an email or a price - **Answer yes or no** to your question, such as "Is this job remote?" - **Sort** each row into one of your categories - **Follow your own instruction**, such as cleaning up a title Each answer goes into a new column of your results, called an AI column, with one answer for every row. Your scraped data is never changed. Run the recipe once first: AI Enhance works on data that is already scraped, so the recipe needs at least one run. It doesn't need a full run: click 'Test' on the recipe page to scrape a single page, and AI Enhance can use those rows. ![The AI Enhance panel open beside a recipe's results, adding an is_remote column that checks whether each job allows remote work](https://simplescraper.io/docs/assets/ai-columns-docked-beside-table.pdVi8D2A.png) ## What each type of AI column does The examples below use a recipe that scrapes job listings. A **Summarize** column writes a short summary of the properties you choose. - Reads `job_description`, a 600-word posting, with 'Length' set to 'Two bullets' - Writes 'Builds data pipelines for the billing team' and 'Owns reporting for finance and sales' An **Extract details** column finds one detail in each row and writes it to the column. - Reads `job_description` and asks for 'The email address to apply to' - Writes 'careers@example.com' A **Yes or no** column answers one question about each row. - Reads `job_title` and `locations` and asks 'Does this job allow remote work?' - Writes 'Yes' for a job in 'Remote - US' and 'No' for a job in 'Austin, TX' A **Categorize** column puts each row in one of the categories you list and never invents another. - Reads `department` with the categories 'Engineering', 'Research' and 'Sales' - Writes 'Engineering' for 'Applied AI Engineering' and 'Sales' for 'Technical Success' Yes or no and Categorize columns fill every row with the answer the AI finds most likely. A Categorize column also offers 'Add an 'Other' category for rows that fit none', ticked for a new column, so a row about something outside your list is marked 'Other' instead of being put in the nearest category. A **Custom instruction** column follows your own instruction for every row. - Reads `job_title` and follows 'Spell out abbreviations in the title' - Writes 'Senior Software Engineer (Payments)' for 'Sr. Software Engineer (Payments)' ## Add an AI column 1. In the [Simplescraper dashboard](https://simplescraper.io/dashboard), open a recipe's 'Results' tab and click 'AI Enhance' above the table. 2. Pick a starting point from the list, or click 'New AI column' if the recipe already has one. ![The empty AI Enhance panel listing five starting points: Summarize, Extract details, Yes or no, Categorize and Write your own instruction](https://simplescraper.io/docs/assets/ai-columns-empty-state.QqzKNY1P.png) 3. Type a name such as `is_remote` in 'New AI column name', fill in the column type's fields and tick the properties the AI should read under 'Properties to include'. 4. Click 'Preview on 3 rows' to check the answers. 5. Click 'Save', or 'Save and run' to fill the column for every row now. To fill the column every time the recipe runs, including scheduled runs, set 'When it runs' to 'After each scrape' before you save. ## Credits AI Enhance uses 1 credit per 800 words, counting the data it reads plus the answer it writes. The panel shows an estimate before you preview or run. ## AI columns in your integrations - **Google Sheets:** the new column is added to the right of your existing columns. - **Email:** AI columns are included unless you switch one off in the email panel. - **Airtable:** switch the column on in the Airtable panel and add a field with the same name to your table. Without that field, the column is skipped, every other field still arrives and the panel says which field was skipped. ## What AI Enhance cannot do AI Enhance reads only the data already in each row, so it can't click, open links, move to the next page or search the web. ## FAQ ### Does AI Enhance change my scraped data? No. Each AI column writes only to its own column, and deleting a column keeps the values it already wrote. ### What happens when a row doesn't contain the answer? It depends on the column type: - **Summarize and Custom instruction:** the cell stays empty. - **Extract details:** the cell follows the 'If the detail is not there' setting, 'Leave empty' or "Write 'Not stated'". - **Yes or no:** the column always answers, writing 'No' or 'false' depending on 'Answer as'. - **Categorize:** every row gets the category it fits best. With 'Add an 'Other' category for rows that fit none' ticked, a row that fits none of yours gets 'Other'. ### How many AI columns can a recipe have? Up to 10. ### What happens to my older AI Enhance setup? It keeps working as before. It appears first in the panel's list, marked 'Legacy'. ## Related - [Credits](https://simplescraper.io/docs/credits) - how credits are counted across Simplescraper - [Scheduling your webscraping](https://simplescraper.io/docs/scheduling-webscraping) - pair a schedule with columns that run after each scrape - [Deep scraping](https://simplescraper.io/docs/deep-scraping-urls) - bring linked pages into your rows so AI Enhance can read them For questions or help, reach out via chat. ## Extracting websites to Markdown https://simplescraper.io/docs/extract-markdown # Extracting Websites as Markdown data > Convert any webpage into clean Markdown — ideal for AI analysis. Simplescraper enables you to easily extract and save a webpage, or an entire website's text content, in Markdown format. Markdown retains the page formatting and is a preferred format when analyzing web date using AI models such as OpenAI's ChatGPT and Anthropic's Claude. There are a number of ways to save website data as Markdown format with Simplescraper: ## Via Crawl entire websites ![image-auto0crawl](https://simplescraper.io/docs/assets/image-auto-crawl.cA6nPA9I.png) - Visit the [Simplescraper dashboard](https://simplescraper.io/dashboard) and click 'Get new data' in the sidebar, then select 'Crawl entire websites' - Enter the URL of the website you wish to save as Markdown - Under the crawl options, choose the maximum number of pages to scrape and any URL patterns to restrict the crawl to - Click the 'Test' button to run a quick crawl on two pages. After a few seconds, results will appear - check them to ensure the data looks correct before proceeding - Click 'Run' and Simplescraper will navigate through the website, saving a Markdown version of each page - When the crawl is completed, download options for Markdown, JSON, and CSV will appear ### Continuing a crawl If you originally set a limit on the number of pages or ran out of credits and want to scrape more: - In the crawl history at the bottom of 'Crawl entire websites', click the 'Continue' button next to one of your previous crawls - The website's URL will load. Increase the page limit as needed (for example, if you scraped 1000 pages before and now want to scrape 3000, enter 3000) - Click 'Run' and Simplescraper will continue to crawl the website, scraping only those pages that were not previously scraped - When the task is completed, updated Markdown, JSON and CSV download options will appear ## Via a list of URLs If you already know exactly which pages you want, there's no need to crawl: - Open 'Get new data' in the sidebar, then select 'Scrape webpages' - Choose 'Multiple URLs' and paste one URL per line (a single page works the same way via 'Single URL') - Under 'Data required', select 'Markdown' - Click 'Get data' and Simplescraper will save each page as clean Markdown, with the same Markdown, JSON, and CSV download options when the run completes ## Via a scrape recipe - When saving a scrape recipe ([this guide](https://simplescraper.io/docs/how-scrape-save-recipes) covers saving recipes), click the Advanced options section and toggle the 'Extract Markdown' button to the on position - ![Extract Markdown](https://simplescraper.io/docs/assets/extract-markdown.BBhE3FOv.png) - Run your recipe and the Markdown will appear in its own column and a 'download Markdown' button will be available - ![Preview Markdown](https://simplescraper.io/docs/assets/markdown-preview.56uYDTps.png) - Note that if the Markdown is very large (over 10MB), the file will be downloaded as a zip ## Via the API - When calling the Simplescraper API, include `extractMarkdown: true` in the body of the request - ``` const apikey = 'ap1k3y'; const requestBody = { extractMarkdown: true, }; const response = fetch(`https://api.simplescraper.io/v1/recipes/${recipeId}/run`, { method: 'POST', headers: { 'Authorization': `Bearer ${apikey}`, 'Content-Type': 'application/json' }, body: JSON.stringify(requestBody) }) ``` - Please read the full API guide for more details on data extraction via the API: [https://simplescraper.io/docs/api-guide](https://simplescraper.io/docs/api-guide) ## Extracting URLs When crawling entire websites, you can also extract all URLs discovered during the crawling process. In the crawl options, under 'Collect URLs', choose from the following: - Don't collect URLs: No URLs will be extracted - All URLs (internal + external): Extract every URL found on the crawled pages - Internal URLs only: Extract URLs from the same domain only - External URLs only: Extract URLs from different domains only ![image-20250613054411787](https://simplescraper.io/docs/assets/extract-urls.Dh0ucfuV.png) The extracted URLs will be available as a CSV download option alongside the Markdown content when the crawl completes. ## MCP server https://simplescraper.io/docs/mcp-server # Using Simplescraper's MCP Server Connect Simplescraper to Claude, Cursor, ChatGPT, or any AI tool that supports MCP. Once connected, your AI assistant can extract structured data from any page, discover the URLs on a site, read pages as clean Markdown, create and run scrape recipes, and work with your previous scrape results - all from a normal chat prompt, with no scraping code to write. Model Context Protocol (MCP) is the open standard these tools use to call external services like Simplescraper. If you're new to it, our [complete guide to the MCP protocol](https://simplescraper.io/blog/how-to-mcp) covers the basics. Simplescraper's MCP server URL:: [https://mcp.simplescraper.io/mcp](https://mcp.simplescraper.io/mcp) ### What you can do | Ask your AI assistant | What Simplescraper does | | --- | --- | | "Pull product names, prices, and ratings from this page" | Extracts the named fields with AI, no CSS selectors needed | | "Find every page on stripe.com" | Reads the site's sitemap and returns the full URL list (no credits used) | | "Read this page and summarize it" | Fetches the page and converts it to clean Markdown | | "Create a recipe for nytimes.com that grabs headlines" | Builds a new scrape recipe you can reuse | | "Run my competitor scraper and show the new data" | Executes a saved recipe and returns fresh results | | "What changed since the last run?" | Pulls previous results and run history to compare | | "Add these URLs to my product scraper" | Updates the recipe's batch URL list for a bulk run | Start with URL discovery: `extract_urls` reads a site's sitemap and returns every page URL without using any credits. It's a natural first step - map a site, then point `smart_extract` or a recipe at the pages that matter. * * * ### Quick Setup #### Claude Desktop 1. Open Claude Desktop settings 2. Navigate to Settings (bottom-left) → Integrations 3. Click "Add MCP Server" 4. Enter these details: - **Name**: Simplescraper - **URL**: `https://mcp.simplescraper.io/mcp` 5. Complete the authentication flow to connect your Simplescraper account 6. Start using natural language to control your scrapes! Claude Integrations availability: Integrations are currently available on the Max plan, with broader availability coming soon. #### Cursor IDE Cursor supports MCP with OAuth authentication. When you add the server, Cursor will open a browser window for you to log in with your Simplescraper account. **Global Setup (All Projects):** 1. Create `~/.cursor/mcp.json` in your home directory 2. Add this configuration: ``` { "mcpServers": { "simplescraper": { "url": "https://mcp.simplescraper.io/mcp" } } } ``` 3. When you toggle the server on in Cursor, it will prompt you to authenticate with your Simplescraper account **Project-Specific Setup:** For tools specific to a project, create a .cursor/mcp.json file in your project directory: 1. Create `.cursor/mcp.json` in your project root 2. Use the same JSON configuration as above 3. The server will only be available in that project **Via Cursor Settings UI:** 1. Open Cursor Settings (Ctrl/Cmd + ,) 2. Navigate to MCP in the sidebar 3. Click "Add Server" 4. Enter: - **Name**: Simplescraper - **URL**: `https://mcp.simplescraper.io/mcp` 5. When prompted, log in with your Simplescraper account Do not add an Authorization header manually: Cursor handles authentication automatically through OAuth. Adding `"headers": {"Authorization": "Bearer ..."}` with your API key will prevent the login flow from working. Use the config above with just the URL. #### Other AI Applications Most AI applications that support MCP will have a similar setup process. Look for MCP or "Tools" settings and add our server URL: `https://mcp.simplescraper.io/mcp` * * * ### Available Capabilities Simplescraper exposes 14 tools your LLM can call to scrape sites and manage your recipes. #### Data Extraction | Tool | What it does | Example prompt | | --- | --- | --- | | `run_recipe` | Run a saved recipe to extract fresh data from its configured URL. Consumes credits. | *"Run my Twitter profile scraper now and show me the new data"* | | `extract_urls` | Discover all page URLs on a site by parsing its sitemap. Free - no credits deducted. | *"Get me a list of all pages on stripe.com"* | | `google_search` | Search Google and get up to 20 ranked results (title, URL, snippet, position) plus the answer box, knowledge graph, 'People also ask' and related searches. 2 credits per ten results. See [Google Search API](https://simplescraper.io/docs/google-search-api). | *"Search Google for the best CRM for small teams and summarize the top results"* | | `extract_markdown` | Fetch one page and return its body as clean Markdown. Best for reading articles, blog posts, docs, or any page content. | *"Read [https://example.com/blog/post](https://example.com/blog/post) and summarize it"* | | `smart_extract` | Use AI to extract specific named fields from a page (price, title, author, etc) - no CSS selectors required. | *"Extract product names, prices, and ratings from [https://example.com/products](https://example.com/products)"* | | `get_latest_results` | Get the most recent scraped data for a recipe (existing data, no new scrape). | *"Show me the latest results from my Twitter profile scraper"* | | `get_results` | Fetch data from a specific scrape run by results\_id. | *"Get the data from results ID abc123"* | | `get_results_history` | List previous scrape runs for a recipe with timestamps and page counts. | *"When did my Amazon scraper last run, and how many times this week?"* | #### Recipe Management | Tool | What it does | Example prompt | | --- | --- | --- | | `list_recipes` | Find recipes by domain, name keyword, or recent activity. | *"List all my recipes that scrape from linkedin.com"* | | `get_recipe` | View full details of a specific recipe, including selectors. | *"Show me the selectors in my Amazon price tracker recipe"* | | `create_recipe` | Build a new scrape recipe with CSS selectors. | *"Create a recipe for nytimes.com that grabs headlines and timestamps"* | | `update_recipe` | Modify an existing recipe's name, URL, or selectors. | *"The price selector on my Amazon scraper needs updating to .new-price-class"* | #### Batch Operations | Tool | What it does | Example prompt | | --- | --- | --- | | `update_batch_urls` | Add or replace the list of URLs for batch (crawler) scraping. Pair with `extract_urls` to discover URLs first, then `run_recipe` to execute. | *"Add these 10 product URLs to my product details scraper"* | #### Screenshots | Tool | What it does | Example prompt | | --- | --- | --- | | `screenshot` | Capture a screenshot of a page and get back a hosted image URL. 1 credit per request. | *"Take a full-page screenshot of [https://example.com](https://example.com)"* | ### Advanced Usage Examples #### URL Discovery and Batch Scraping ``` "Find all product pages on example.com/shop, add them to my product scraper recipe, and run the scrape." ``` Claude will parse the sitemap, filter URLs to match your description (e.g. paths containing `/product/`), update the batch URL list, and execute the recipe. #### Complex Workflows ``` "Check my Amazon price tracker recipe. If prices have changed by more than 10% since the last run, create a summary report of the changes." ``` #### Multi-Step Operations ``` "Find all recipes that haven't run in the last week, run them, and summarize any that return errors." ``` #### Data Transformation ``` "Get the latest results from my job listings scraper and format them as a markdown table sorted by salary." ``` * * * ### Getting Help - **Support**: Reach out via chat on the Simplescraper website - **Learn more about MCP**: Read our [complete guide to the MCP Protocol](https://simplescraper.io/blog/how-to-mcp) ## Scraping data into Google Sheets https://simplescraper.io/docs/saving-data-google-sheets # Scraping data automatically into Google Sheets You can automatically save your scrape results to Google Sheets, making it possible to work with the data the instant that it's scraped. Here's how. *Privacy first: Simplescraper requests the least amount of permissions necessary which means that it cannot view, know about or access any existing files on your Google Drive. It can only access files that have been created with Simplescraper.* ## Connecting to Google Sheets To connect Simplescraper to Google Sheets click on any of your recipes and then click the 'Send' tab. Click the 'connect' button and follow the instructions. A new tab will open asking you to confirm Simplescraper's access. Once you confirm you will be redirected to a page asking you to re-login to Simplescraper. Once completed you can close the tab. Simplescraper is now connected to Google Drive. Now when you run a scrape job your scrape results will automatically be saved to a sheet on your Google drive with the same name as your recipe. ## Choosing to replace or append data The first time that Simplescraper runs with the 'save to Google sheets' option selected it will create a new sheet with the same name as the recipe. When you run the recipe again, the new result data can either replace any existing data in that Google Sheet or else be appended to any data that is already there. If you choose to append the scrape result data you can choose to identify a unique field so that Simplescraper can avoid duplicates. The unique field will depend on the data that you are scraping - a URL, title or any static value are perfect options. ## Customizing or adding formulas to your data in Google Sheets When Simplescraper updates your Google Sheet with new data, it may overwrite the entire sheet (tab). This means that formatting and formulas applied to the sheet will be lost. The solution is to create a second sheet (tab) that references the data created by Simplescraper and apply formulas and formatting there. **The solution:** 1. Create a second sheet: - Sheet1: Data from Simplescraper - Sheet2: Your newly created sheet for analysis and formatting 2. Reference Data: in Sheet2, cell A1, enter this formula: ``` =Sheet1!A:Z ``` This copies all data from Sheet1. Adjust 'Z' to the last column with data. 3. Add your formulas and formatting to Sheet2. 4. Keeping Rows in Sync: If Sheet1 has more rows than Sheet2, extend your formulas and formatting: - Select the last row in Sheet2 with formatting and formulas. - Drag the small blue square in the bottom-right corner down to match Sheet1's row count. This copies your formulas and formatting to new rows while maintaining Sheet1 references. This approach preserves your raw data in Sheet1 while allowing you to freely analyze and format in Sheet2. When new data is added to Sheet1, your formulas in Sheet2 will automatically update. ## Scraping data into Airtable https://simplescraper.io/docs/scraping-data-into-airtable # Scraping website data into Airtable With Simplescraper you can scrape data from any website directly into your Airtable bases quicky and easily. This guide has been updated to show you how to use Personal Access Tokens (PAT) instead of the now deprecated API keys. *Follow the steps below or watch the short video above* ### Activating the Airtable Integration - In Simplescraper, navigate to the Send tab of your recipe and locate the Airtable integration - Toggle the Activate switch on to enable the integration ### Setting up Your Personal Access Token To add your Personal Access Token in Airtable: - Visit your Personal Access Token page on Airtable: [https://airtable.com/create/tokens](https://airtable.com/create/tokens) - Click the '+ Create new token' button - Name your token something like 'Simplescraper access' - Under Scopes, select 'data.records:read' and 'data.records:write' - Choose the access level for your PAT: - For flexibility, select 'All current and future bases in all current and future workspaces' - If you'll only use a single base, choose access to that specific base - After creating your token, copy it to your clipboard (token is shown only once) - Paste this token into the Personal Access Token field in Simplescraper and press Enter to save ### Adding Your Base ID - Open the Airtable base you wish to save data into - Note the string after '[https://airtable.com/](https://airtable.com/)' in your URL bar (begins with 'app', e.g., 'appQjzhfyaBCde') - Copy this base ID and paste it into the Base ID field in Simplescraper - Press Enter to save Alternative method: - Navigate to Airtable's [API page](https://airtable.com/api) - Click on your desired base - Find the Base ID highlighted in green in the introduction section ### Setting Your Table Name - Navigate to your Airtable base - Copy the name of the table where you want the data to appear - Paste this name into the Table name field in Simplescraper - Press Enter to save ### Configuring Properties - In the Airtable Integration section in Simplescraper, toggle on the property names in the 'Included properties' section for the data you want to save - For each selected property, create a corresponding single-line text field in your Airtable table with the exact same name - Set up these fields in Airtable before you begin - For link properties, create an additional field with '\_link' added to the property name - Example: For 'jobtitle' that's a link, create both 'jobtitle' and 'jobtitle\_link' fields - If a property has no matching field in your table, the send still goes through: that property is left out and every other field arrives ### Choosing a Save Method Under 'Save method', choose how each send treats the records already in your table: - **Append new results to existing data:** each send adds new records and keeps the old ones. - **Replace existing data with new results:** each send adds the new records, then deletes the old ones. The old records are deleted only after the new ones exist, so a send that fails never leaves your table empty. 'Test integration' never deletes records. ### Testing and Running - Click 'Test integration' to verify the connection - If successful, you'll see the message 'Test passed - successfully connected.' If an included property has no matching field, the message names it and asks you to add the field or switch the property off - Run your scrape recipe, and when complete, your data will automatically save to your Airtable base - Under the buttons, the panel shows the outcome of the last send, for example 'Last sent 8 Oct 14:02 with 12 records.', and names any field that was left out ## Troubleshooting If the test integration button didn't return a 'test passed' message it's likely an issue with your Airtable configuration. Follow the steps below for solutions to common error messages: - 'Something went wrong - please try again.': make sure to press the ENTER key after entering the Personal Access Token, Base ID and Table name into the input fields - 'Not found - base ID or table name may be incorrect or may not exist': verify that the base ID matches the base that you wish to scrape data into, and ensure that a table with the exact same name that you entered in the 'Table name' section exists in that base - 'Airtable has no field named ...': the named property was left out of the send because your table has no field with that name. Add a single line text field with exactly that name in Airtable, or switch the property off - 'None of the included properties has a field in your Airtable table.': nothing was sent. Add a single line text field with the same name for each property you send - 'No Airtable API key is saved.' or 'The saved Airtable API key can't be read.': paste your Personal Access Token into the API key field again and press Enter Note: **your Airtable Personal Access Token is encrypted before being securely stored.** If you have any questions please reach out to us via chat. ## Scraping data into Zapier https://simplescraper.io/docs/scraping-data-into-zapier # Scraping website data into Zapier Zapier connects with over 2000 apps meaning that once you scrape website data from Simplescraper into Zapier you can send it almost anywhere on the web. So let's do it. *Follow the steps below or watch the 1 minute video above* - In the [Zapier dashboard](https://zapier.com/app/dashboard) click on the 'Make A Zap' button on the left-hand side - When the Zap editor opens, search for 'webhook' and click the 'Webhooks by Zapier' option. In the menu that pops up click the dropdown below *Trigger Event* and select 'Catch Hook 'and then click the Continue button - At the *Set Up Trigger* menu you should see a Custom Webhook URL. Copy this URL and then click Save & Continue - You should now be at the *Test Trigger* menu. Before you test the trigger you'll need to send the scraped data from Simplescraper to Zapier so open the [Simplescraper dashboard](https://simplescraper.io/dashboard) in a new tab or window and go to the recipe whose data you want to send to Zapier - Navigate to the Send tab of that recipe and paste the URL that you copied earlier into the webhooks input field and press the enter key to save it - Now click the Run Recipe button in Simplescraper. Because we've set the webhook URL the scraped data will be sent to that URL when the recipe finishes. It's time to test that it's working so jump back to Zapier before the scraping task is completed - In Zapier click the Test Trigger button which tells Zapier to detect any incoming webhook data. **You may need to click this button a few times as it needs to be clicked right around the time that the scrape task finishes in Simplescraper** - When the request has been found you'll see a 'We found a request!' message in Zapier and the scrape results should be visible. Almost done, now click the Continue button - You're now on the Action menu. From here you can tell Zapier what to do with the scraped data it received via the webhook. After you've selected what you want to do with the data, continue to follow the instructions and save your Zap - That's it. Now whenever you run that recipe, the scraped data will be sent to Zapier and will trigger the action that you set in the previous step. Easy! If you have any questions about this please reach out to us via chat. You may also find our [Scraping data into Integromat guide](https://simplescraper.io/docs/scraping-data-into-integromat/) useful. ## Scraping data into Make (Integromat) https://simplescraper.io/docs/scraping-data-into-integromat # Scraping website data into Make (formerly Integromat) Simplescraper enables you to scrape website data directly into the automation tool Make (formerly Integromat) to use in your scenarios. It's quick and easy to setup so let's get started. *Follow the steps below or watch the 1 minute video above* - Open the [Make dashboard](https://www.make.com/) and click on the 'Create a New Scenario' button on the right-hand side - In the search box, under 'What services do you want to integrate?', type 'webhooks' and select the webhooks option that appears and then click continue - On the scenario page, click the webhooks module that you just added and select 'Custom webhook' on the menu that appears - Click the 'Add' button on the right-hand side and give the webhook a name and then click save - After clicking save, you will see that a webhook URL has been generated. Copy this URL and jump over to the [Simplescraper dashboard](https://simplescraper.io/dashboard) in a new tab or window and go to the recipe whose data you want to send to Make - Navigate to the Send tab of that recipe and paste the URL that you copied into the webhooks input field and press the enter key to save it - Now click the 'run recipe' button in Simplescraper. Because we've set the webhook URL, the scraped data will be sent to that URL when the recipe finishes. It's time to test that it's working so jump back to Make once the scrape task has completed - Make should have detected the scraped data and will show a 'successfully determined' message. Click the OK button to save your configuration - In Make add the JSON module as a second step and choose JSON parse. On the JSON menu, click on the JSON string field and select value from the Webhook step. This is required in order for the data to be formatted correctly - That's it. You can now add any other modules such as Airtable, Google Sheets, or wherever else you wish to send your data. Whenever you run your recipe the scraped data will be sent to Make and on to those destinations. Easy! If you have any questions about this please reach out to us via chat. You may also find our [Scraping data into Zapier guide](https://simplescraper.io/docs/scraping-data-into-zapier/) useful. ## Scraping data into your email inbox https://simplescraper.io/docs/scraping-data-into-email # Scraping website data into your email inbox Simplescraper's email integration allows you to receive scraped data directly in your inbox as soon as it's been extracted. You can choose to receive a preview of the scraped data, an attachment containing the scraped data - or both. ## Setting up your email integration ![image-auto0crawl](https://simplescraper.io/docs/assets/scrape-email-1.DsK8qHeq.png) To send your scraped data directly to your email inbox: - In the [Simplescraper dashboard](https://simplescraper.io/dashboard), navigate to any of your recipes and click the 'Send' tab - Locate the Email integration and toggle the 'Activate' switch to the 'on' position ## Configuring your Email settings #### Email Address - The email address will always be the address you used when signing up to Simplescraper #### Preview Rows - Set the number of rows you want to preview directly in the email body (maximum 500) - For larger datasets, you'll still be able to access all data via the attachment #### Attachment Settings Simplescraper gives you several options for how your data is attached: - **Include Attachment**: Toggle on to receive the full dataset as an attachment - Note: Even if toggled off, emails will always include a link to view the full results - **Attachment Format**: Choose your preferred file format from the dropdown menu - CSV is the default and works well with spreadsheet applications - JSON and Markdown formats are also available - **Size Threshold (MB)**: Set the maximum attachment size (up to 5MB) - If your data exceeds this size, the email will include a download link instead #### Included Properties - Toggle on/off which data properties you want to include in your email and attachment - Each enabled property will appear as a column in your data #### How emails are structured When you receive a scrape result email, it will include: - A summary of the scrape job (recipe name, date/time, number of records) - A preview of your data (limited to the number of rows you specified) - The full dataset as an attachment (if enabled and below size threshold) - A link to view the complete results in Simplescraper * * * ### Troubleshooting If you're not receiving emails: - Ensure you've activated the Email integration before running your scrape recipe - Check your spam/junk folder as automated emails sometimes get filtered - Verify the 'Activate' toggle is switched on ### Recommended Use Cases The email integration is particularly valuable when: - **Combined with scheduling**: Set up a schedule and receive regular data updates directly to your inbox without any manual intervention - **Paired with [AI enhance](https://simplescraper.io/docs/enhance-scraped-data-using-ai)**: Send summaries or categorized versions of your data via email while sending raw data to Google Sheets, giving you both insights and complete datasets If you have questions about this integration, please reach out via chat. You may also find our [Google Sheets integration](https://simplescraper.io/docs/saving-data-google-sheets/) or [Zapier integration](https://simplescraper.io/docs/scraping-data-into-zapier/) guides useful for other ways to access your data. ## Sending data to a webhook https://simplescraper.io/docs/sending-data-to-webhook # Sending scraped data to a webhook Simplescraper can send your scraped data to any URL as a JSON webhook. When a scrape completes, Simplescraper makes an HTTP POST request to the URL you provide, delivering the results as a JSON array. This works with custom backends, serverless functions, automation platforms, or any service that accepts incoming HTTP requests. ## Setting up a webhook 1. In the [Simplescraper dashboard](https://simplescraper.io/dashboard), navigate to your recipe and click the **Integrations** tab 2. Paste your webhook URL into the webhook input field and press Enter to save 3. Run your recipe. When the scrape finishes, Simplescraper sends the results to that URL ## Webhook format Simplescraper sends a `POST` request with `Content-Type: application/json`. The body is a JSON array of objects, where each object is one scraped row. Here is an example of a webhook payload from a recipe that extracts the time: ``` [ { "time": "04:55:47pm", "time_link": "https://time.is/Yokohama/Yokohama#time_zone", "url": "https://time.is/Yokohama", "index": 1, "timestamp": 1772438149938, "timestampString": "Mon, 02 Mar 2026 07:55:49 GMT", "uid": "NLDQWBAD67RC3WPMtvgP" } ] ``` Each object contains: - **Your scraped properties** (like `time` in the example above). These match the property names you defined in your recipe - **Links** extracted alongside properties, suffixed with `_link` (like `more_info_link`) - **Metadata fields** added automatically: | Field | Description | | --- | --- | | `url` | The page that was scraped | | `url_uid` | Index of the URL in the recipe's URL list | | `index` | Row number within the results | | `timestamp` | Unix timestamp (milliseconds) of when the data was scraped | | `timestampString` | Human-readable date string | | `uid` | Unique identifier for this result | ## When the webhook fires The webhook fires after each URL is scraped and its results are saved. If your recipe scrapes multiple pages, you will receive one webhook POST request per page. When [AI Enhance](https://simplescraper.io/docs/enhance-scraped-data-using-ai) runs in automatic mode, the webhook waits for it. Once AI Enhance finishes, the run's rows are sent together, split into requests of up to 1 MB each. Testing tip: To inspect the exact payload, try [webhook.site](https://webhook.site) - it gives you a free, temporary URL that logs incoming requests with full headers and body. ## Related guides - [Scraping data into Zapier](https://simplescraper.io/docs/scraping-data-into-zapier) - [Scraping data into Make (Integromat)](https://simplescraper.io/docs/scraping-data-into-integromat) - [Scraping data into Google Sheets](https://simplescraper.io/docs/saving-data-google-sheets) - [Extract API v1](https://simplescraper.io/docs/api-guide) ## Need help? Click the chat button in the bottom-right corner to get in touch. ## Adding a password https://simplescraper.io/docs/adding-password-login # Adding a password login Adding a password allows you the option to share your account and no longer depend on Google authentication to login. It takes about 3 minutes to setup and requires that you have first signed up with Google and are logged in to the Simplescraper dashboard. - In the Simplescraper dashboard ([https://simplescraper.io/dashboard](https://simplescraper.io/dashboard)), click your name in the bottom-left corner and then click 'My account' - Click 'setup' beside password in the 'Login methods' section - A popup will appear asking you to reauthenticate your account. This is just like logging in again so click 'Authenticate' and select your Google email just as you would when logging in - You should now be on the 'set a password' screen. Enter a password with a minimum of 6 characters and then reenter it to confirm and then click 'Save password' - You should see a success message telling you that you can now login with your email and the password that you just set - And that's it. Password holders can now login and run scrape recipes on your account ## Your subscription https://simplescraper.io/docs/managing-subscription # Your subscription ## How it works Subscriptions are billed monthly. You can upgrade, downgrade, or cancel at any time from your account. - **Credits** refresh each billing period and are valid for the duration of your subscription. See [Credit usage](https://simplescraper.io/docs/credits) for details on how credits are consumed. - **Cancellation** takes effect at the end of your current billing period. You keep access to your remaining credits until then. - **Your recipes** remain on Simplescraper if you cancel, so you can pick up where you left off should you choose to resubscribe. - **Credit topups** are one-time purchases that stack on top of your subscription credits and are available until your next renewal date. For larger plans or custom jobs, please get in touch via the chat button or email [mike@simplescraper.io](mailto:mike@simplescraper.io). ## Upgrading, adding credits, or changing your plan 1. Visit the [Simplescraper dashboard](https://simplescraper.io/dashboard), click your name in the bottom-left corner of the webpage and then click 'My account' ![image dashboard](https://simplescraper.io/docs/assets/image-20240610094907299.DUnP5fg3.png) 2. Under the Plan section, click the 'change plan' button ![image dashboard](https://simplescraper.io/docs/assets/image-20240610095129127.gbQD_aul.png) 3. A page will appear showing your current plan, and an option to update your subscription or top up your credits. Depending on your preferences, click on 'Update plan' or 'Get credits'. ![image dashboard](https://simplescraper.io/docs/assets/image-20240610095511370.CKvn2YiA.png) * * * ![image dashboard](https://simplescraper.io/docs/assets/image-20240610095714248.DT_HBpY7.png) 4\. A popup will open allowing you to add your payment details and subscribe / top up ![image dashboard](https://simplescraper.io/docs/assets/image-20240610104104387.Dn29mpaG.png) ## Delete account https://simplescraper.io/docs/delete-account # Delete account You can delete your account at any time from the dashboard. Please note this is irreversible and cannot be undone. - In the Simplescraper dashboard ([https://simplescraper.io/dashboard](https://simplescraper.io/dashboard)), click your name in the bottom-left corner and then click 'My account' - Scroll to the bottom of the page to the 'Delete account' section - Click 'Delete my account' and you'll be asked to type 'delete' to confirm - Click 'Confirm deletion' to permanently remove your account - You'll be signed out automatically after a few seconds Any active subscription is cancelled at the same time.