Extracting Websites as Markdown data

Convert any webpage into clean Markdown — ideal for AI analysis.

Simplescraper enables you to easily extract and save a webpage, or an entire website's text content, in Markdown format. Markdown retains the page formatting and is a preferred format when analyzing web date using AI models such as OpenAI's ChatGPT and Anthropic's Claude.

There are a number of ways to save website data as Markdown format with Simplescraper:

Via a website crawl

1. Open the Simplescraper dashboard and click 'New crawl' in the sidebar.
2. Under 'Website to crawl', enter the URL of the website you want to save as Markdown.
3. To change how much of the site is crawled, click 'Options'. While it is closed, the 'Options' row lists the current settings. Open, it has three:
   • Page limit: the most pages the crawl will save.
   • Only crawl paths: restricts the crawl to matching URLs, for example /blog/. Use * to match anything.
   • Collect URLs: also saves the links found on each page (see Extracting URLs).
4. Click 'Test' to crawl two pages first, and check that the data looks right.
5. Click 'Run', check the page and credit estimate, then click 'Confirm'. 'Crawl results' opens straight away, shows the page being read and counts pages as they are saved.
6. When the crawl finishes, choose how the pages are packaged:
   • One file per page: each page is its own file, and the files download together as a .zip.
   • Combine pages into one: every page goes into one file per format.
7. Click 'Markdown', 'JSON' or 'CSV' to download.

Continuing a crawl

If you set a page limit or ran out of credits and want to scrape more of the site:

1. On the crawl page, open the 'Crawl history' tab and click 'Continue' next to the crawl.
2. The website's URL and its settings load. Click 'Options' and raise 'Page limit' to the new total. For example, if you scraped 1000 pages before and now want 3000, enter 3000.
3. Click 'Run', then 'Confirm'. Simplescraper scrapes only the pages that were not scraped before.
4. When the crawl finishes, the downloads include every page from both runs.

Via a list of URLs

If you already know exactly which pages you want, there's no need to crawl:

• Open 'Get new data' in the sidebar, then select 'Scrape webpages'
• Choose 'Multiple URLs' and paste one URL per line (a single page works the same way via 'Single URL')
• Under 'Data required', select 'Markdown'
• Click 'Get data' and Simplescraper will save each page as clean Markdown, with the same Markdown, JSON, and CSV download options when the run completes

Via a scrape recipe

• When saving a scrape recipe (this guide covers saving recipes), click the Advanced options section and toggle the 'Extract Markdown' button to the on position

  • 

• Run your recipe and the Markdown will appear in its own column and a 'download Markdown' button will be available

  • 

• Note that if the Markdown is very large (over 10MB), the file will be downloaded as a zip

Via the API

• When calling the Simplescraper API, include extractMarkdown: true in the body of the request

  • const apikey = 'ap1k3y';
    
    const requestBody = {
      extractMarkdown: true,
    };
    
    const response = fetch(`https://api.simplescraper.io/v1/recipes/${recipeId}/run`, {
      method: 'POST',
      headers: {
          'Authorization': `Bearer ${apikey}`,
          'Content-Type': 'application/json'
      },
      body: JSON.stringify(requestBody)
    })

• Please read the full API guide for more details on data extraction via the API: https://simplescraper.io/docs/api-guide

Extracting URLs

A website crawl can also save every URL it finds on the pages it crawls. Click 'Options' on the crawl page, then under 'Collect URLs' choose which URLs to save:

• Do not collect URLs: no URLs are saved.
• Internal URLs: URLs on the same domain as the website.
• External URLs: URLs on other domains.
• All URLs: every URL found on the crawled pages, internal and external.

When the crawl finishes, the result says how many URLs were collected, and they download as a CSV from the 'URLs' button beside the Markdown, JSON and CSV downloads.