Simplescraper

Sub-page (deep) scraping ​

When the data you need sits on the pages a list links to, such as the details behind every posting on a job board, scrape the list of links first and then the same fields from every linked page. This is sub-page (deep) scraping, and it runs in your browser with no saved recipe.

How it works

  1. Scrape the list of URLs you want details from.
  2. Visit one of those URLs and pick the data you want.
  3. Scrape all the URLs the same way.

Scrape every page a list links to ​

  1. On the page with the list, open the extension and scrape the list. Its results need a column of links, such as each job's URL.

    The Results card of a job list scrape, 50 rows, with the job_url column of links outlined

  2. Open one of the linked pages and click 'Single fields'.

  3. Click each value you want from the page, such as the job title, location and description, then click 'Continue'.

    A job page with its title, location and description outlined, and the Single fields tab listing the three fields above the Continue button

  4. Check the field names and click 'Use these fields'.

    The Confirm fields step with program_title, job_location and application_description, and the AI rename and Use these fields buttons outlined

  5. Under 'Choose pages to scrape', choose 'A list of URLs' and click 'Import from results'.

    The Choose pages to scrape step with A list of URLs chosen, an empty URL box and the Import from results button outlined

  6. Under 'Run', pick the scrape that collected the links. The list holds every scrape in this browser that kept a column of links, saved or not, and opens on the newest one from the same site. If that scrape has more than one column of links, choose one under 'Links column'.

    The Import from results sheet with the List 1 job-boards.greenhouse.io run chosen, three of its links previewed and the Add 50 URLs button outlined

  7. Click 'Add URLs', then 'Scrape URLs'.

    The URL box filled with 50 job links, with the 50 URLs count and the Scrape 50 URLs button outlinedThe Results card showing 6 rows from 6 URLs with program_title, job_location and application_description columns, and the CSV and JSON buttons outlined

To download the rows, or to resume a run you stopped, see Download results.

Repeat it later ​

  • Save it as a recipe: click 'Save as recipe' when the run finishes. The recipe keeps its list of URLs, so you can run it again from 'My recipes'.
  • Use a detail recipe you already saved: open it from 'My recipes', choose 'A list of URLs', import the links with 'Import from results' and click 'Run now'.
  • Run it on a schedule or with your browser closed: see Sub-page (deep) scraping in the cloud.

For questions or help, reach out via chat.