Sub-page (deep) scraping
Scrape a list of links, then automatically extract data from each linked page.
Sub-page (deep) scraping lets you scrape a list of links from a main webpage and then collect data from each page those links lead to. This is useful for extracting data from platforms like job boards or hotel listings, where each listing's details are on individual subpages.
To accomplish this, you'll use two recipes: one recipe to scrape the links and another recipe to extract the data behind each link.
To scrape the sub-pages from your browser without saving a recipe, see Sub-page (deep) scraping in the browser.
How to use sub-page scraping
First Recipe: Create and save a recipe that scrapes a complete list of links that you're interested in. For example, if you're scraping job listings, this recipe scrapes the link of each job post
Second Recipe: Create and save another recipe that will scrape the content behind a single link from this list. For instance, if you're scraping job listings, this second recipe would be set to extract job information (salary, job description, requirements etc) from a single job description page
Run and Import: Run the first recipe via cloud scraping to get a list of links. Import these links into your second recipe from its 'Bulk Scrape' tab: click 'Options', then 'Import URLs'. Then, run the second recipe to scrape the content from each link
Note: When extracting a property that includes a link, Simplescraper will automatically save this link to a column named 'propertyname_link'.
Example: Shopify App Store sub-page scraping
To scrape app links and their contact details from the Shopify App Store, you'll use two Simplescraper recipes:
First, create a recipe to scrape the list of app links available on
https://apps.shopify.com/Then create a second recipe focused on extracting contact details from a single app page
Save the first recipe as 'Shopify - app links' and the second as 'Shopify - contact info'
The video below shows the first step on a job board: scraping a list and its column of links with the extension.
Once both recipes are saved, start by running 'Shopify - app links'. The result will be a column filled with the extracted links.
Next, open the 'Shopify - contact info' recipe, go to its 'Bulk Scrape' tab, click 'Options' and choose 'Import URLs'.
Under 'Import URLs from', select 'Shopify - app links'. Under 'Properties with links', select the property containing the links (in this example, 'App link'), then click 'Confirm'. You're set to begin sub-page scraping.
Now all you need to do is click 'Run recipe' on the second recipe and the contact details for each of the apps will be scraped.
The video below shows the steps to import the links from the first recipe into the second recipe.
Automatically running an imported recipe
When running a recipe that is importing links from another recipe you may wish to run the imported recipe first so that the most recent links are scraped.
To do this automatically, open the 'Bulk Scrape' tab of the recipe that is importing the links (the 'second recipe') and switch on 'Run imported recipe first'. This ensures the link-collecting recipe runs automatically before scraping the individual pages, keeping your scraped data current.
Note:
- 'Run imported recipe first' only shows on the 'Bulk Scrape' tab once a recipe has been imported
- 'Run imported recipe first' is not applicable for requests made via the API. It's best suited for recipes that are scheduled or run manually from the dashboard. To recreate this via the API, call the first recipe and the second recipe sequentially