Note: this section involves crawling a list of URLs so be sure to also read the Crawling a List of URLs guide
You may find yourself wanting to scrape a list of URLs and then the contents behind each of those individual URLs. For example a job board or hotel listings website where we aim to extract the content behind every link in a list of results.
Here's how to accomplish this using Simplescraper:
So we use two recipes. The first recipe returns a list of URLs and the second recipe scrapes each of those individual URLs.
A detailed example:
When running a recipe that is importing URLs from another recipe you may wish to run the imported recipe first so that the most recent URLs are scraped.
To do this automatically, navigate to the crawl tab of the recipe that is importing the URLs and toggle the 'run imported recipe first' option. Now any time you run that recipe it will first run the imported recipe so that the latest URLs are always scraped.
Note that this option only becomes visible on the crawl tab once a recipe has been imported.