Crawling lists of URLs

The crawler allows you to paste up to 5000 URLs at a time to be scraped with Simplescraper. This method is recommended as it is faster than navigating individually through pages.

To use the crawler, save a recipe as normal and then click the 'crawl' tab and paste the URLs that you wish to scrape into the text area. Simplescraper will detect that there are URLs in the crawler and will scrape using these URLs instead of the original URL and pagination settings.

If you wish to scrape multiple pages but only have the URL of the first page, click to the second page of the website and note the URL. There should be a structured pattern like 'page=2', 'p=2', 'page/2' etc. This value should be incremented for each page that you intend to scrape. For example if you're scraping 100 pages then the last URL will contain a value like 'page=100'. Contact us via chat if you need help with this.

Alternatively you can import a list of URLs that you scraped with a different recipe as explained in this Deep Scraping guide.