Find every page on a site. In seconds · with no credits used.
Enter a website and Simplescraper reads its sitemap, nested sitemaps included, and lists every page URL. Copy the list, download it, or give it to a scraper.
- Looking for a sitemap
- Found sitemap_index.xml · 6 sitemaps
- Reading sitemaps
Sitemap discovery
Simplescraper finds the sitemap from the usual paths and robots.txt, then follows a sitemap index into every nested sitemap.
Free URL extraction
Listing URLs never touches your credit balance. A subscription allows 500 requests a day and the free tier 50.
Partial results on large sites
A request runs for up to 60 seconds. A very large site returns the URLs found so far, marked time_limit_reached.
The URL extractor API
Send a domain to the extract-urls endpoint and get every URL in its sitemap back as JSON. Cap the list with urlLimit, or sample a very large site with sitemapLimit.
curl -X POST https://api.simplescraper.io/v1/extract-urls \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "domain": "https://lumensupply.com", "urlLimit": 5000 }'
The domain, with its protocol
Only domain is required. Authenticate with the same Bearer key as every other endpoint.
{
"success": true,
"data": {
"sitemapUrl": "https://lumensupply.com/sitemap_index.xml",
"urls": [
"https://lumensupply.com/products/trailhead-40l-pack",
"https://lumensupply.com/products/ridgeline-rain-shell",
"https://lumensupply.com/collections/packs"
// one entry for every URL in every sitemap
],
"status": "completed"
}
}Every URL in one array
Duplicates across nested sitemaps are merged. The sitemap that was found comes back too.
| Status | What it means | What you get |
|---|---|---|
completed | The sitemap was found and read to the end. | Every URL, up to your urlLimit. |
no_sitemap_found | No sitemap at the usual paths or in robots.txt. | No URLs, and a message pointing to deep scraping instead. |
time_limit_reached | The site was still being read after 60 seconds. | The URLs found so far. |
Agents map the site first.
Over MCP the same discovery is the extract_urls tool, and it is free there too. An assistant finds the pages, then points a scraper or Smart Extract at the ones that matter. AI agents over MCP
extract_urls· lumensupply.com · 3,412 URLs · 0 creditsHow URL extraction works
The sitemap is the site's own list of its pages, so there is nothing to configure and nothing to wait for.
Enter a website
Type a domain into the URL extractor, or send it to the API with your key.
Get every URL
Copy the list, download it as CSV, or paste it into a scraper's URL list to scrape every page.
FAQ
How do I extract all links from a website?
Paste a website's address into the URL extractor and it lists every page it can find, from the site's sitemap. Filter the list by any part of the address, such as a domain or /blog/, then copy it or download it. Useful for building sitemaps or finding every product in a category.
How many links can the URL extractor find?
There is no cap unless you set one. A request runs for up to 60 seconds and returns every URL found by then. Through the API you get 50 requests a day on the free tier and 500 with a subscription.
Can I extract specific types of links only?
Yes. Filter the list by any part of the address, for example /blog/ for internal blog posts or /products/ for the product pages of one category.
Extract every URL on a website today.
Every URL in a site's sitemap for free, and 100 credits to scrape the pages you pick.
Start free