How to Scrape Shopify Data: a no-code and developer guide ​

cover shopify


Shopify powers over 4 million active stores, ranging from niche boutiques to global giants like Allbirds and Gymshark.

For marketers researching competitors, entrepreneurs seeking inspiration, or analysts tracking e-commerce trends, the data within these stores – product details, pricing, inventory levels – is a goldmine. But how do you access it quickly and efficiently?

In this guide, we'll walk through the process of extracting this data from Shopify stores using Simplescraper's easy to use no-code scraping tools. By the end, you’ll have a working workflow that pulls product data automatically. We’ll also cover options for developers needing API access.


Table of contents

Method 1: Using products.json (Quick but Limited) ​

It's a bit of an open secret that many Shopify stores have an easy way to extract basic product data: the products.json endpoint. Simply add /products.json to the main domain of a Shopify store to access a JSON feed of products.

For example try opening this URL in your browser: https://allbirds.co.uk/products.json

This method has some advantages:

  • Super fast setup (just one URL)
  • Returns structured data already in JSON format
  • Includes basic product information like titles, prices, and variants

But it also has limitations:

  • Only returns the first 250 products
  • Doesn't include all product details (often missing descriptions, images, etc)
  • Many regional Shopify stores may not expose this endpoint

If you need more detailed data, require more than 250 products, or find the endpoint unavailable, the products.json method falls short. For complete data extraction, our next method - using a dedicated web scraper - works better.

Method 2: Collecting and Scraping Shopify URLs ​

This approach involves identifying the Shopify data you care about and instructing a web scraper to retrieve it.

It gives you complete control over what data to extract, with no limitations on the number of products or types of information you can retrieve. We'll use Simplescraper's no-code web scraping platform to make the process quick and easy.

Our no-code data scraping toolkit ​

Simplescraper offers several tools specifically suited for Shopify data extraction:

  • URL Extractor: To quickly gather all product URLs from a store
  • Smart Extract: AI-powered tool that identifies and extracts specific data points
  • Batch Scraper: To scrape hundreds or thousands of product pages at once
  • AI Enhance: To analyze extracted data for insights, summaries, or transformations

These tools work together to remove the hassle of web scraping and make scaling to thousands of pages as simple as scraping a single page - all without needing to write a line of code.

Let's see these tools in action.

Step 1: Scraping Product URLs ​


extract urls

First, we need to gather all the product URLs from the Shopify store we wish to scrape. We'll use Allbirds UK (https://www.allbirds.co.uk) as our example:

  1. Visit Simplescraper's URL extractor at https://simplescraper.io/extracturls
  2. Enter the store's main URL (in our case, https://www.allbirds.co.uk)
  3. Click "Extract URLs" and wait a few seconds for the results
  4. Use the filter option to show only product URLs (look for patterns like /products/ in the URLs)
  5. Click "Download TXT" or "Copy to clipboard" to save the list for the next step

You should now have a list of all product URLs from the store, ready for extraction.

Step 2: Create a Scrape Recipe ​


create recipe

Now that we have all product URLs, let's create a recipe to extract specific data from each product page:

  1. Visit any product page from your list of URLs (e.g., https://www.allbirds.co.uk/products/womens-natural-run-long-sleeve-tee-hazy-cargo)

  2. Navigate to Simplescraper's Smart Extract (or just type scrape.new in your browser)

  3. Enter the product URL and a data schema listing what you want to extract. For example:

    product name, current price, previous price, product description

  4. Click "Extract Data" and wait a few seconds for results

  5. Review the extracted data to ensure it captured everything correctly

  6. Click "Save as scrape recipe" to save your extraction configuration as a recipe on the Simplescraper dashboard

The Smart Extract tool uses AI to identify the exact elements on the page that contain your requested information. If any data points are missing or incorrect, you can refine your schema and try again, or use the Simplescraper extension for manual selection.

Step 3: Batch Scrape All Products ​


batch scrape

With our extraction recipe created, we can now apply it to all product URLs:

  1. In the Simplescraper dashboard where you saved your recipe, click on the recipe you just created
  2. Navigate to the "Crawl" tab
  3. Copy and paste all the product URLs you downloaded in Step 1
  4. Click "Run recipe" to start the batch extraction process

Simplescraper will now visit each product URL and extract all the data points you specified. Depending on the number of products, this might take a few minutes.

Step 4: Analyze Your Results ​


batch results

Once the batch scrape completes, you'll have a comprehensive dataset of all products from the store. To analyze this information:

  1. Download your results as CSV or JSON
  2. Import the data into your analysis tool of choice (Excel, Google Sheets, Airtable, etc.)
  3. Start analyzing patterns, pricing strategies, and other insights

Alternatively, you can use Simplescraper's AI Enhance feature to automatically analyze the data:

  1. Go to the "AI Enhance" tab in your recipe
  2. Toggle "Activate" to on
  3. Choose a prompt like "Analyze pricing patterns and identify discount strategies" or create your own custom prompt
  4. Select which data properties to include in the analysis
  5. Click "Run AI" to generate insights

For Developers: API Integration ​


batch results

For developers or those with more technical requirements, Simplescraper also offers API access to programmatically extract Shopify data. To scrape via API:

  1. In the dashboard, visit the recipe you created and click on the "API" tab to find your API endpoint
  2. Make a POST request to this endpoint to run the extraction programmatically
  3. Include parameters like sourceUrl to change the target URL or runAsync for larger jobs

Here's a simple example using JavaScript:

javascript
// an example of how to call the Simplescraper API
async function extractShopifyData(apiKey, recipeId, shopifyUrl) {
  const url = `https://api.simplescraper.io/v1/recipes/${recipeId}/run`;
  
  const requestBody = {
    sourceUrl: shopifyUrl,
    runAsync: false
  };
  
  try {
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(requestBody)
    });
    
    const data = await response.json();
    return data;
  } catch (error) {
    console.error('Error extracting data:', error);
    return null;
  }
}

// call the function
extractShopifyData('your-api-key', 'your-recipe-id', 'https://allbirds.co.uk/products/womens-wool-runner-up-mizzles-natural-black');

This gives you the flexibility to integrate Shopify data extraction into your own applications or workflows. More info Simplescraper's API can be found here.

Automating your Webscraping Workflow ​

Now that you've mastered basic Shopify data extraction, you can take things further and build a comprehensive scraping workflows that run on autopilot:

  • Scrape automatically: Schedule your recipe to run daily or weekly to track price changes, new products, or inventory fluctuations
  • Competitive Analysis: Create multiple recipes to extract data from multiple Shopify stores and compare them side-by-side
  • Integration with Other Tools: Send your extracted data automatically to tools like Airtable, Google Sheets, or Zapier
  • Custom Analysis: Use the AI Enhance feature with custom prompts to generate sophisticated insights

Conclusion ​

Scraping Shopify data doesn't require code or complexity - in fact, it's surprisingly simple. In just a few steps, you can build a reliable workflow that runs automatically, scales easily, and can be repeated on virtually any website.

Whether you're tracking prices, analyzing trends, or just curious - you’re now equipped to extract the data you need. If you have questions about this guide or web scraping in general, feel free to reach out via our chat.

Happy scraping!

Links mentioned in this article:

More articles:

How to build an automatic trading bot with Simplescraper and GPT-4 Vision

Building a free beach-chair detector using Simplescraper, LLMs and public webcams

Build with Simplescraper

Turn websites into structured data in seconds.

Try it out