How to Compare Multiple Competitor Websites with Simplescraper MCP

As a founder, marketer, or product person, competitive research tells you where your competitors are spending their marketing effort, which customer segments they're hunting, and how they're positioning their product offerings.

If done well, it's a goldmine of actionable information. But the steps required are time consuming:

• Open each competitor's site
• Find their sitemap (not always easy)
• Paste the URLs into a spreadsheet and filter to the pages you actually care about
• Scrape each page
• Manually build comparison tables

The new, free Extract URLs feature in the Simplescraper MCP server makes this process easy. From any AI tool you can extract every URL from any website to analyse immediately. So instead of a tiring process, a workflow now looks like this:

1. Pull every URL from a competitor's site (Simplescraper MCP)
2. Map each site's structure (AI)
3. Scrape the pages that matter for a deeper read (Simplescraper MCP)
4. Compare content strategies side by side (AI)
5. Understand your competitors and improve your business

To show the full benefits of this workflow, we'll choose a popular SaaS category, social media scheduling tools, and run a real teardown across four businesses within this category:

• Postiz - open-source, AI-first scheduling to 30+ networks
• SocialPilot - all-in-one management and analytics
• Planable - collaboration and approval workflows
• Zernio - developer-first social posting API across 15+ platforms

Our goal is simple: how much can we learn about these companies from their URLs, and how valuable is the information?

You'll need an AI tool like Claude Code (which we'll use in this example), Cursor, or Codex, a free Simplescraper account, and the Simplescraper MCP - we'll cover the install steps below.

Step 1: Pull All URLs From Each Site

Here's the first prompt:

"Extract all the URLs from postiz.com, socialpilot.co, planable.io and zernio.com."

Claude reads that, picks the right MCP tool (extract_urls), and calls it four times - once per domain. Each call finds the site's sitemap and returns the full URL list.

The response Claude hands you back:

Site	URLs in sitemap
postiz.com	3,046
socialpilot.co	2,834
planable.io	609
zernio.com	288

The numbers give you the rough scale, but what each site actually invests in lives one layer deeper, in the URL paths themselves.

Step 2: See What Each Site Invests In

The natural next move is to have Claude sort those URLs into a shape we can read - grouped by their top-level path, with localized copies folded together, so we can see per-site where the page count is concentrated.

"For each site, group the URLs by top-level path and show me the top 6 sections per site with counts."

Here is what comes back, one site at a time.

postiz.com

• 3,046 total URLs
• English only, no localization

Path	URLs
/compare	2,080
/blog	425
/tools	244
/alternatives	64
/channels	30
/claude-cowork	26

socialpilot.co

• 2,834 total URLs
• 1,349 English, 1,485 across 5 locales (de, es, fr, it, pt)

Path	URLs
/blog	369
/reviews	276
/compare	246
/social-media-terms	158
/product-update	155
/tools	126

planable.io

• 609 total URLs
• 573 English, 36 across 3 locales (de, es, fr)

Path	URLs
/blog	480
/guides	17
/webinars	14
/customers	8
/resources	3
/ai-tools	1

zernio.com

• 288 total URLs
• English only, no localization

Path	URLs
/blog	189
/changelog	34
/alternatives	22
/social-media-tips	15
/tiktok-api	1
/instagram-api	1

A few things stand out right away.

• Postiz is overwhelmingly a comparison-page play. 2,080 /compare/ URLs make up 68% of its entire sitemap, which is a clear bet on long-tail SEO around competitor matchups. SocialPilot is running a smaller version of the same idea, with 246 /compare/ plus 276 /reviews/.
• Planable is the exact opposite shape. 480 of its 609 URLs - around 79% - are blog posts. It's the same product category with almost the inverse strategy.
• All four publish blogs at meaningful scale. Between 188 and 480 posts each, which tells you that editorial is table stakes in this niche rather than a differentiator.
• Localization is light across the four. Only SocialPilot ships meaningfully across locales, Planable touches three, and Postiz and Zernio are English-only. That leaves a lot of empty international shelf space for anyone willing to translate.

All of that comes from the URL structure alone, before we have scraped a single page. By hand, the same exercise would mean sorting 6,777 links and squinting at prefixes, and a 3,000-page site would take ten times as long as a 300-page one. Here, both reduce to the same six-row table.

Step 3: Find What They All Write About, Then Read One Side by Side

Each URL from Step 1 already includes a slug that summarizes its post in three or four words. That's enough for Claude to map themes across the four blogs without us scraping anything yet.

"Look at the blog slugs across the four sites. What themes does each one focus on, and which topics do all four of them cover?"

Each site has its own emphasis, but the overlap is real. All four cover Instagram, LinkedIn, Facebook, and TikTok at scale, and all four publish on scheduling, content, strategy, tools, and management. The differences are angle rather than subject - Postiz mixes automation and tactics, SocialPilot leans growth and marketing, Planable leans workflow and approval, Zernio leans API and code. None of that required us to read a single post.

Now that we know what's covered across the board, we can pick a single topic every one of them publishes on and read what each says about it side by side.

"Each of these four has a post about scheduling LinkedIn posts. Find the closest match in each sitemap, pull its markdown, and tell me how each one approaches the topic."

Two of the four come back cleanly and two don't. Postiz times out - its stack throws up the kind of defenses some Webflow plus Cloudflare sites aim at automated browsers - and SocialPilot's post returns an error mid-fetch. Planable and Zernio both deliver substantive articles, and the contrast between them is where the value of this whole exercise shows up.

Planable's "Schedule LinkedIn posts: 2 quickest ways to save time" (around 2,400 words) is a marketing comparison piece. It walks through LinkedIn's native scheduler and then Planable's, side by side, with a feature-by-feature table that quietly makes the case for Planable on every row. The article is full of screenshots, eleven reasons to use it, and customer quotes pulled from G2 reviews. The pitch underneath everything is "6X faster" scheduling for teams that need approval and multi-platform support.

Zernio's "A Modern Guide to Schedule LinkedIn Posts and Maximize Your Reach" (around 3,640 words) is a developer guide. It covers four methods - the native scheduler, Zernio's REST API, Zernio's dashboard, and no-code automation via Zapier or n8n - and the API section has working Python and Node.js code that schedules a post in about twenty lines each. There's a comparison table here too, but it's organized by technical skill level rather than feature checkmarks. The pitch underneath everything is one unified API endpoint across LinkedIn and nine other platforms.

Both posts target the same keyword, in the same product category, with the same intent to rank for "schedule LinkedIn posts" - but each is written for a completely different reader. Planable is selling collaboration to marketing teams. Zernio is selling automation to developers. The structural choices each one makes - product-led screenshots versus working code samples, G2 quotes versus comparison-by-skill-level - say more about who they think their best customer is than any positioning page ever would.

This is the part that's genuinely awkward to do without MCP. The old way meant writing a script, looping the API calls, storing the results, and handling both retries and the pages that simply block you. The new way is to wait under a minute and let Claude hold whatever came back, ready to compare.

Step 4: Build the Report

By this point the 6,777 URLs have already been collapsed into the structure map from Step 2, and what's sitting in Claude's context is that map plus the two posts that scraped cleanly - not the raw links. Now we ask it to pull everything together into one report.

"Build me a competitor research report from everything we have. Include a comparison table with total URLs per site, top sections each is investing in, and the share that's localized. Then give me five signals I should act on."

In Claude.ai you can ask for it as a rendered HTML artifact and open it in a browser; in Claude Desktop or Cursor you get the same content as Markdown. Either way, here's what it produced for this run.

Scale and investment

Site	Total URLs	English	Locales	Top 3 sections
Postiz	3,046	3,046	0	/compare 2,080, /blog 425, /tools 244
SocialPilot	2,834	1,349	5	/blog 369, /reviews 276, /compare 246
Planable	609	573	3	/blog 480, /guides 17, /webinars 14
Zernio	288	288	0	/blog 189, /changelog 34, /alternatives 22

Five signals

1. Postiz and SocialPilot run the same comparison-SEO play at different intensities.
   • Postiz is 68% /compare/ pages.
   • SocialPilot mixes /compare/, /reviews/, and /social-media-terms/ for roughly 700 long-tail pages.
   • The bet is already proven, so anyone entering this category late will have to pick a path through it.
2. Planable is the mirror image: 79% editorial.
   • Almost the entire sitemap is blog posts.
   • It's the same category with the opposite bet, and both seem to work.
3. Blog scale is table stakes, but only two of the four blogs are readable.
   • All four publish between 188 and 480 posts.
   • Postiz blocks automated browsers and SocialPilot returns error pages, which suggests they're counting on something other than organic search for that content - paid social, trust signals, or simple anti-bot defaults.
4. Localization is empty space here.
   • Only SocialPilot ships more than a handful of locales.
   • Whoever translates seriously has a category-level wedge nobody else is taking.
5. Postiz is courting AI tool users specifically.
   • Pages at /claude, /codex, /claude-cowork/*, /openclaw, and /hermes-agent are landing pages built around AI coding tools, not social networks.
   • That's a tell about who they think their next sign-ups are coming from.

That's the comparison from one prompt. You can drill into anything that catches your eye - "show me 10 random URLs from /compare" or "who is Postiz comparing themselves against across 2,080 pages?" - it's all in the same conversation, with as much depth as you want.

Step 5: Run This on Your Own Market

The chain is the same regardless of category. Whether you're in SaaS, e-commerce, dev tools, or a niche marketplace, the play stays the same: pick four or five competitors, install the Simplescraper MCP, and run the same prompts.

Five questions worth asking each time:

1. Where is each competitor's URL volume concentrated? A site with thousands of pages in one section is making a deliberate bet. Find out what.
2. Who localizes, and into what? International SEO is a signal of where they think growth is. Match it, skip it, or pick the locales they missed.
3. What programmatic patterns are running? /alternatives/, /vs/, /compare/, /tools/[niche], /agent/, /[city] - if anyone in your space is doing this, it's the lowest-effort SEO bet someone has already proven works.
4. What's their blog cadence and what topics dominate? Look at posts per month, top recurring terms, and recent shifts. They tell you what each competitor is optimizing for.
5. What's the gap? Sections every competitor has except one, locales nobody translates into, topics nobody owns. Those gaps are where you can move first.

Each question is one prompt against the URLs you've already loaded, so the whole chain runs in a single conversation. The patterns tend to surface inside an hour.

Install the Simplescraper MCP

There are three quick install paths, depending on which client you use.

Claude Desktop. Open Settings → Developer → Edit Config, then add Simplescraper to your claude_desktop_config.json:

{
  "mcpServers": {
    "simplescraper": {
      "command": "npx",
      "args": ["-y", "@simplescraper/mcp"],
      "env": { "SIMPLESCRAPER_API_KEY": "YOUR_API_KEY" }
    }
  }
}

Once you restart Claude Desktop, the Simplescraper tools will show up in the MCP picker automatically.

Cursor. Add the same config block to your Cursor MCP settings (Settings → MCP → Add new server).

Codex. Run codex mcp add simplescraper --env SIMPLESCRAPER_API_KEY=YOUR_API_KEY -- npx -y @simplescraper/mcp in your terminal.

Your API key sits on the Simplescraper account page - the free tier covers URL extraction in full, and paid tiers add credits when you start scraping pages.

Want the URL Extractor on its own? See the URL Extractor API docs. Want the full MCP server setup? Read the Simplescraper MCP guide.

---

That's the whole loop. Four sitemaps pulled, paths grouped, two posts read side by side, one report at the end, all inside a single Claude conversation. It works on any market. Pick four competitors, paste them into the first prompt, and see what surfaces.