How to detect if you've been soft-blocked while scraping
If you're checking only the status code to tell whether a scrape worked, you're probably letting blocks slip through as successes. A site that does not want your traffic often does not return a 403. It returns a 200 with a Cloudflare interstitial, a Datadome "Please enable JS" page, or a near-empty shell where the article used to be, and your pipeline stores that garbage as a good result. By the time you notice, you have a few thousand rows of "Just a moment..." in your database.
The fix is to treat detection as three signals read together rather than one status code. We'll build a small script that reads both the response headers and the full body so a 200-OK challenge page can't slip through, scores the status code while treating 200 as inconclusive on its own, weighs the body against a floor because a challenge page is far smaller than the article you expected, scans body and headers for known vendor fingerprints, and combines the three into a single verdict with the reasons behind it so a caller can log, retry, or escalate. A response only counts as a soft block when the signals line up, which is what keeps a genuinely short page from tripping the check. It runs in about 70 lines of Node.js with no dependencies beyond the built-in fetch.
The complete script
// detect-soft-block.mjs
/* known challenge-page fingerprints, keyed by the anti-bot vendor that emits them.
each marker is a string that appears in the block page's HTML or headers and
is unlikely to appear in legitimate article content. extend per target site. */
const CHALLENGE_FINGERPRINTS = {
cloudflare: ['cf-chl-', 'Just a moment...', 'Checking your browser before accessing', '/cdn-cgi/challenge-platform/'],
datadome: ['datadome', 'dd_cookie', 'geo.captcha-delivery.com'],
perimeterx: ['_px', 'Access to this page has been denied', 'px-captcha'],
imperva: ['Incapsula incident', '_Incapsula_Resource', 'Request unsuccessful'],
akamai: ['ak_bmsc', 'Access Denied', 'Reference '],
generic: ['Attention Required', 'Enable JavaScript and cookies to continue', 'unusual traffic from your computer']
}
/* status codes that lean toward a block on their own. 200 is deliberately absent:
a soft block hides behind 200, so a 200 is inconclusive until the body is read. */
const BLOCK_STATUS = new Set([401, 403, 429, 503])
/* below this many bytes, a page that should carry an article is suspiciously thin.
tune per target: a normal article body is tens of kilobytes, a challenge page is a few. */
const MIN_BODY_BYTES = 2000
/* scan body text and headers for any vendor fingerprint. returns the matched
vendor and marker, or null when nothing matches. */
function matchFingerprint(bodyText, headerBlob) {
const haystack = bodyText + '\n' + headerBlob
for (const [vendor, markers] of Object.entries(CHALLENGE_FINGERPRINTS)) {
for (const marker of markers) {
if (haystack.includes(marker)) return { vendor, marker }
}
}
return null
}
/* read the three signals and combine them into one verdict.
blocked is true only when the signals corroborate, not on any single weak hint. */
async function detectSoftBlock(url) {
const res = await fetch(url, {
headers: { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' },
redirect: 'follow'
})
const body = await res.text()
const headerBlob = [...res.headers].map(([k, v]) => k + ': ' + v).join('\n')
/* signal 1: status code. a block-leaning code is strong but not sufficient. */
const statusBlocked = BLOCK_STATUS.has(res.status)
/* signal 2: body size. a thin body where you expected an article is a hint, not a verdict. */
const bodyBytes = Buffer.byteLength(body, 'utf8')
const bodyThin = bodyBytes < MIN_BODY_BYTES
/* signal 3: content fingerprint. a vendor marker in the body or headers is the strongest signal. */
const fingerprint = matchFingerprint(body, headerBlob)
/* a Cloudflare-specific header that flags an active challenge before the body is parsed. */
const cfMitigated = res.headers.get('cf-mitigated') === 'challenge'
/* combine: a fingerprint or the cf-mitigated header is decisive on its own;
otherwise a block-leaning status backed by a thin body counts as a block. */
const blocked = Boolean(fingerprint) || cfMitigated || (statusBlocked && bodyThin)
const reasons = []
if (fingerprint) reasons.push('fingerprint:' + fingerprint.vendor + ':' + fingerprint.marker)
if (cfMitigated) reasons.push('header:cf-mitigated=challenge')
if (statusBlocked) reasons.push('status:' + res.status)
if (bodyThin) reasons.push('body-thin:' + bodyBytes + 'b')
return { url, blocked, status: res.status, bodyBytes, vendor: fingerprint?.vendor ?? null, reasons }
}
const verdict = await detectSoftBlock('https://en.wikipedia.org/wiki/Web_scraping')
console.log(JSON.stringify(verdict, null, 2))
if (verdict.blocked) process.exitCode = 1node detect-soft-block.mjsHow it works
Read the body and the headers together. A status-only check throws away the two signals that catch a 200-OK block. The script awaits res.text() for the body and flattens res.headers into a single string so the fingerprint scan can match against header values like Server: cloudflare and cf-mitigated, not just the HTML. One redirect trap to know about: with redirect: 'follow', a 403 that bounces to a challenge URL surfaces as the final hop's 200, so res.status reports the challenge page's code rather than the block, and the fingerprint and cf-mitigated checks below catch it regardless; set redirect: 'manual' if you need the original status.
Score the status code without trusting it alone. BLOCK_STATUS holds 401, 403, 429, and 503, the codes that lean toward a block. A 200 is deliberately left out of that set, because the soft block this page is about hides behind a 200. A block-leaning status raises suspicion but does not decide the verdict by itself.
Measure the body against a floor. MIN_BODY_BYTES is set to 2000 here. Most challenge pages weigh a few kilobytes while a real article weighs tens, so a body under the floor is a hint that something other than the page came back. Tune the floor to the size of the pages you actually scrape; a 2 KB floor is right for articles and wrong for a sparse API JSON response.
Match against vendor fingerprints. CHALLENGE_FINGERPRINTS is a lookup table keyed by anti-bot vendor, each value a list of marker strings that show up in that vendor's block page. matchFingerprint walks the table and returns the first vendor and marker it finds in the combined body-plus-headers blob, so the verdict can name which service blocked you. Add markers for the sites you scrape as you observe new block pages. Vendors do rewrite their challenge markup, so a body marker like Checking your browser before accessing can stop appearing and quietly miss live blocks; the Server and cf-mitigated headers change far less often, which is why they sit in the table as backstops.
Combine the signals into one verdict. A fingerprint match or a cf-mitigated: challenge header is decisive on its own, since both are specific to block pages. Absent those, the script requires a block-leaning status and a thin body together before it calls the response blocked, which stops a short but legitimate page from being flagged. The returned reasons array records every signal that fired so the caller can log why. Two things this verdict can't tell you on its own. Because it runs a plain HTTP request, a blocked verdict on a cloudflare or datadome vendor may reflect a TLS or JavaScript challenge that a real browser would clear, so retry through Puppeteer or a patched headless Chrome before deciding the site is unreachable. And a one-shot check on a single URL misses a slow ramp where a site serves real pages for the first few hundred requests and then starts returning 429s, so call detectSoftBlock across a sample of the run and back off when the blocked rate climbs.
Use this when
You want a single function that tells you whether a scraped response is the page you asked for or a block dressed up as one, so your pipeline can retry, escalate to a browser, or quarantine the row instead of saving challenge HTML as data.
Skip this when
The site never fronts an anti-bot service and a status check is enough (just read res.status); you need to get past the block rather than name it (render with Puppeteer or a patched browser); you are fighting TLS-level fingerprinting before the body is even returned (use an impersonating HTTP client); or you need to recover automatically after detection (pair this with a backoff-and-retry loop).