How to Scrape a Website That Blocks Bots (Without Building a Proxy Farm)

What do you do when a website blocks your scraper?

You stop trying to out-code the block and switch to rotating residential proxies plus a real rendered browser, because the block is almost never about your parsing logic, it is about your request looking like a bot before your code even runs. Cloudflare, DataDome, PerimeterX and Akamai fingerprint the TLS handshake, the HTTP/2 frame order, the browser's JavaScript environment and your IP's reputation, then decide to serve a CAPTCHA or a blank challenge page before your scraper ever sees the HTML it wants. Fixing that yourself means building a headless browser farm, buying and rotating residential IPs, and re-tuning fingerprints every time the vendor updates its detection, which is a full engineering team's job, not a weekend script.

The practical shortcut is to rent that infrastructure per request instead of building it. A hosted scraping API that already does rotating anti-bot proxies and JavaScript rendering, like Scrape Any Site, takes a URL and returns clean HTML, Markdown or structured JSON, so the anti-bot problem becomes someone else's maintenance burden instead of yours.

How can I tell if a site is blocking me because of anti-bot protection?

Check the response for a challenge page, a 403 with no useful body, or a status that only appears when you remove your browser's usual headers, because those three symptoms almost always mean a bot-detection vendor intercepted the request before your target site's own code ran. A plain requests.get() that returns a page titled "Just a moment..." or "Access Denied" with a Cloudflare or Akamai reference ID in the footer is the giveaway. A quieter version of the same thing is a 200 response with an empty body or a JavaScript-only shell, which means the site rendered its real content client-side and your scraper never executed that JavaScript.

A second, sneakier failure mode is soft-blocking: the request succeeds and returns a page, but the data is wrong or throttled, rate-limited to a crawl, or subtly different from what a real browser sees. This happens when the anti-bot system trusts you enough to respond but not enough to give you the real content, and it is the hardest failure to notice because nothing errors out.

Common anti-bot signals and what triggers them

• TLS/JA3 fingerprint mismatch: your HTTP client's TLS handshake doesn't match a real browser's, even with the right User-Agent header.
• Missing JavaScript execution: the page needs a browser to run scripts that set cookies or compute a token before the real content loads.
• IP reputation: datacenter IPs (AWS, most VPS providers) are flagged far more aggressively than residential or mobile IPs.
• Request pattern: too many requests too fast, from one IP, with no mouse movement or scroll events, reads as a bot even if every header is perfect.
• Missing or stale cookies: some vendors require a challenge cookie set on the first visit to be replayed on every later request.

What actually works to get past Cloudflare, DataDome and similar protection?

Combine a real headless browser that executes JavaScript with rotating residential or mobile proxies, and treat both as required, not optional, because either one alone still gets caught. A headless browser without proxy rotation solves the JavaScript-execution problem but still gets IP-banned after a few hundred requests. Proxy rotation without a real browser solves the IP problem but still fails the fingerprint check on sites that inspect the TLS handshake or run a JavaScript environment probe.

This is exactly the stack a general-purpose scraping actor gives you without building it: Scrape Any Site renders the page in a real browser, rotates through Apify's anti-bot proxy pool automatically, and returns the result as clean HTML, Markdown, or a structured JSON object if you describe the fields you want in plain English. You paste a URL, choose CSS-selector extraction if you know the exact fields or AI extraction if the page layout varies, and get a dataset back, no proxy account, no browser farm, no fingerprint tuning.

A minimal Python example

import requests

run_input = {
    "startUrls": [{"url": "https://example-protected-site.com/product/123"}],
    "extractionMode": "ai",
    "aiInstructions": "Extract product name, price, availability and SKU as JSON",
}

resp = requests.post(
    "https://api.apify.com/v2/acts/YOUR-ACTOR-ID/run-sync-get-dataset-items",
    params={"token": "YOUR_APIFY_TOKEN"},
    json=run_input,
    timeout=120,
)
items = resp.json()
print(items[0]["price"], items[0]["availability"])

Swap extractionMode to CSS-selector based extraction once you know the page structure, since it costs less and returns identical fields on every run, where AI extraction is worth the extra cost mainly for pages whose layout you have not mapped yet or that change often.

How much does it cost to scrape a protected site this way?

Expect a few cents per page rather than a few thousandths of a cent, because you are paying for browser rendering time and proxy bandwidth, not just a raw HTTP request. A typical run against a JavaScript-heavy, anti-bot-protected page costs somewhere in the range of $0.01 to $0.05 per page depending on render time and page weight, so scraping 1,000 protected product pages runs roughly $10 to $50, and Apify's $5 free credit for new accounts covers a few hundred pages to test the approach before you commit budget. That is meaningfully more than an unprotected static page, which is the trade-off: you are renting the anti-bot infrastructure per request instead of building and maintaining it yourself.

What goes wrong in practice, and how do you fix it?

The most common failure once you switch to a rendering-plus-proxy actor is a run that returns the challenge page anyway on a small percentage of requests, because no anti-bot bypass is 100 percent on every single attempt, especially against a vendor that just shipped a new fingerprint check. The fix is not to give up on the actor, it is to add a retry with a fresh proxy session: most hosted scraping APIs let you set a session-rotation option that forces a brand-new IP and browser fingerprint on retry instead of reusing the one that got flagged. Retrying the exact same session just gets flagged again, since the block was tied to that specific fingerprint and IP pair, not to your account.

A second practical issue is cost creep on large catalogues: if you run AI extraction on every page of a 50,000-product catalogue, you are paying the AI-extraction premium 50,000 times for a page layout that is identical on every product. Run AI extraction once to map the CSS selectors for a given site, then switch the same actor to CSS-selector mode for the bulk run. You get the same fields at a fraction of the cost, and the run finishes faster because selector-based extraction skips the AI call entirely.

How do you keep watching a protected site after the first scrape?

Set up a scheduled monitor instead of re-running your scraper manually, because the whole point of getting past the block once is to not have to fight it again every time you want fresh data. Website Change Monitor checks a page on a schedule and returns a clean diff of exactly what was added or removed since the last check, with a severity score, so you get alerted only when something meaningful actually changes, a price move, a new product, a policy edit, rather than having to re-read the whole page yourself. It sends alerts to Slack, Discord, Telegram or any webhook, which is the difference between a one-off scrape and an ongoing competitive-intelligence feed.

This pairing is the pattern worth remembering: use a rendering-and-proxy scraper for the first extraction and for any ad-hoc structured pull, then hand recurring watch duty to a change monitor so you are not paying render costs to re-check a page that has not moved.

Is this legal, and are there sites you still should not scrape this way?

Scraping publicly accessible pages is generally legal in most jurisdictions, but a site's anti-bot protection existing at all is a signal that the owner does not want automated access, so check the terms of service and avoid anything behind a login wall you do not have permission to access, and never scrape personal data you plan to resell without a legitimate basis under GDPR or similar regional rules. Rate-limit your own requests even when the tooling could go faster, both because it is the considerate thing to do and because hammering a site increases the odds the next fingerprint update targets you specifically.

Frequently asked questions