Stop Getting Blocked While Scraping: 403s, Bans, Limits

A scraper that gets blocked usually has one of a handful of problems, and each one needs a different response. After this post you can tell them apart from the response alone, retry with a budget instead of a loop, rotate your identity without breaking your own session, and notice a rising block rate before it becomes an outage.

Technical measures do not decide whether you may collect the data. For that, read is web scraping legal.

Read the response first

Classify every response before you decide what to do:

The last row is easy to miss. A scraper that checks only the status code treats an empty result list or placeholder text as success. Keep a known-good copy of a page and compare against it.

The Cloudflare, DataDome and Akamai rows come from the research notes behind the solver guides. The notes name the Akamai causes as TLS mismatch, bad IP reputation and automation flags, so treat that block as a diagnosis task first.

Kinds of block

• One banned IP. Every request from that address is refused, and the same request from another address passes. DataDome's t=bv is this case: the IP is banned at network level and solving cannot clear it. For the cookie side of the same vendor, see DataDome solver API.
• A refused range. Fresh addresses from the same provider fail too. The proxies guide says anti-bot vendors commonly refuse datacenter ranges, so the fix is a different proxy class, not another address.
• A client the site distrusts. The IP is fine, because a real browser passes from it, but your client fails. Check the user agent, Client Hints, header order and TLS. See headers, user agent and TLS.
• A blocked account. Logged-in requests fail while anonymous ones pass. A new IP will not help. Repeat the request without the login cookie to confirm, then stop and look at what the account did.
• A challenge loop. You solve, replay, and get challenged again. The notes list the causes: the IP or user agent differs from the solve, the proxy rotated, or the client's TLS fingerprint differs from a browser. Cloudflare can also revoke a cf_clearance early, so a challenge after a working stretch is not always your mistake. Akamai's _abck shows the same pattern (Akamai solver API).

Back off and set a retry budget

Retrying is where scrapers burn money and goodwill. Set four limits before you write the loop:

• Attempts per URL. A hard cap, so one stubborn URL cannot run forever.
• Solves per URL and identity. A solve is a charged task, and a task that finishes as failed automatically returns its original charge once. Costs covers the arithmetic.
• Waits with jitter. Doubling delays, randomised between zero and the ceiling, so parallel workers do not retry in step.
• A circuit breaker. When the block rate crosses a threshold, pause the job instead of pushing harder.

SurSolver has its own limit, separate from the target's. Submissions are capped at 60 per minute per account by default. A refused submission answers ERROR_NO_SLOT_AVAILABLE with a (retry in Ns) hint, creates no task and charges nothing. See Rate limits.

The starting values below are placeholders, not benchmarks. Tune them to your own traffic. The code classifies a response, decides the next step and runs the loop. You pass in your own get, solve and rotate functions, so it works with any HTTP client and any solving helper:

import random
import time
from collections import deque

CHALLENGE_MARKERS = {
    'cloudflare-challenge': 'Just a moment',
    'datadome': 'captcha-delivery.com',
    'akamai': 'Pardon Our Interruption',
}
STATUS_ERROR_FLOOR = 400
STATUS_RATE_LIMITED = 429
STATUS_FORBIDDEN = 403
MAX_ATTEMPTS = 4
MAX_SOLVES_PER_IDENTITY = 1
BACKOFF_BASE_SECONDS = 2.0
BACKOFF_CAP_SECONDS = 60.0
RETRY_AFTER_JITTER_SECONDS = 1.0
MONITOR_WINDOW = 50
MONITOR_THRESHOLD = 0.2


class BlockRateExceeded(RuntimeError):
    pass


def classify(status: int, body: str) -> str:
    if status >= STATUS_ERROR_FLOOR:
        for name, marker in CHALLENGE_MARKERS.items():
            if marker in body:
                return f'challenge:{name}'
    if status == STATUS_RATE_LIMITED:
        return 'rate-limited'
    if status == STATUS_FORBIDDEN:
        return 'denied'
    return 'ok'


def backoff_delay(attempt: int, retry_after: float | None = None) -> float:
    if retry_after is not None:
        return retry_after + random.uniform(0, RETRY_AFTER_JITTER_SECONDS)
    ceiling = min(BACKOFF_CAP_SECONDS, BACKOFF_BASE_SECONDS * 2**attempt)
    return random.uniform(0, ceiling)


def retry_after_seconds(headers) -> float | None:
    value = headers.get('Retry-After', '')
    return float(value) if value.isdigit() else None


def next_step(kind: str, solves_used: int) -> str:
    if kind == 'ok':
        return 'done'
    if kind == 'rate-limited':
        return 'wait'
    if kind.startswith('challenge:') and solves_used < MAX_SOLVES_PER_IDENTITY:
        return 'solve'
    return 'rotate'


class BlockMonitor:
    def __init__(self) -> None:
        self.recent: deque[bool] = deque(maxlen=MONITOR_WINDOW)

    def record(self, kind: str) -> None:
        self.recent.append(kind != 'ok')

    def tripped(self) -> bool:
        if len(self.recent) < MONITOR_WINDOW:
            return False
        return sum(self.recent) / MONITOR_WINDOW > MONITOR_THRESHOLD


def fetch(url, get, solve, rotate, monitor: BlockMonitor):
    solves_used = 0
    for attempt in range(MAX_ATTEMPTS):
        if monitor.tripped():
            raise BlockRateExceeded('block rate above threshold: pause the job')
        response = get(url)
        kind = classify(response.status_code, response.text)
        monitor.record(kind)
        step = next_step(kind, solves_used)
        if step == 'done':
            return response
        if step == 'wait':
            time.sleep(backoff_delay(attempt, retry_after_seconds(response.headers)))
        elif step == 'solve':
            solve(url, kind)
            solves_used += 1
        else:
            rotate()
            solves_used = 0
            time.sleep(backoff_delay(attempt))
    raise RuntimeError(f'{url}: still blocked after {MAX_ATTEMPTS} attempts')

Your solve is a createTask and getTaskResult helper like the one in the Python integration guide. It must also apply the result to the session get uses, with the same proxy and user agent.

Rotate or persist

Treat identity as one bundle: exit IP, user agent, language, TLS client and cookie jar. Keep the bundle while it works. Solved cookies are bound to the IP and user agent that earned them, so changing the IP alone invalidates the jar.

Rotate the bundle as a unit when it stops working: new IP, new cookie jar, same rules for user agent and language, then a fresh session. Do not carry a cookie across to another IP. The research notes make the same point for DataDome: rotate the cookie together with the proxy.

Do not rotate on every request for the cookie-based types. A rotating gateway breaks the "same IP" rule these types rely on. Hold one sticky session per identity, as described in proxies for captcha solving.

Monitor the block rate

You cannot tune what you do not measure. For every request, log the target host, the proxy or session id, the status, the classification above and the number of solves used. Compute the share of non-ok responses over a sliding window, per target and per proxy pool.

A rising rate has an order of causes worth checking:

1. Your request pattern: too fast, too regular, too many parallel workers on one IP.
2. Identity consistency: any piece of the bundle changing mid-session.
3. The proxy class, if one pool is worse than the rest.

Watch the money alongside. POST /getBalance reports the cash balance, and a block-heavy run shows up there before it shows up in your logs.

Next steps

Start by logging the classification of every response for a day, without changing anything else. Then add the retry budget and circuit breaker, and open Errors to see how each failure of a task reaches you. When a challenge page is the cause, pick the matching solver page and send a createTask.