Solve CAPTCHAs in a Python Scraper with an API

By the end of this guide you will have a Python module that sends a CAPTCHA to the SurSolver API and returns the solution, and a requests scraper that uses it to submit a reCAPTCHA v2 form. It retries only where a retry makes sense, because every accepted task is charged.

Where a CAPTCHA stops a scraper

A scraper meets a CAPTCHA in two places: a form whose submit needs a token (search, login, sign-up), or a page that serves a challenge instead of content. This guide covers the first case, for the most common widget, reCAPTCHA v2 on a plain HTML form.

On such a page the widget is a div.g-recaptcha with a data-sitekey attribute, and the token goes into a field named g-recaptcha-response in the form data. If the page builds its widget from JavaScript there is no attribute to read; how to identify any CAPTCHA and extract its parameters shows the Network tab route to the same key.

A reusable solver module

Save this as sursolver.py. It follows the API envelope exactly: API errors are HTTP 200 with errorId: 1, so the code branches on errorId and treats a non-200 status or a body without errorId as a transport failure. It polls getTaskResult every 3 seconds, at most 60 times, and stops at the first errorId: 1.

# sursolver.py
import os
import random
import re
import time

import requests

API = "https://api.sursolver.com"
CLIENT_KEY = os.environ["SURSOLVER_API_KEY"]
POLL_INTERVAL_SECONDS = 3
POLL_ATTEMPTS = 60
REQUEST_TIMEOUT_SECONDS = 30
DEFAULT_RETRY_SECONDS = 5
MAX_JITTER_SECONDS = 2
RETRY_HINT = re.compile(r"\(retry in (\d+)s\)")


class SolverError(Exception):
    """An error the API reported itself: HTTP 200 with errorId 1."""

    def __init__(self, code, description):
        super().__init__(f"{code}: {description}")
        self.code = code
        self.description = description


def call(path, body):
    response = requests.post(f"{API}{path}", json=body, timeout=REQUEST_TIMEOUT_SECONDS)
    response.raise_for_status()
    data = response.json()
    if "errorId" not in data:
        raise RuntimeError(f"{path} answered without errorId")
    if data["errorId"] != 0:
        raise SolverError(data["errorCode"], data["errorDescription"])
    return data


def create_task(task):
    body = {"clientKey": CLIENT_KEY, "task": task}
    try:
        return call("/createTask", body)
    except SolverError as error:
        if error.code != "ERROR_NO_SLOT_AVAILABLE":
            raise
        # Refused before a task existed, so nothing was charged. Wait the hinted time.
        hint = RETRY_HINT.search(error.description)
        wait = int(hint.group(1)) if hint else DEFAULT_RETRY_SECONDS
        time.sleep(wait + random.uniform(0, MAX_JITTER_SECONDS))
        return call("/createTask", body)


def solve(task):
    created = create_task(task)
    for _ in range(POLL_ATTEMPTS):
        result = call(
            "/getTaskResult",
            {"clientKey": CLIENT_KEY, "taskId": created["taskId"]},
        )
        if result["status"] == "ready":
            return result["solution"]
        time.sleep(POLL_INTERVAL_SECONDS)
    raise TimeoutError(f'task {created["taskId"]} was not ready in {POLL_ATTEMPTS} polls')

Keep the key in an environment variable. It is a secret, and the authentication guide says to keep it server-side and out of repositories.

Submit the form

The scraper loads the form, reads the site key, solves, and posts the token with the other fields. Solving happens after the page is loaded and right before the post, because a token is single-use and short-lived: about two minutes, and a shorter window is safer.

# spider.py
import re

import requests

from sursolver import CLIENT_KEY, SolverError, call, solve

FORM_URL = "https://example.com/search"
QUERY = "example"
SITE_KEY_PATTERN = re.compile(r'data-sitekey="([^"]+)"')
SUCCESS_MARKER = "Results for"  # text that only the page after a passed check contains
MAX_SOLVE_ATTEMPTS = 2
REQUEST_TIMEOUT_SECONDS = 30


def search(session, query):
    for attempt in range(1, MAX_SOLVE_ATTEMPTS + 1):
        form = session.get(FORM_URL, timeout=REQUEST_TIMEOUT_SECONDS)
        form.raise_for_status()
        fields = {"q": query}

        match = SITE_KEY_PATTERN.search(form.text)
        if match is not None:
            try:
                solution = solve({
                    "type": "ReCaptchaV2TaskProxyLess",
                    "websiteURL": FORM_URL,
                    "websiteKey": match.group(1),
                })
            except SolverError as error:
                if error.code != "ERROR_FAILED":
                    raise
                print(f"attempt {attempt}: {error}")
                continue
            # userAgent is present only when the solver reports one.
            if solution.get("userAgent"):
                session.headers["User-Agent"] = solution["userAgent"]
            fields["g-recaptcha-response"] = solution["gRecaptchaResponse"]

        response = session.post(FORM_URL, data=fields, timeout=REQUEST_TIMEOUT_SECONDS)
        if match is None or SUCCESS_MARKER in response.text:
            return response
        print(f"attempt {attempt}: the site rejected the token")
    raise RuntimeError(f"no accepted result after {MAX_SOLVE_ATTEMPTS} attempts")


if __name__ == "__main__":
    print("balance:", call("/getBalance", {"clientKey": CLIENT_KEY})["balance"])
    with requests.Session() as session:
        print(search(session, QUERY).text[:200])

SUCCESS_MARKER is the one part you must write for your own target: the check that tells an accepted submission from a rejected one is specific to the site. The getBalance call at the start prints the cash balance of the key's owner, so a long run does not begin on an empty account.

Retries without paying for the same mistake twice

A task is charged when createTask accepts it, and a task that finishes as failed, including server-side expiry, automatically returns its original charge once. The solver surface has no idempotency header, so a retried submission is a new, separately charged task. The code above follows from that:

• It retries a refused submission (ERROR_NO_SLOT_AVAILABLE) once after the hinted wait, because a refusal creates and charges nothing.
• It starts a new task after ERROR_FAILED, which is final for that task, and after a token the site rejected. The number of attempts is capped.
• It lets TimeoutError through instead of re-submitting. A task that was accepted and not ready after 60 polls is already charged; check the account before you decide to pay for another.

The API has no endpoint for reporting a bad token, so keep your own count of rejected submissions per site. If a charge looks wrong, send support the taskId and the exact response, as the billing guide describes.

Keep the solve and the scraper on one proxy

For reCAPTCHA v2 the proxyless name works on most pages, because Google's verification does not report which IP solved the challenge. Some pages tie the solve and the submit to one address: Google-owned surfaces, or sessions pinned to an IP. For those, send ReCaptchaV2Task with proxy details and route your own session through the same proxy.

PROXY_HOST = "proxy.example.com"
PROXY_PORT = 8080
PROXY_LOGIN = "YOUR_PROXY_LOGIN"
PROXY_PASSWORD = "YOUR_PROXY_PASSWORD"

proxy_url = f"http://{PROXY_LOGIN}:{PROXY_PASSWORD}@{PROXY_HOST}:{PROXY_PORT}"
session.proxies = {"http": proxy_url, "https": proxy_url}

task = {
    "type": "ReCaptchaV2Task",
    "websiteURL": FORM_URL,
    "websiteKey": site_key,
    "proxyType": "http",
    "proxyAddress": PROXY_HOST,
    "proxyPort": PROXY_PORT,
    "proxyLogin": PROXY_LOGIN,
    "proxyPassword": PROXY_PASSWORD,
}

Send the split fields rather than a single proxy string. A four-part string is always read as user:password:host:port; pasting a string in another order, such as host:port:user:pass, is not rejected and would be charged while using the wrong proxy. Every format is listed in the proxies guide, and proxies for CAPTCHA solving covers sticky sessions and proxy types.

Troubleshooting

• ERROR_INVALID_REQUEST: errorDescription names the field. The usual cause is an empty websiteKey because the regex matched nothing.
• ERROR_ZERO_BALANCE: no task was created. Top up and run again.
• ERROR_TYPE_NOT_SUPPORTED: the task.type is misspelled or not available; compare it with the names in the public catalog (GET /v1/task-types).
• HTTP 500, or a body without errorId: a server fault, not an answer about your task. If you cannot tell whether the task was accepted, check getBalance before you submit again.
• A token the site rejects: check for the Enterprise variant (enterprise.js instead of api.js means ReCaptchaV2EnterpriseTaskProxyLess), an invisible widget solved without isInvisible, a token used twice or older than about two minutes, and a user agent that differs from the one the solution reports (headers, user agent and TLS).

Next steps

Create an API key in the dashboard, put it in SURSOLVER_API_KEY and run spider.py against a form you are allowed to automate. The task lifecycle and errors guides cover every status and code, and the reCAPTCHA v2 guide goes deeper on the widget itself. For driving a real browser instead of requests, continue with the Playwright recipe. If you have not checked the legal side of your target yet, is web scraping legal lists the four questions to answer first.