How to Choose a CAPTCHA Solving Service: A Benchmark Method

You will leave this post with a method for choosing a captcha-solving service on evidence you collect yourself: a short list of criteria, a benchmark you can run in an afternoon, a script that records it and a worksheet to compare candidates. Before any of that, is web scraping legal frames the four questions your use of scraped data has to answer.

There are no rankings and no competitor numbers here. Any published success rate or speed figure depends on which captcha types, which sites, which hours and what definition of "success" the author used. Yours will differ. And since SurSolver is one of the services you may be comparing, treat anything below about SurSolver as a statement of its documented behaviour, not a verdict.

Start from your workload

Before you open a vendor page, write down four things:

• Task types. Which captchas and anti-bot systems your targets use. The identification guide helps if you do not know yet.
• Volume and shape. Tasks per day, and how bursty. Ten thousand tasks spread over a day and ten thousand in ten minutes are different problems.
• Constraints. Whether your targets need a proxy, whether you must replay results from a fixed IP, and what latency your flow can tolerate.
• Integration. Your language and whether you can poll from a worker.

A service that is excellent for someone else's workload can be poor for yours. This list becomes the input to the benchmark.

Criteria

Coverage. Does it list the exact types you need, under task names you can send? Check the catalogue yourself rather than a marketing page. SurSolver publishes its catalogue at GET /v1/task-types, with every type's accepted names, parameters and result keys, and its available services at GET /v1/services, no key needed. There is nothing to take on trust.

Success on your targets. Measure whether the target accepted the result, not whether the service returned ready. A token that comes back and is rejected is a failure.

Time to result. Measure the whole path, from createTask to the first reply that carries the solution. Vendors poll at different intervals, so use the same interval for every candidate (SurSolver documents 3 seconds), and remember that your timing resolution is no better than that interval.

What a failure costs. This is the criterion most often skipped. Find out in writing whether a task is charged when it is accepted or when it succeeds, whether failed tasks are refunded and how. SurSolver charges on acceptance and automatically returns the original charge for failed tasks once; see Billing & packages. That is a documented rule, not a flaw or a feature. It changes your cost per useful result, which is why the benchmark below records it. CAPTCHA solving cost optimization shows how to compute that figure.

Price structure. Per-task price per type, minimum top-up, whether subscription packages exist, what volume pricing looks like. SurSolver's per-service price is price_per_request on GET /v1/services; volume rates are by agreement with support, and GET /v1/billing-terms returns the smallest top-up in USD.

API quality. Read the error catalogue before you integrate. Are errors in a documented envelope? Do you know what to do on each code? SurSolver answers API errors as HTTP 200 with errorId: 1 and uses ten documented errorCode values; an unexpected fault is HTTP 500 without errorId. See Errors.

Limits. Per-minute submission limits, concurrency caps and queue behaviour at your burst size. SurSolver's defaults are in Rate limits; test whether they fit your peak.

Support. Send a real question with a real taskId during the benchmark and note how long the answer takes and whether it is specific.

Fit. SurSolver is an HTTP API with createTask, getTaskResult and getBalance. There is no browser extension, no callback delivery (you poll) and no image or text recognition tasks. If you need those, say so on your worksheet.

Run your own benchmark

1. Use real inputs. Take task parameters from the sites you will actually hit. A vendor's demo page tells you little.
2. Same inputs for every candidate. Same URLs, same keys, same proxy where it applies, same polling interval.
3. Your concurrency. Run at the burst size you expect, not one task at a time.
4. Several runs at different hours. One run is an anecdote.
5. Enough samples. With 30 tasks, a measured 90% success rate has a 95% margin of roughly plus or minus 11 points, so 90% and 95% are indistinguishable (source: normal approximation of a binomial proportion, 1.96 × √(0.9 × 0.1 / 30) ≈ 0.107). Plan for hundreds per type.
6. Include failures in the denominator. Timeouts, refused submissions and wrong results all count against cost per useful result.
7. Record the target's verdict. Add a column that you fill from the target side.
8. Reconcile spend. Compare your task count with the account's ledger instead of trusting a floating-point balance. getBalance returns a JSON number and is fine for a quick check, not for accounting.

This Python harness uses only the standard library. It follows the SurSolver envelope, which uses the common createTask / getTaskResult format. For another provider, change BASE_URL, the key and the task names, and read that provider's error documentation first.

import csv
import json
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE_URL = 'https://api.sursolver.com'
POLL_SECONDS = 3
TIMEOUT_SECONDS = 180
HTTP_TIMEOUT_SECONDS = 30
CONCURRENCY = 5
FIELDS = ['index', 'task_type', 'task_id', 'outcome', 'error_code', 'seconds']


def post(path, payload):
    request = urllib.request.Request(
        f'{BASE_URL}{path}',
        data=json.dumps(payload).encode('utf-8'),
        headers={'content-type': 'application/json'},
    )
    with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT_SECONDS) as response:
        return json.load(response)


def run_one(args):
    index, client_key, task = args
    started = time.monotonic()
    row = {'index': index, 'task_type': task['type'], 'task_id': '', 'outcome': '',
           'error_code': '', 'seconds': ''}
    try:
        created = post('/createTask', {'clientKey': client_key, 'task': task})
        if created.get('errorId') != 0:
            row.update(outcome='refused', error_code=created.get('errorCode', ''))
            return row
        row['task_id'] = created['taskId']
        while time.monotonic() - started < TIMEOUT_SECONDS:
            time.sleep(POLL_SECONDS)
            result = post('/getTaskResult', {'clientKey': client_key, 'taskId': row['task_id']})
            if result.get('errorId') != 0:
                row.update(outcome='error', error_code=result.get('errorCode', ''))
                break
            if result.get('status') == 'ready':
                row['outcome'] = 'ready'
                break
        else:
            row['outcome'] = 'timeout'
    except (urllib.error.URLError, json.JSONDecodeError, TimeoutError) as err:
        row.update(outcome='transport', error_code=type(err).__name__)
    row['seconds'] = round(time.monotonic() - started, 1)
    return row


def main(client_key, tasks_path, out_path):
    with open(tasks_path, encoding='utf-8') as handle:
        tasks = json.load(handle)
    jobs = [(i, client_key, task) for i, task in enumerate(tasks)]
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        rows = list(pool.map(run_one, jobs))
    with open(out_path, 'w', newline='', encoding='utf-8') as handle:
        writer = csv.DictWriter(handle, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)


if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2], sys.argv[3])

tasks.json is a list of the task objects you would send, one per sample. Open the CSV, add an accepted_by_target column from your side, and compute cost per useful result as described in the cost post. The harness spends real money on every accepted task, so size the sample to a budget you set first.

Comparison worksheet

Copy this into a spreadsheet. Weight the rows by what matters to you, then fill one column per candidate with measured values and the date of measurement.

Red flags

• A success rate or speed figure with no definition, sample size, date or method.
• A price shown without saying what happens to a failed task.
• No documented list of errors, or errors that differ between the docs and the API.
• A comparison page, including this one, that ranks its own product first. Ask for the method, not the ranking.
• No way to see what each task cost you.
• Claims that every captcha type is supported, with no list of task names to send.

Next steps

Pick one task type you really need and run the harness against a small budget. The quickstart covers createTask, getTaskResult and getBalance. If you already have an integration, Migration lists what stays the same and what to adjust. Questions about volume go to support.