Building a Polite Web Scraper: Rate Limits, Retries, and Backoff

Choose concurrency deliberately

Concurrency is the number of requests in flight at once. More workers do not automatically mean more useful throughput; once the target starts throttling, extra workers only create more retries.

Start low, measure the success rate, and increase until the success rate begins to fall. That point is your practical ceiling for that target, and it can differ by endpoint and time of day.

Retry only what is worth retrying

Timeouts, connection resets, and 502/504 responses are reasonable retry candidates. A 400 or 403 is not. Retrying a policy rejection wastes time and looks abusive.

Cap retries per URL, and make the cap visible in logs. If most requests need three retries, the concurrency or proxy configuration is wrong.

Backoff with jitter

Exponential backoff doubles the delay after each failure. Jitter adds randomness so that many workers do not retry in lockstep. Together they reduce the thundering-herd effect that turns a brief throttle into a sustained outage.

import random
import time
import requests

def fetch(url, session, attempts=5):
    for attempt in range(attempts):
        response = session.get(url, timeout=30)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
        elif response.status_code in (502, 504):
            delay = 2 ** attempt
        else:
            return response
        time.sleep(delay + random.uniform(0, 0.75))
    return response

Respect the rules and the target

Check the site terms, robots directives, and any published API limits before collecting. Prefer official APIs where they exist. Cache responses you already have instead of re-fetching them, and schedule work outside peak hours when possible.

These practices reduce load and also reduce your own cost. A cache hit is cheaper than a proxy request and cannot be rate-limited.

• Set a per-target request budget and monitor it.
• Honour Retry-After when the server provides it.
• Cache successful responses with a sensible TTL.
• Log status codes, latency, and retry counts per target.

Distributed rate limiting

Once more than one worker talks to the same target, a per-process limit is not enough. Two workers each respecting ten requests per second still produce twenty. Use a shared counter, a queue with a fixed drain rate, or a token bucket stored in a common service.

Assign each target its own budget. A broad crawl might touch several domains with very different tolerances; a single global limit is either too strict for the tolerant ones or too aggressive for the sensitive ones.

Observability for collection jobs

Track success rate, latency percentiles, retry counts, and status-code distribution per target. These four signals explain most failures. A rise in retries with stable latency suggests throttling; a rise in latency with stable retries suggests network or proxy issues.

Alert on trends rather than single errors. A handful of 429 responses is normal; a sustained increase over ten minutes is not. Dashboards should show the same window you use when investigating, so the chart and the logs agree.

Reading robots and terms before writing a crawler

robots.txt and a site's terms are not the same document, and neither replaces judgment about load. Read both before choosing paths and pacing. If a path is disallowed for automated clients, do not route around it with a proxy; the proxy changes the network path, not the permission.

Some sites publish a crawl-delay directive or a dedicated API. Prefer the API when one exists: it is usually more stable, cheaper to operate, and easier for the publisher to support. Use scraping for what the API does not cover.

Write down the basis for the collection in the project notes: which pages, which frequency, and why. That note is what lets a reviewer approve the job and what protects the team if a target changes its policy later.

Incremental collection is the strongest politeness measure available. Store the ETag or Last-Modified value for each page and send a conditional request on the next pass. When nothing has changed, the target answers with a small response and no page body, which costs the publisher almost nothing and saves your bandwidth as well.