Rate limits

Per plan

Limits apply per account: all keys of a team share them. Placeholder values until launch.

• Requests per minute counts every API call, including polling, usage and cache hits. Each response carries x-ratelimit-limit and x-ratelimit-remaining; over the limit you get 429 rate_limited with Retry-After.
• Concurrent requests counts requests that fetch a page or query search engines at the same moment. One too many gets 429 concurrency_limit (Retry-After: 1) at once rather than waiting in a queue. Crawl pages and batch URLs use the same slots (batches at most half of them), so size your own parallelism to leave room.
• Pages per crawl and URLs per batch cap a single job; more is a 400 invalid_request. Split large work into several jobs.

Per target host

Independently of your plan, we throttle how fast any one target host is fetched, across all customers. A crawl or a burst against a single site may therefore be slower than your plan's limit. This is deliberate. Batches are paced further: about one request every two seconds per site (see batch pacing).

When a site asks us to slow down

If a target answers 429 Too Many Requests, or 503 with a Retry-After header, we do not retry, switch tiers or route around it. You get 429 target_rate_limited with retry_after (seconds; free), and new requests to that site from any customer get the same answer until the wait is over. A 429 without Retry-After pauses the site for 30 seconds, then longer if it repeats. This is what keeps sites from escalating to CAPTCHAs and blocks.

Transient failures (network errors, 502, 503 without Retry-After, 504) are retried at most twice with exponential backoff and jitter, inside your timeout_ms. If the site keeps failing you get its status back (not charged when it is a 5xx).

To ask less of a site in the first place, reuse results with max_age (see caching) and use respect_robots, which also honors the site's Crawl-delay.

Handling 429

Over a limit you get 429 with rate_limited, concurrency_limit or target_rate_limited and a Retry-After header (seconds). 429s are never charged. Wait at least that long, and back off exponentially with jitter if they continue. The same loop is right for 5xx; other 4xx should not be retried unchanged (see Errors). The JavaScript client does this for you.

// call: () => fetch(...). Retries 429 and 5xx; returns the last response.
async function withRetry(call, attempts = 5) {
  for (let i = 0; ; i++) {
    const res = await call();
    if (res.status !== 429 && res.status < 500) return res; // success, or a 4xx to fix
    if (i === attempts - 1) return res;
    const retryAfter = Number(res.headers.get('retry-after')) || 0;
    const backoff = Math.min(30, 2 ** i) * (0.5 + Math.random());
    await new Promise(r => setTimeout(r, Math.max(retryAfter, backoff) * 1000));
  }
}

Abuse signals (for example a sudden spike against one host, or very high block rates) can flag an account for review.