Guides

Async jobs & polling

Crawls and batches run in the background. You start one, get a job_id back at once, and then poll it (or get a webhook) until it finishes.

Lifecycle

Async job endpoints
Step	Crawl	Batch
Start (202, returns job_id)	`POST /v1/crawl`	`POST /v1/batch`
Status	`GET /v1/crawl/:id` (includes the pages)	`GET /v1/batch/:id` (counts only)
Results	`pages` in the status	`GET /v1/batch/:id/results` (paginated JSON, or JSONL/CSV)
Cancel	`DELETE /v1/crawl/:id`	`DELETE /v1/batch/:id`
Retry-safe submit	`Idempotency-Key` header	`Idempotency-Key` header
Webhooks	No	Yes

Both have the same status values: queued → running → done, failed or canceled. The last three are final. done means the job ran to its end, not that every page succeeded: check error (set when it stopped early, for example when credits ran out) and the per-page or per-URL results. failed means the job itself could not run (the crawl's start URL is disallowed, our engine kept failing, the account was suspended).

Starting, polling, listing, downloading and canceling are free (0 credits); you pay for pages fetched. They do count toward your requests per minute, so poll at a sensible pace.

Polling

Start at about one second and back off to ten. Treat 429 and 5xx on a status call as "try again later", not as the job failing. Give up after a deadline that fits the job: batches pace each site at about one request every two seconds, so 1,000 URLs on a single site take about half an hour.

JavaScript (fetch)

const BASE = 'https://api.104-236-84-15.sslip.io/v1';
const headers = { Authorization: `Bearer ${process.env.PAGETEER_API_KEY}` };
const sleep = ms => new Promise(r => setTimeout(r, ms));

/** Polls GET /v1/{kind}/{id} until the job is done, failed or canceled. */
async function waitForJob(kind, id, { timeoutMs = 30 * 60_000 } = {}) {
  const deadline = Date.now() + timeoutMs;
  let delay = 1000;
  for (;;) {
    const res = await fetch(`${BASE}/${kind}/${id}`, { headers });
    if (res.status === 429 || res.status >= 500) {
      await sleep(Math.max(Number(res.headers.get('retry-after')) * 1000 || 0, delay));
    } else {
      const job = await res.json();
      if (!res.ok) throw new Error(`${job.error.code}: ${job.error.message}`);
      if (['done', 'failed', 'canceled'].includes(job.status)) return job;
      if (Date.now() > deadline) throw new Error(`${kind} ${id} still ${job.status}`);
      await sleep(delay);
    }
    delay = Math.min(delay * 1.5, 10_000);
  }
}

const crawl = await waitForJob('crawl', jobId);
console.log(crawl.status, crawl.error, crawl.pages.length);

Python (requests)

import os
import time

import requests

BASE = "https://api.104-236-84-15.sslip.io/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}


def wait_for_job(kind: str, job_id: str, timeout_s: float = 1800) -> dict:
    """Polls GET /v1/{kind}/{id} until the job is done, failed or canceled."""
    deadline = time.monotonic() + timeout_s
    delay = 1.0
    while True:
        res = requests.get(f"{BASE}/{kind}/{job_id}", headers=HEADERS, timeout=30)
        if res.status_code == 429 or res.status_code >= 500:
            time.sleep(max(float(res.headers.get("retry-after", 0)), delay))
        else:
            res.raise_for_status()
            job = res.json()
            if job["status"] in ("done", "failed", "canceled"):
                return job
            if time.monotonic() > deadline:
                raise TimeoutError(f"{kind} {job_id} still {job['status']}")
            time.sleep(delay)
        delay = min(delay * 1.5, 10)


batch = wait_for_job("batch", job_id)
print(batch["status"], batch["counts"], batch["outcomes"])

The JavaScript client does this for you: pageteer.crawl.wait(id), pageteer.batch.wait(id) or pageteer.crawl.run(request) (start and wait).

Paginating batch results

GET /v1/batch/:id/results returns { items, next }, items in index order. Pass next as after for the following page, until next is null. Pages hold up to limit items (default 100, max 1,000).

The cursor is the item's index, so it stays valid while the batch runs and across restarts: store the last next and continue from it later.
Filter with status (for example failed) and shrink pages with include_content=false.
Reading while the batch runs: items that are still queued or running come back without results. Either wait until the batch is final, or read status=succeeded pages and re-read from your last cursor later. (Once an item finishes it never changes again.)
For everything at once, format=jsonl or format=csv stream the whole batch in one response, no cursor needed.

JavaScript (fetch)

async function* batchResults(id, query = {}) {
  let after = -1;
  for (;;) {
    const qs = new URLSearchParams({ ...query, after: String(after), limit: '1000' });
    const res = await fetch(`${BASE}/batch/${id}/results?${qs}`, { headers });
    const page = await res.json();
    if (!res.ok) throw new Error(`${page.error.code}: ${page.error.message}`);
    yield* page.items;
    if (page.next === null) return;
    after = page.next;
  }
}

for await (const item of batchResults(jobId, { status: 'succeeded' })) {
  console.log(item.index, item.url, item.result.markdown.length);
}

Python (requests)

def batch_results(job_id: str, **query):
    after = -1
    while True:
        res = requests.get(
            f"{BASE}/batch/{job_id}/results",
            headers=HEADERS,
            params={**query, "after": after, "limit": 1000},
            timeout=60,
        )
        res.raise_for_status()
        page = res.json()
        yield from page["items"]
        if page["next"] is None:
            return
        after = page["next"]


failed = [item["url"] for item in batch_results(job_id, status="failed", include_content="false")]

Idempotent submits

A network error after POST /v1/batch or POST /v1/crawl leaves you unsure whether the job was created. Send an Idempotency-Key header and retrying is safe:

The key is any 1–255 visible ASCII characters, unique per job you mean to create (a UUID, or a name such as listings-2026-10-02).
Same key and same body: you get the original 202 (same job_id, and the generated webhook_secret again) with the header idempotent-replayed: true. No second batch is created and nothing is charged twice.
Same key, different body: 409 idempotency_conflict, with the original job_id in the error. "Same body" means the same request after defaults are applied.
Keys are scoped to your account and remembered as long as the batch exists (30 days).

Replay

HTTP/1.1 202 Accepted
idempotent-replayed: true
x-credits-used: 0

{"job_id":"6ac0f917934df03715b56c42","status":"queued","total":3,"accepted":2,"rejected":1,"webhook_secret":"whsec_51fe…"}

POST /v1/crawl takes the same header with the same rules (it replays { job_id }). Crawl keys are kept as long as the crawl job. The JavaScript client sends a random Idempotency-Key with every batch by default; for crawls, pass one in the request headers.

Canceling

DELETE on a running job stops it and returns the job as it is now. Work already done stays (and stays charged); a page or URL in flight is refunded; open batch URLs become canceled. Canceling a finished job is a no-op, so it is safe to call twice.

Retention

Batch jobs, their results and webhook deliveries are kept for 30 days after the batch was created, then deleted: download what you need. The dashboard lists your crawls and batches too.

Prefer webhooks for long batches: one batch.completed event replaces hundreds of polls. See Webhooks.