Batch

Body parameters

Request bodies can be up to 8 MB. URLs that are malformed or point at private or internal addresses fail straight away (free, counted in rejected) without failing the batch. An invalid CSS selector, an unreachable webhook URL or a tier your plan does not include fail the whole request with a 4xx.

Start a batch

curl -X POST https://api.104-236-84-15.sslip.io/v1/batch \
  -H "Authorization: Bearer $PAGETEER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"items":[{"url":"https://example.com","id":"ex-1","metadata":{"source":"docs"}},{"url":"https://www.iana.org/domains/reserved","id":"iana-1"},{"url":"http://10.0.0.1/","id":"bad"}],"mode":"http","formats":["markdown"],"extract":{"selectors":{"heading":"h1"}},"metadata":{"run":"docs-audit"},"webhook":{"url":"https://hooks.example.com/pageful"}}'

const res = await fetch('https://api.104-236-84-15.sslip.io/v1/batch', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.PAGETEER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "items": [
      {
        "url": "https://example.com",
        "id": "ex-1",
        "metadata": {
          "source": "docs"
        }
      },
      {
        "url": "https://www.iana.org/domains/reserved",
        "id": "iana-1"
      },
      {
        "url": "http://10.0.0.1/",
        "id": "bad"
      }
    ],
    "mode": "http",
    "formats": [
      "markdown"
    ],
    "extract": {
      "selectors": {
        "heading": "h1"
      }
    },
    "metadata": {
      "run": "docs-audit"
    },
    "webhook": {
      "url": "https://hooks.example.com/pageful"
    }
  }),
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.post(
    "https://api.104-236-84-15.sslip.io/v1/batch",
    headers=headers,
    json={
        "items": [{
            "url": "https://example.com",
            "id": "ex-1",
            "metadata": {
                "source": "docs",
            },
        }, {
            "url": "https://www.iana.org/domains/reserved",
            "id": "iana-1",
        }, {
            "url": "http://10.0.0.1/",
            "id": "bad",
        }],
        "mode": "http",
        "formats": ["markdown"],
        "extract": {
            "selectors": {
                "heading": "h1",
            },
        },
        "metadata": {
            "run": "docs-audit",
        },
        "webhook": {
            "url": "https://hooks.example.com/pageful",
        },
    },
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

{
  "job_id": "6ac0f917934df03715b56c42",
  "status": "queued",
  "total": 3,
  "accepted": 2,
  "rejected": 1,
  "webhook_secret": "whsec_51fece5d09ba37aae6b7acfec1c7f19aac0f574c6d629ce6"
}

The private address was rejected up front (rejected: 1). webhook_secret appears only when we generated it: store it to verify webhook signatures. Submitting is free, but needs credits for at least one URL (else 402, with a reason).

Retry-safe submits

Send an Idempotency-Key header (1–255 visible ASCII characters) so a retried submit never starts a second batch. The same key with the same body returns the original 202 with an idempotent-replayed: true header. The same key with a different body is a 409 idempotency_conflict. Keys are per account and last as long as the batch (30 days). Details in Async jobs.

curl -X POST "https://api.104-236-84-15.sslip.io/v1/batch" \
  -H "Authorization: Bearer $PAGETEER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: listings-2026-10-02" \
  -d @batch.json

Check progress

{
  "id": "6ac0f917934df03715b56c42",
  "status": "done",
  "counts": { "total": 3, "queued": 0, "running": 0, "succeeded": 2, "failed": 1, "canceled": 0 },
  "outcomes": { "forbidden_target": 1, "ok": 2 },
  "credits_used": 2,
  "error": null,
  "metadata": { "run": "docs-audit" },
  "webhook": { "url": "https://hooks.example.com/pageful", "events": ["batch.completed"] },
  "created_at": "2026-10-03T12:46:15.308Z",
  "updated_at": "2026-10-03T12:46:17.432Z",
  "completed_at": "2026-10-03T12:46:17.432Z"
}

• status: queued → running → done, failed or canceled. A batch is done when every URL has finished, whether it succeeded or not; stop polling at any of the last three.
• counts are per URL status; outcomes counts URLs per outcome code. New codes may appear over time, so treat the keys as open-ended.
• error is set when the batch stopped early (credits ran out, or the account was suspended: failed).

Polling is free. A good loop polls every few seconds with backoff; see Async jobs & polling, or let a webhook tell you instead.

Get results

curl "https://api.104-236-84-15.sslip.io/v1/batch/6ac0f917934df03715b56c42/results?limit=2" \
  -H "Authorization: Bearer $PAGETEER_API_KEY"

const res = await fetch(`https://api.104-236-84-15.sslip.io/v1/batch/6ac0f917934df03715b56c42/results?${new URLSearchParams({"limit":"2"})}`, {
  headers: { Authorization: `Bearer ${process.env.PAGETEER_API_KEY}` },
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.get(
    "https://api.104-236-84-15.sslip.io/v1/batch/6ac0f917934df03715b56c42/results",
    headers=headers,
    params={
        "limit": 2,
    },
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

{
  "items": [
    {
      "index": 0,
      "id": "ex-1",
      "url": "https://example.com",
      "status": "succeeded",
      "outcome": "ok",
      "http_status": 200,
      "final_url": "https://example.com",
      "title": "Example Domain",
      "tier_used": "none",
      "credits": 1,
      "attempts": 1,
      "error": null,
      "metadata": { "source": "docs" },
      "result": {
        "markdown": "This domain is for use in documentation examples without needing permission. …",
        "data": { "heading": null }
      },
      "completed_at": "2026-10-03T12:46:17.427Z"
    },
    {
      "index": 1,
      "id": "iana-1",
      "url": "https://www.iana.org/domains/reserved",
      "status": "succeeded",
      "outcome": "ok",
      "http_status": 200,
      "final_url": "https://www.iana.org/domains/reserved",
      "title": "IANA-managed Reserved Domains",
      "tier_used": "none",
      "credits": 1,
      "attempts": 1,
      "error": null,
      "metadata": null,
      "result": {
        "markdown": "# IANA-managed Reserved Domains\n\nCertain domains are set aside, …",
        "data": { "heading": "IANA-managed Reserved Domains" }
      },
      "completed_at": "2026-10-03T12:46:16.849Z"
    }
  ],
  "next": 1
}

Items come in index order. Pass next as after to get the next page, until next is null (see pagination). Results are readable while the batch runs, so you can stream them out as they finish.

Item fields

Download everything

format=jsonl streams one item per line (the same objects); format=csv streams one row per URL with the columns index, id, url, status, outcome, http_status, final_url, title, tier_used, credits, attempts, error_code, error_message, completed_at, metadata (as JSON) and one data.<name> column per selector. Both come as attachments named batch_<id>.jsonl / .csv; status and include_content filter them too. Results are kept for 30 days. You can also download them from the Batches page in the dashboard.

curl "https://api.104-236-84-15.sslip.io/v1/batch/6ac0f917934df03715b56c42/results?format=jsonl" \
  -H "Authorization: Bearer $PAGETEER_API_KEY" -o results.jsonl

index,id,url,status,outcome,http_status,final_url,title,tier_used,credits,attempts,error_code,error_message,completed_at,metadata,data.heading
0,ex-1,https://example.com,succeeded,ok,200,https://example.com,Example Domain,none,1,1,,,2026-10-03T12:46:17.427Z,"{""source"":""docs""}",
2,bad,http://10.0.0.1/,failed,forbidden_target,,,,,0,0,forbidden_target,"Private, loopback and reserved network targets are not allowed",2026-10-03T12:46:15.308Z,,

List batches

Your most recent batches, newest first. limit: 1–100, default 20 (anything else is a 400).

{
  "batches": [
    {
      "id": "6ac0f917934df03715b56c42",
      "status": "done",
      "total": 3,
      "succeeded": 2,
      "failed": 1,
      "canceled": 0,
      "credits_used": 2,
      "metadata": { "run": "docs-audit" },
      "created_at": "2026-10-03T12:46:15.308Z",
      "completed_at": "2026-10-03T12:46:17.432Z"
    }
  ]
}

Cancel

Returns the batch (same shape as GET) with status: "canceled". URLs not fetched yet become canceled and are not charged, including any in flight at that moment. Results already fetched stay available, and a batch.completed webhook is sent. Canceling a finished batch changes nothing, so it is safe to repeat.

Outcomes and retries

Every URL ends succeeded, failed or canceled. We retry timeouts, errors on our side and site 5xx responses, up to 3 attempts with growing delays (from 30 seconds). When a site asks us to slow down (target_rate_limited), we pause that site for everyone and come back after its Retry-After, without using up an attempt (up to 10 times). We never solve CAPTCHAs or log in: those URLs, like robots.txt disallows and blocks, fail straight away, are never retried and are free:

In your usage log (dashboard and request logs) each batch URL appears with the status Pageteer returned (200 for a fetched page), not the site's; the site's status is the item's http_status. A URL canceled while in flight is refunded and logged with 0 credits.

Other codes from the errors list can appear too, and so can new ones. Treat unknown codes as failures for that URL. To re-run only the failures, fetch ?status=failed&include_content=false and submit those URLs as a new batch.

Pacing

A batch never floods a single site. Each target host gets one batch request every couple of seconds, across all batches, so 1,000 URLs on one site trickle in over about half an hour while URLs on other sites run in parallel. Batches on the same site take turns, so a small batch is not stuck behind a large one. A batch uses at most half your plan's concurrency, so your live API calls keep working.

Webhooks

With webhook set, we POST a signed batch.completed event when the batch finishes (also on cancel), and optionally a batch.item.completed event per URL. Payloads, retries and signature verification in Node and Python are on the Webhooks page.

Credits

Each successful URL costs the same as a render with the same mode and tier: 1,000 pages in HTTP mode cost 1,000 credits, in browser mode 5,000. Failed, rejected and canceled URLs are free, and so are submitting, polling and downloading results. When credits run out mid-batch, the batch stops (done, with error set) and the remaining URLs fail as insufficient_credits or spend_limit_reached.

Errors

Errors of the API calls themselves (per-URL failures are outcomes, above):