Crawl

Body parameters

Crawls do not take wait_for, timeout_ms, headers, cookies or max_age: each page uses the default 30-second budget. Need those per URL? Use batch.

Start a crawl

curl -X POST https://api.104-236-84-15.sslip.io/v1/crawl \
  -H "Authorization: Bearer $PAGETEER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.iana.org/help/example-domains","max_pages":2,"max_depth":1,"mode":"http","include":["/help/**","/domains/**"]}'

const res = await fetch('https://api.104-236-84-15.sslip.io/v1/crawl', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.PAGETEER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "url": "https://www.iana.org/help/example-domains",
    "max_pages": 2,
    "max_depth": 1,
    "mode": "http",
    "include": [
      "/help/**",
      "/domains/**"
    ]
  }),
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.post(
    "https://api.104-236-84-15.sslip.io/v1/crawl",
    headers=headers,
    json={
        "url": "https://www.iana.org/help/example-domains",
        "max_pages": 2,
        "max_depth": 1,
        "mode": "http",
        "include": ["/help/**", "/domains/**"],
    },
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

{ "job_id": "6ac0f90f934df03715b56c1c" }

Starting a crawl is free, but needs credits for at least one page (else 402, with a reason). The start URL goes through the usual checks first (403 forbidden_target for private or denylisted hosts).

Retry-safe starts

Send an Idempotency-Key header (1–255 visible ASCII characters) and a retried start never creates a second crawl: the same key with the same body returns the original 202 with idempotent-replayed: true; the same key with a different body is 409 idempotency_conflict. See Async jobs.

Poll for results

curl "https://api.104-236-84-15.sslip.io/v1/crawl/6ac0f90f934df03715b56c1c" \
  -H "Authorization: Bearer $PAGETEER_API_KEY"

const res = await fetch('https://api.104-236-84-15.sslip.io/v1/crawl/6ac0f90f934df03715b56c1c', {
  headers: { Authorization: `Bearer ${process.env.PAGETEER_API_KEY}` },
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.get(
    "https://api.104-236-84-15.sslip.io/v1/crawl/6ac0f90f934df03715b56c1c",
    headers=headers,
    params={},
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

{
  "id": "6ac0f90f934df03715b56c1c",
  "status": "done",
  "pages": [
    {
      "url": "https://www.iana.org/help/example-domains",
      "depth": 0,
      "credits": 1,
      "tier_used": "none",
      "final_url": "https://www.iana.org/help/example-domains",
      "status": 200,
      "title": "Example Domains",
      "markdown": "# Example Domains\n\nA number of domains such as `example.com`…"
    },
    {
      "url": "https://www.iana.org/domains",
      "depth": 1,
      "credits": 1,
      "tier_used": "none",
      "final_url": "https://www.iana.org/domains",
      "status": 200,
      "title": "Domain Name Services",
      "markdown": "# Domain Name Services\n\nWe operate and maintain a number of …"
    }
  ],
  "progress": { "done": 2, "failed": 0, "queued": 20, "total": 2 },
  "credits_used": 2,
  "error": null,
  "created_at": "2026-10-03T12:46:07.126Z",
  "updated_at": "2026-10-03T12:46:10.824Z"
}

• status: queued → running → done, failed or canceled. Stop polling at any of the last three.
• progress: done pages fetched, failed pages that errored, queued URLs still waiting in the frontier (not all will be visited) and total = max_pages.
• pages grows as the crawl runs, in visit order; credits_used is the total so far.
• error explains an early stop even when status is done (see When a crawl stops early).
• sitemap appears when the job was seeded from sitemaps (below).

Poll every few seconds with backoff; polling is free and counts toward your rate limit. See Async jobs & polling for a complete loop. Your crawl jobs are also listed in the dashboard. Jobs and their pages are kept for 30 days after they are created, then deleted.

Page fields

Cancel

Stops a queued or running crawl and returns the job (same shape as GET) with status: "canceled". Pages already fetched stay in the job and stay charged; a page that was in flight is refunded. Canceling a finished job changes nothing and returns it as it is, so it is safe to repeat.

Export a site

Downloads a finished crawl (done, failed or canceled) as one file. Free: the pages were charged when they were crawled. It is streamed, so start writing it to disk straight away.

curl "https://api.104-236-84-15.sslip.io/v1/crawl/6ac0f90f934df03715b56c1c/export?format=llms.txt" \
  -H "Authorization: Bearer $PAGETEER_API_KEY" -o llms.txt

curl "https://api.104-236-84-15.sslip.io/v1/crawl/6ac0f90f934df03715b56c1c/export?format=zip" \
  -H "Authorization: Bearer $PAGETEER_API_KEY" -o site.zip

# Example Domains

> As described in RFC 2606 and RFC 6761, a number of domains such as example.com are reserved for documentation.

Crawled from https://www.iana.org/help/example-domains on 2026-10-03.

## Pages

- [Example Domains](https://www.iana.org/help/example-domains): As described in RFC 2606 and RFC 6761, a number of domains…
- [Domain Name Services](https://www.iana.org/domains): We operate and maintain a number of key parts of the DNS…

• Markdown, llms.txt and zip need page content: crawl with formats: ["markdown"] (the default) or text. Failed pages and pages without content are left out (Markdown says how many); JSONL keeps everything.
• A crawl that is still queued or running is 409 conflict. A crawl whose stored pages are too large for one file is 413 payload_too_large; if an export would grow past the limit while streaming, it ends with an explicit truncation note instead of stopping silently.
• The dashboard's crawl page has the same downloads under Download.

Sitemap seeding

Most sites list their pages in a sitemap. Reading it enumerates every listing, store or product page without loading the HTML pages in between, so a crawl makes fewer page loads and is less likely to trip a site's bot defenses.

• Discovery: the Sitemap: lines in the site's robots.txt, or /sitemap.xml when there are none. Sitemap indexes are followed, and gzipped (.xml.gz) and plain-text sitemaps are read. A start URL that ends in .xml or .xml.gz is used as the sitemap itself and is not rendered.
• sitemap: "only" visits only the sitemap URLs and follows no links. The start URL just names the site. "include" visits the start URL, then the sitemap URLs, and keeps following links as usual.
• Sitemap URLs go through the same filters as links: the start host only, include / exclude and robots.txt. Pages are visited in sitemap order, up to max_pages.
• Reading sitemaps is free; you pay only for the pages visited. At most 25 sitemap files and 50,000 URLs are read per job. The job's sitemap field reports {urls, seeded, sitemaps, truncated}: the URLs found, the URLs queued after filtering, the sitemap files read, and whether a limit was hit.
• With "only", a job whose sitemaps yield no usable URL ends as failed with nothing charged.

curl -X POST https://api.104-236-84-15.sslip.io/v1/crawl \
  -H "Authorization: Bearer $PAGETEER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://realty.example/","sitemap":"only","include":["/homes/**"],"mode":"http","formats":["structured"],"max_pages":500}'

const res = await fetch('https://api.104-236-84-15.sslip.io/v1/crawl', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.PAGETEER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "url": "https://realty.example/",
    "sitemap": "only",
    "include": [
      "/homes/**"
    ],
    "mode": "http",
    "formats": [
      "structured"
    ],
    "max_pages": 500
  }),
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.post(
    "https://api.104-236-84-15.sslip.io/v1/crawl",
    headers=headers,
    json={
        "url": "https://realty.example/",
        "sitemap": "only",
        "include": ["/homes/**"],
        "mode": "http",
        "formats": ["structured"],
        "max_pages": 500,
    },
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

When a crawl stops early

• Credits run out: the job ends done with the pages fetched so far and error starting with insufficient_credits or spend_limit_reached (see 402 reasons).
• CAPTCHA: the site answered with a CAPTCHA or bot challenge. That page is free, and the job ends done with error starting with captcha_required. We do not solve CAPTCHAs and do not keep requesting pages from a site that shows one.
• The start URL is disallowed by robots.txt (with respect_robots): the job ends failed (error: "robots.txt disallows the start URL").
• Sitemaps unusable (with sitemap: "only"): no URL left after the filters, or the sitemaps could not be read (error starts with sitemap_unavailable): failed, nothing charged.
• The account is suspended: failed.
• Our engine keeps failing (5 pages in a row): failed with error: "engine_unavailable". Failed pages are free; start the crawl again later.

Single pages that fail (blocked, timeouts, a login wall, a robots.txt disallow) do not stop the crawl: they are recorded with error, counted in progress.failed and not charged. When a page gets 429 (or 503 with Retry-After) the crawl waits that long and retries the page, up to three times, instead of pushing on.

Credits

Each page costs the same as a render with the same mode and tier: 1 in HTTP mode, 5 in a browser. 200 pages in browser mode cost 1,000 credits. Reading robots.txt and sitemaps, polling and canceling are free; failed pages are free.

Errors

Errors of the API calls themselves (page failures are recorded in the job instead):