Browser API

Map

List a site's URLs without fetching their content: what its sitemaps list plus the links on the start page, in one quick request. Use it to pick pages before a batch or a crawl.

POST https://api.104-236-84-15.sslip.io/v1/map

Body parameters

Map parameters
Name	Type	Description
`url`required	string	Any page on the site. Its links are read, and sitemap discovery starts at its origin.
`limit`	integer (1–5,000)	At most this many URLs are returned; truncated says whether more matched. Default: `1000`.
`search`	string	Keep only URLs containing every space-separated term, ignoring case (`blog 2026`). Matched against the URL as sent and percent-decoded.
`same_path`	boolean	Keep only URLs under the start URL's path: `https://a.com/docs` keeps `/docs` and `/docs/…`, not `/docsearch`. Default: `false`.
`include_subdomains`	boolean	Also keep URLs on subdomains of the start host (`blog.a.com` for `a.com`). By default only the start host is kept, with or without `www.` Default: `false`.
`use_sitemap`	boolean	Read the sitemaps listed in robots.txt and /sitemap.xml (sitemap indexes and .gz files are followed). false = only the links on the start page. Default: `true`.
`respect_robots`	boolean	Honor robots.txt (user agent PageteerBot): disallowed URLs are left out, and a disallowed start page is not fetched (its sitemaps are still read). Default: `true`.
`timeout_ms`	integer (1000–120000)	Time budget for the whole request. Sitemap files not read by then are skipped (truncated). Default: `30000`.

How URLs are found

robots.txt is read first (with respect_robots) to filter URLs.
Sitemaps: every Sitemap: line in robots.txt plus /sitemap.xml, following sitemap indexes, gzip and plain-text sitemaps (up to 25 files and 50,000 URLs).
The start page is fetched once over plain HTTP (no browser, no JavaScript) and its links are added after the sitemap URLs.

Pages are never fetched beyond the start page, so map is fast and cheap, but it only sees what sitemaps list and the start page links to. Links that JavaScript adds are missed; for those, run a crawl in browser mode.

A start page that fails (blocked, a CAPTCHA, a login wall, a timeout) fails the request only when the sitemaps gave no URL either. Otherwise you get the sitemap URLs. All target guards apply as on render: private addresses and denied domains are refused.

Example

curl

curl -X POST https://api.104-236-84-15.sslip.io/v1/map \
  -H "Authorization: Bearer $PAGETEER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/blog","search":"pricing","limit":100}'

JavaScript (fetch)

const res = await fetch('https://api.104-236-84-15.sslip.io/v1/map', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.PAGETEER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "url": "https://example.com/blog",
    "search": "pricing",
    "limit": 100
  }),
});
const data = await res.json();
if (!res.ok) throw new Error(data.error.message);
console.log(data, res.headers.get('x-credits-remaining'));

Python (requests)

import os
import requests

headers = {"Authorization": f"Bearer {os.environ['PAGETEER_API_KEY']}"}
res = requests.post(
    "https://api.104-236-84-15.sslip.io/v1/map",
    headers=headers,
    json={
        "url": "https://example.com/blog",
        "search": "pricing",
        "limit": 100,
    },
    timeout=120,
)
res.raise_for_status()
print(res.json(), res.headers["x-credits-remaining"])

Response

200 OK

x-credits-used: 1

{
  "url": "https://example.com/blog",
  "urls": [
    { "url": "https://example.com/blog/pricing-update", "source": "sitemap", "lastmod": "2026-10-02" },
    { "url": "https://example.com/pricing", "source": "sitemap" },
    { "url": "https://example.com/pricing/enterprise", "source": "links" }
  ],
  "count": 3,
  "truncated": false,
  "credits": 1
}

Map response fields
Name	Type	Description
`url`	string	The URL you sent.
`urls[]`	object[]	`{ url, source, lastmod? }`, sitemap URLs first (in sitemap order), then the start page's links. `source` is `sitemap` or `links` (a URL in both is `sitemap`); `lastmod` is the sitemap's `<lastmod>`, verbatim, when it had one. URLs are normalized: no `#fragment`, lower-case host, no default port.
`count`	integer	Number of URLs returned.
`truncated`	boolean	true when more URLs matched than limit, or a sitemap was cut short (too many sitemap files, a file over 50 MB, or the time budget ran out).
`credits`	integer	Credits charged; same as x-credits-used.

Credits

1 credit per request, however many URLs come back. A request that finds no URL is free, and so is every error. Test keys (pk_test_…) get a fixed site map for free; magic URLs such as https://test.pageteer.com/error/captcha return the matching error (Test keys).

Errors

Map errors
Status	Code	When	Charged	What to do
400	`invalid_request`	A parameter is missing, invalid or unknown (the message and error.issues name it).	No	Fix the request; retrying it unchanged fails again.
401	`invalid_api_key`	No key, an unknown key, or a revoked or expired key.	No	Check the Authorization header and the key on the API keys page.
402	`insufficient_credits`	Not enough credits left this month for this request, and billable usage is off. `error.reason` says why: `free_credits_used`, `payment_failed` or `subscription_inactive`.	No	Act on the reason at `error.billing_url`: add a card, update the payment method, or restart the subscription. Or wait for the 1st (UTC).
402	`spend_limit_reached`	Billable usage is on and this month’s spend limit is reached (`error.reason`: `spend_limit`).	No	Raise the spend limit at `error.billing_url` (the Billing page), or wait for the 1st (UTC).
403	`forbidden_target`	The URL is denylisted, resolves (or redirects) to a private or internal address, or sends cookies or credentials to a login, account or checkout page.	No	Do not retry. See the Acceptable Use Policy.
403	`robots_disallowed`	robots.txt disallows the start page and the sitemaps gave no URL.	No	Skip the URL. Turn respect_robots off only for sites you own or may crawl.
403	`org_suspended`	The account is suspended.	No	Contact support.
422	`target_blocked`	The start page was blocked and the sitemaps gave no URL.	No	Try a stronger unblock tier (or "auto"); if auto already failed, retry much later.
422	`captcha_required`	The start page answered with a CAPTCHA and the sitemaps gave no URL.	No	Do not retry soon. Use the site’s API or data feed if it has one.
422	`login_required`	The site redirected to a login or sign-in page (final_url says where). We never log in.	No	The content is not public: skip it.
429	`rate_limited`	Over your requests per minute, or too many requests to one site across the platform right now.	No	Wait Retry-After seconds, then back off with jitter.
429	`concurrency_limit`	Too many of your requests are in flight at once.	No	Wait for one to finish (Retry-After: 1) or lower your parallelism.
429	`target_rate_limited`	The site asked us to slow down (429, or 503 with Retry-After), or per-site pacing would not let your request start in time. We pause that site for everyone.	No	Wait error.retry_after seconds (also in Retry-After).
504	`timeout`	The site did not answer within timeout_ms.	No	Retry, maybe with a larger timeout_ms or mode "http".
500 / 502 / 503	`internal_error`	Something failed on our side (rendering engine, LLM extraction or the API itself), or the rendering engine is at capacity (503, Retry-After: 5).	No	Retry with backoff (honor Retry-After); quote x-request-id if it persists.