Guides

Webhooks

Instead of polling a batch, let it call you: we POST signed JSON events to your URL when the batch finishes and, if you like, after every URL.

Set up

Add webhook to POST /v1/batch:

Request body

{
  "urls": ["https://example.com/a", "https://example.com/b"],
  "webhook": {
    "url": "https://hooks.example.com/pageful",
    "events": ["batch.completed", "batch.item.completed"],
    "secret": "a-long-random-string-you-keep"
  }
}

url: a public http(s) URL. It is checked like a target URL (it must resolve, no private or internal addresses, not denylisted) when you submit and again before every delivery.
events: batch.completed (the default) and/or batch.item.completed.
secret: 16–256 characters. Leave it out and we generate one (whsec_…) and return it once, as webhook_secret in the 202. Store it: it is never shown again.

Webhooks are per batch; crawls do not send them (poll them instead). Monitors take the same webhook object for their monitor.changed and monitor.paused events, signed and retried the same way.

What we send

batch.completed

POST /pageful HTTP/1.1
content-type: application/json
user-agent: Pageteer-Webhooks/1.0
pageteer-event: batch.completed
pageteer-delivery: 6ac0f91a934df03715b56c58
pageteer-signature: t=1759495577,v1=5f1c0e…

{
  "id": "evt_6ac0f91a934df03715b56c57",
  "type": "batch.completed",
  "created_at": "2026-10-03T12:46:17.433Z",
  "data": {
    "batch": {
      "id": "6ac0f917934df03715b56c42",
      "status": "done",
      "counts": { "total": 3, "queued": 0, "running": 0, "succeeded": 2, "failed": 1, "canceled": 0 },
      "outcomes": { "forbidden_target": 1, "ok": 2 },
      "credits_used": 2,
      "error": null,
      "metadata": { "run": "docs-audit" },
      "webhook": { "url": "https://hooks.example.com/pageful", "events": ["batch.completed"] },
      "created_at": "2026-10-03T12:46:15.308Z",
      "updated_at": "2026-10-03T12:46:17.432Z",
      "completed_at": "2026-10-03T12:46:17.432Z"
    },
    "results_url": "https://api.104-236-84-15.sslip.io/v1/batch/6ac0f917934df03715b56c42/results?format=jsonl"
  }
}

Webhook events
Name	Type	Description
`batch.completed`	once per batch	When the batch ends: `done`, `canceled` or `failed`. `data.batch` is the same object as `GET /v1/batch/:id`; `data.results_url` downloads every result as JSONL (send your API key with it).
`batch.item.completed`	once per fetched URL	After each URL the worker finished (succeeded or failed): `data = { batch_id, item }`, where `item` is a result item. If the event would be over 1 MB, `item.result` is `null` and `item.result_omitted` is `true`: fetch it from the results endpoint. URLs rejected at submit or canceled send no item event.
`monitor.changed`	once per detected change	A monitor found a change: `data = { monitor, change }`, with the diff (sent without it over 1 MB, `diff_omitted: true`).
`monitor.paused`	when a monitor pauses itself	`data = { monitor, reason, error? }`: repeated failures, credits or the spend limit ran out, the organization was suspended, or the plan lost a feature it uses.

Webhook headers
Name	Type	Description
`pageteer-event`	string	The event type, same as type in the body.
`pageteer-delivery`	string	Id of this delivery (the same on every retry of it).
`pageteer-signature`	t=<unix>,v1=<hex>	HMAC-SHA256 signature of the body; see below. t changes on every attempt.

Delivery and retries

Answer with any 2xx within 10 seconds. Do slow work after answering (or in a queue).
Anything else (a non-2xx status, a timeout, a connection error) is retried with exponential backoff: about 30 s, 1 min, 2 min, 4 min, … (±20%, at most 6 hours apart), for up to 8 attempts in all. Then the delivery is dropped; the batch and its results are unaffected.
Redirects are not followed: point the webhook at the final URL.
A URL that stops passing the checks (for example it now resolves to a private address) is not retried.
Deliveries can arrive more than once and out of order (item events can arrive after batch.completed). Deduplicate on the body's id, and treat GET /v1/batch/:id as the source of truth.

Verify the signature

pageteer-signature is t=<unix seconds>,v1=<hex>, where the hex is the HMAC-SHA256 of <t>.<raw body> keyed with your webhook secret. To verify:

Read the raw request body, before any JSON parsing. Re-serialized JSON will not match.
Compute HMAC-SHA256 of t + "." + body with your secret and compare it to v1 in constant time.
Reject timestamps more than 5 minutes from now, so a captured request cannot be replayed later.

Node.js

verify.js

import { createHmac, timingSafeEqual } from 'node:crypto';

/** true when `header` (pageteer-signature) is a valid, recent signature of `rawBody`. */
export function verifyPageteerSignature(secret, rawBody, header, toleranceSec = 300) {
  const parts = Object.fromEntries(
    String(header ?? '')
      .split(',')
      .map(part => part.trim().split('=', 2))
  );
  const t = Number(parts.t);
  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > toleranceSec) return false;
  const expected = createHmac('sha256', secret).update(`${parts.t}.${rawBody}`).digest();
  const received = Buffer.from(parts.v1 ?? '', 'hex');
  return received.length === expected.length && timingSafeEqual(received, expected);
}

Express

import express from 'express';

const app = express();
const seen = new Set(); // use your database in production

// Verify against the raw bytes: re-serialized JSON will not match the signature.
app.post('/pageful', express.raw({ type: 'application/json' }), async (req, res) => {
  if (!verifyPageteerSignature(process.env.PAGETEER_WEBHOOK_SECRET, req.body, req.get('pageteer-signature'))) {
    return res.sendStatus(400);
  }
  const event = JSON.parse(req.body.toString('utf8'));
  if (seen.has(event.id)) return res.sendStatus(200); // a redelivery
  seen.add(event.id);
  res.sendStatus(200); // answer fast (within 10 s), then do the work
  if (event.type === 'batch.completed') await importResults(event.data.batch.id);
});

Python

verify.py

import hashlib
import hmac
import time


def verify_pageful_signature(secret: str, raw_body: bytes, header: str, tolerance: int = 300) -> bool:
    """True when header (pageteer-signature) is a valid, recent signature of raw_body."""
    parts = dict(part.strip().split("=", 1) for part in header.split(",") if "=" in part)
    t, v1 = parts.get("t", ""), parts.get("v1", "")
    if not t.isdigit() or abs(time.time() - int(t)) > tolerance:
        return False
    expected = hmac.new(secret.encode(), t.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, v1)

Flask

import json
import os

from flask import Flask, request

app = Flask(__name__)


@app.post("/pageful")
def pageful_webhook():
    raw = request.get_data()  # the raw bytes, before any JSON parsing
    if not verify_pageful_signature(os.environ["PAGETEER_WEBHOOK_SECRET"], raw, request.headers.get("pageteer-signature", "")):
        return "", 400
    event = json.loads(raw)
    # Deduplicate on event["id"]; answer within 10 seconds and queue slow work.
    if event["type"] == "batch.completed":
        enqueue_import(event["data"]["batch"]["id"])
    return "", 200

Both samples are tested against the signatures our webhook sender produces. If verification fails, check that you hash the raw bytes and that you use the secret of this batch: each batch has its own.