Crawl Pre-Flight

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Open endpoint ↗

Price per call: $0.02
Chain: Base
Method: GET
Paid calls · 30 days: 2

Call it from code

This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.

import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";

const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });

const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain", { method: "GET" });
console.log(await res.json());

https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain