Crawl Pre-Flight
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
crawl-preflight.postnov01.workers.devBasePolygonArbitrum last paid 13h ago
- Category
- Search & web
- Starting price
- $0.02
- Paid calls · 30 days
- 2
- Unique payers · 30 days
- 1
Paid endpoints
1 endpoint, sorted by usage. Prices are per request.