robots.txt & Sitemap Analyzer

Give it domains. For each it parses robots.txt (user-agents, allow/disallow, crawl-delay, declared sitemaps, blocks-all) and the XML sitemap(s) (URL counts, nested sitemaps, lastmod coverage), and flags crawlability issues - with a signal and findings. Keyless, read-only. Up to 3 items per call; failed lookups are not charged.

Open endpoint ↗

Price per call: $0.02
Chain: Base
Method: POST
Paid calls · 30 days: 4

Call it from code

This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.

import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";

const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });

const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://agent-data.centorprat.workers.dev/tools/robots-sitemap-analyzer", { method: "POST" });
console.log(await res.json());

https://agent-data.centorprat.workers.dev/tools/robots-sitemap-analyzer