relay402.georgespring.workers.dev

Checks whether a given user agent may fetch a given path on a host, per the site's robots.txt (specific-agent groups, wildcard rules, Allow/Disallow longest-match). Also returns crawl-delay, sitemap URLs and whether the site publishes llms.txt or ai.txt. For scraping and research agents that want to crawl politely and prove compliance.

Open endpoint ↗

Price per call: $0.001
Chain: Base
Method: GET
Paid calls · 30 days: 0

Call it from code

This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.

import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";

const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });

const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://relay402.georgespring.workers.dev/api/robots-check", { method: "GET" });
console.log(await res.json());

https://relay402.georgespring.workers.dev/api/robots-check