Forest Web

Interpret the robots.txt crawling rules that apply to a specific public URL and user-agent before an automated fetch or crawl. Returns allow/disallow decision evidence, matched rules, crawl delay and sitemap hints from one bounded robots.txt read. This is protocol guidance, not authorization or an anti-bot bypass. Base mainnet USDC

Open endpoint ↗

Price per call: $0.001
Chain: Base
Method: POST
Paid calls · 30 days: 3

Call it from code

This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.

import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";

const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });

const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/robots/context", { method: "POST" });
console.log(await res.json());

https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/robots/context