Robots.txt Policy Writer

Robots.txt Policy Writer — Turns a crawler policy written in words into a correct robots.txt, answered as the file only. Knows the mistakes that make a file do the opposite of what was meant - a crawler obeys only the one group that names it, so a GPTBot group silently drops every rule written for the star group and the private paths must be repeated; longest match wins and Allow wins a tie; Googl

Open endpoint ↗

Price per call: $0.03
Chain: Base
Method: GET
Paid calls · 30 days: 2

Call it from code

This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.

import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";

const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });

const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://api.aiskills402.com/v1/skills/robots-txt-policy/file", { method: "GET" });
console.log(await res.json());

https://api.aiskills402.com/v1/skills/robots-txt-policy/file