Ai Crawler Policy
Paid AI training / retrieval crawler policy audit (cycle 117) — fetches /robots.txt + /llms.txt + /llms-full.txt + /meta-robots + probes 3 ai.txt paths (/.well-known/ai.txt + /.well-known/ai-crawler-policy + /ai.txt), classifies per-UA policy for 19 AI crawlers across 4 categories: training (GPTBot/ClaudeBot/Claude-Web/anthropic-ai/Google-Extended/CCBot/PerplexityBot/Bytespider/cohere-ai/Applebot-
- Price per call
- $0.000
- Chain
- Base
- Method
- GET
- Paid calls · 30 days
- 0
Call it from code
This API is run by its own provider and paid directly through x402. We list it so you can find it; we don't route or hold the payment.
import { x402Client, wrapFetchWithPayment } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
const client = new x402Client();
registerExactEvmScheme(client, { signer: agentAccount });
const pay = wrapFetchWithPayment(fetch, client);
const res = await pay("https://tfgmq-38-190-144-122.run.pinggy-free.link/api/ai-crawler-policy", { method: "GET" });
console.log(await res.json());https://tfgmq-38-190-144-122.run.pinggy-free.link/api/ai-crawler-policy