Agent-Ready Website Kit

Make a site readable for AI crawlers and agents. Give us a URL. We check how 17 AI crawlers and agents (GPTBot, ClaudeBot, Claude-SearchBot, OAI-SearchBot, PerplexityBot, Google-Extended, ChatGPT-User, Perplexity-User and more) can reach the site, including live user-agent probes that catch firewall blocks robots.txt doesn't show, score six areas and generate ready-to-install files. This removes technical barriers. It can't guarantee that any AI system will crawl, rank, cite, or send traffic to your site.

Calls

Route	Price	What you get
`POST /api/skills/agent-ready-audit`	$0.0502 (50200 atomic)	score, per-bot table, checks with fixes, preview of llms.txt and the robots AI block
`POST /api/skills/agent-ready-kit`	$0.2502 (250200 atomic)	the audit plus every generated file and a zip (base64)
`POST /api/skills/agent-ready-recheck`	$0.0102 (10200 atomic)	`{"report_id"}` from an audit in the last 30 days: re-audit + what got fixed or regressed
`GET /api/skills/agent-ready/sample`	free	full example report for a demo site

License: the kit zip is covered by the Grokzilla No-Resale License (included as LICENSE.md). Use and change the files for your own sites and client work; don't resell or republish the kit as a kit or template.

{"url": "https://example.com/", "max_pages": 10, "crawler_policy": "allow_search_block_training",
 "business": {"name": "Example Co", "logo": "https://example.com/logo.png", "same_as": ["https://x.com/example"]}}

site_type: auto, saas, api, ecommerce, publisher, local_business, docs.
crawler_policy: allow_all_ai, allow_search_block_training (default: search and user-triggered agents must be allowed; blocking training bots is your choice and is not penalized) or custom with custom_policy: {"GPTBot": "block", ...}.
business: facts for the generated files. Anything missing stays TODO; nothing is invented.
max_pages: 1-25 (default 10), sampled from your sitemap and homepage links.
render_js: not available yet; true is refused with 400 before payment.
override_full_disallow (boolean, default false): if your robots.txt says Disallow: / for everyone, the generated robots.txt does not open the site to the AI bots your policy allows. It adds a WARNING line and a conflict note in INSTALL.md instead. Send true only if you really want those bots allowed despite the site-wide block. Non-boolean values get 400 before payment.
runs_a2a_agent (boolean, default false): send true only if the site really runs an A2A agent. Then the kit generates .well-known/agent-card.json and the agent card is scored; otherwise it is N/A and costs no points.

Rubric (100 points)

Crawler access 30 (robots.txt, per-bot policy match, live UA probes, noindex), machine-readable summary 15 (llms.txt per llmstxt.org v2, llms-full.txt, llms.txt links resolve), structured data 20, discoverability 15 (sitemap, canonical host, meta/OG), content readability 10, agent interfaces 10 (OpenAPI, A2A agent card, MCP, /.well-known/x402). Agent interfaces are N/A for sites without an API, and the score is reweighted.

Generated files (kit)

llms.txt, llms-full.txt, robots.txt (your existing rules kept verbatim, AI block appended; allowed bots repeat your User-agent: * rules because a named group replaces it under RFC 9309), sitemap.xml (audited host only), JSON-LD and meta/OG per page, .well-known/agent-card.json (only if you run an A2A agent), OpenAPI hints, and INSTALL.md with Cloudflare/Vercel firewall notes when probes are blocked. ai-plugin.json is never generated (ChatGPT plugins were retired in 2024).

Probes are sent from our servers with each bot's published user-agent string. CDNs that verify real crawlers by IP can treat that differently from the real bot, so a block is strong evidence, not proof.

Errors (not charged)

400 before payment: invalid/private/loopback URL, the marketplace's own hosts, bad enum values, max_pages over 25, non-boolean override_full_disallow/runs_a2a_agent, a recheck without report_id, an invalid or expired report_id.

Blocked or unreachable sites are not charged. Your payment is verified first and only settled after a real result (a non-null score). If the site blocks us (403/429/challenge page) or is unreachable, you get 422 with the blocked result (score: null, status, reason) and charged: false, and the payment is never settled. If settlement fails after a good result, you get 402 and no result.