Search / com.lumnika/ai-crawler-readiness
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
v1.0.0 · active · website · descriptor JSON · versions
protogrid{
"mcpServers": {
"ai-crawler-readiness": {
"type": "http",
"url": "https://shop.lumnika.com/ai-readiness/mcp"
}
}
}claude mcp add --transport http ai-crawler-readiness https://shop.lumnika.com/ai-readiness/mcp[mcp_servers.ai-crawler-readiness]
url = "https://shop.lumnika.com/ai-readiness/mcp"
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"ai-crawler-readiness": {
"type": "remote",
"url": "https://shop.lumnika.com/ai-readiness/mcp",
"enabled": true
}
}
}{
"mcpServers": {
"ai-crawler-readiness": {
"type": "http",
"url": "https://shop.lumnika.com/ai-readiness/mcp"
}
},
"deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=ai-crawler-readiness&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vc2hvcC5sdW1uaWthLmNvbS9haS1yZWFkaW5lc3MvbWNwIn0%3D"
}{
"servers": {
"ai-crawler-readiness": {
"type": "http",
"url": "https://shop.lumnika.com/ai-readiness/mcp"
}
}
}{
"mcpServers": {
"ai-crawler-readiness": {
"httpUrl": "https://shop.lumnika.com/ai-readiness/mcp"
}
}
}extensions:
"ai-crawler-readiness":
enabled: true
name: "ai-crawler-readiness"
type: streamable_http
uri: "https://shop.lumnika.com/ai-readiness/mcp"
timeout: 300
protocol.modern, protocol.stateless, protocol.transport, protocol.list_ttlno remote answered a protocol handshake (not probed yet, unreachable, or local-only)auth.prm, auth.as_metadata, auth.cimdno remote uses OAuthauth.secret_in_urlno templated URLtools.descriptionsevery tool has a descriptiontools.description_lengthdescriptions are concisetools.schemasevery tool has an object input schematools.annotationsno tool declares readOnlyHint or destructiveHint, so clients cannot tell safe calls from risky onestools.directory_hintsno tool declares all four of readOnlyHint, destructiveHint, idempotentHint and openWorldHint; missing: domain_readiness (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); web_openness (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); edge_blocking (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); …tools.token_costabout 839 tokens to load every tooltools.api_dumptools are not a one-to-one API dumpstability.changes, stability.rug_pulltool history not recorded yetdeps.known_vulns, deps.mcp_sdk_version, deps.resolvableno npm or PyPI packageNo tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.
Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.
+dns-namespace +website +reachable +uptime-98% ~updated-31d-ago -no-repository
Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.
| remote | auth | reachable | uptime 30d | p50 | protocol | last ok |
|---|---|---|---|---|---|---|
https://shop.lumnika.com/ai-readiness/mcp streamable-http | unknown | yes | 98% | 438 ms | – | 2026-10-20 |
domain_readiness | Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns `fixes`: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it. |
edge_blocking | Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others. |
measure_domain | Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same `fixes` list as domain_readiness, derived from the fresh measurement. |
recent_changes | What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time. |
web_openness | How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler. |
Schemas: list_tools.
{
"name": "com.lumnika/ai-crawler-readiness",
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-09-29/server.schema.json",
"remotes": [
{
"url": "https://shop.lumnika.com/ai-readiness/mcp",
"type": "streamable-http"
}
],
"version": "1.0.0",
"websiteUrl": "https://shop.lumnika.com/ai-readiness/",
"description": "Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index"
}