Search / io.github.kriton-dev/ai-crawler-access-check
AI Crawler Checker — Robots.txt Policy for Six Bots
R0· no sign-inCheck declared robots.txt permissions for six AI crawlers at URL paths
v1.0.2 · active · website · descriptor JSON · versions
Search / io.github.kriton-dev/ai-crawler-access-check
R0· no sign-inCheck declared robots.txt permissions for six AI crawlers at URL paths
v1.0.2 · active · website · descriptor JSON · versions
{
"mcpServers": {
"ai-crawler-access-check": {
"type": "http",
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"headers": {
"Authorization": "Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"
}
}
}
}claude mcp add --transport http ai-crawler-access-check https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check --header "Authorization: Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"[mcp_servers.ai-crawler-access-check]
url = "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
bearer_token_env_var = "AI_CRAWLER_ACCESS_CHECK_TOKEN"
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"ai-crawler-access-check": {
"type": "remote",
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"enabled": true,
"headers": {
"Authorization": "Bearer {env:AI_CRAWLER_ACCESS_CHECK_TOKEN}"
}
}
}
}{
"mcpServers": {
"ai-crawler-access-check": {
"type": "http",
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"headers": {
"Authorization": "Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"
}
}
},
"deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=ai-crawler-access-check&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vbWNwLmFwaWZ5LmNvbS8%2FdG9vbHM9aGVybWVza2QvYWktY3Jhd2xlci1hY2Nlc3MtY2hlY2siLCJoZWFkZXJzIjp7IkF1dGhvcml6YXRpb24iOiJCZWFyZXIgJHtBSV9DUkFXTEVSX0FDQ0VTU19DSEVDS19UT0tFTn0ifX0%3D"
}{
"servers": {
"ai-crawler-access-check": {
"type": "http",
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"headers": {
"Authorization": "Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"
}
}
}
}{
"mcpServers": {
"ai-crawler-access-check": {
"httpUrl": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"headers": {
"Authorization": "Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"
}
}
}
}extensions:
"ai-crawler-access-check":
enabled: true
name: "ai-crawler-access-check"
type: streamable_http
uri: "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check"
headers:
"Authorization": "Bearer ${AI_CRAWLER_ACCESS_CHECK_TOKEN}"
timeout: 300
Placeholders to fill from your own secret store: ${AI_CRAWLER_ACCESS_CHECK_TOKEN}. The registry never accepts secret values.
protocol.modernsupports 2026-07-28 (server/discover)protocol.statelessanswers without a sessionprotocol.transportstreamable HTTPprotocol.list_ttltools/list carries no ttlMs, so clients re-fetch itauth.prm, auth.as_metadata, auth.cimdno remote uses OAuthauth.secret_in_urlno templated URLtools.descriptions, tools.description_length, tools.schemas, tools.annotations, tools.directory_hints, tools.token_cost, tools.api_dumptool list unknown (behind auth, not probed, or empty)stability.changes, stability.rug_pulltool list unknowndeps.known_vulns, deps.mcp_sdk_version, deps.resolvableno npm or PyPI packageNo tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.
Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.
+website +reachable +uptime-100% +updated-4d-ago -no-repository
Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.
| remote | auth | reachable | uptime 30d | p50 | protocol | last ok |
|---|---|---|---|---|---|---|
https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check streamable-http | none | yes | 100% | 62 ms | 2026-07-28 | 2026-11-07 |
No tools observed. Tools appear once a credential-free probe can list them.
Schemas: list_tools.
And 34 more from this publisher.
{
"name": "io.github.kriton-dev/ai-crawler-access-check",
"_meta": {
"io.modelcontextprotocol.registry/publisher-provided": {
"apifyActor": "hermeskd/ai-crawler-access-check",
"storeTitle": "AI Crawler Checker — Robots.txt Policy for Six Bots",
"storeDescription": "AI crawler checker: declared robots.txt permissions for six fixed AI bots at a URL path, without fetching the page. $0.0015 per path; start extra; usage included; failed results uncharged."
}
},
"title": "AI Crawler Checker — Robots.txt Policy for Six Bots",
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"remotes": [
{
"url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check",
"type": "streamable-http",
"headers": [
{
"name": "Authorization",
"isSecret": true,
"description": "Optional: 'Bearer <your Apify API token>'. MCP clients that support OAuth can sign in to Apify instead. Runs are billed to your Apify account at the pay-per-event prices on the Actor's store page."
}
]
}
],
"version": "1.0.2",
"websiteUrl": "https://apify.com/hermeskd/ai-crawler-access-check",
"description": "Check declared robots.txt permissions for six AI crawlers at URL paths"
}