Search / com.atalaya-ai/ai-crawler-check
Atalaya AI crawler check
R0· no sign-inWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
v1.1.0 · active · website · descriptor JSON · versions
Search / com.atalaya-ai/ai-crawler-check
R0· no sign-inWhich AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
v1.1.0 · active · website · descriptor JSON · versions
{
"mcpServers": {
"ai-crawler-check": {
"type": "http",
"url": "https://atalaya-ai.com/mcp"
}
}
}claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp[mcp_servers.ai-crawler-check]
url = "https://atalaya-ai.com/mcp"
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"ai-crawler-check": {
"type": "remote",
"url": "https://atalaya-ai.com/mcp",
"enabled": true
}
}
}{
"mcpServers": {
"ai-crawler-check": {
"type": "http",
"url": "https://atalaya-ai.com/mcp"
}
},
"deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=ai-crawler-check&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vYXRhbGF5YS1haS5jb20vbWNwIn0%3D"
}{
"servers": {
"ai-crawler-check": {
"type": "http",
"url": "https://atalaya-ai.com/mcp"
}
}
}{
"mcpServers": {
"ai-crawler-check": {
"httpUrl": "https://atalaya-ai.com/mcp"
}
}
}extensions:
"ai-crawler-check":
enabled: true
name: "ai-crawler-check"
type: streamable_http
uri: "https://atalaya-ai.com/mcp"
timeout: 300
protocol.modernnewest supported version is 2025-11-25; 2026-07-28 not supported, older versions are deprecated until 2027-07-28protocol.statelessonly applies to 2026-07-28 serversprotocol.transportstreamable HTTPprotocol.list_ttlonly applies to 2026-07-28 serversauth.prm, auth.as_metadata, auth.cimdno remote uses OAuthauth.secret_in_urlno templated URLtools.descriptionsevery tool has a descriptiontools.description_lengthdescriptions are concisetools.schemasevery tool has an object input schematools.annotations2 of 2 tools declare readOnlyHint or destructiveHinttools.directory_hintsevery tool declares readOnlyHint, destructiveHint, idempotentHint and openWorldHinttools.token_costabout 350 tokens to load every tooltools.api_dumptools are not a one-to-one API dumpstability.changes, stability.rug_pulltool history recorded for 3 of the 7 days neededdeps.known_vulns, deps.mcp_sdk_version, deps.resolvableno npm or PyPI packageNo tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.
Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.
+dns-namespace +website +reachable +uptime-83% +updated-4d-ago -no-repository
Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.
| remote | auth | reachable | uptime 30d | p50 | protocol | last ok |
|---|---|---|---|---|---|---|
https://atalaya-ai.com/mcp streamable-http | none | yes | 83% | 263 ms | 2025-11-25 | 2026-11-07 |
check_ai_crawler_accessread-only | Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawlers that do not run JavaScript (flags client-rendered app shells, common in apps built with Lovable, Bolt, Base44 or Replit), title/meta description/H1, llms.txt, sitemap, and the platform or host setting to change (Cloudflare, Vercel, Netlify). Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers. |
list_ai_crawlersread-only | Lists the AI crawler user-agent tokens Atalaya checks, by vendor and purpose (training, search index, user-triggered fetch), so you can write a robots.txt that keeps AI search while opting out of training. |
Schemas: list_tools.
{
"name": "com.atalaya-ai/ai-crawler-check",
"icons": [
{
"src": "https://atalaya-ai.com/logo-512.png",
"sizes": [
"512x512"
],
"mimeType": "image/png"
},
{
"src": "https://atalaya-ai.com/favicon.svg",
"sizes": [
"any"
],
"mimeType": "image/svg+xml"
}
],
"title": "Atalaya AI crawler check",
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"remotes": [
{
"url": "https://atalaya-ai.com/mcp",
"type": "streamable-http"
}
],
"version": "1.1.0",
"websiteUrl": "https://atalaya-ai.com/mcp-server/",
"description": "Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why."
}