Connect

8 clients · secrets stay placeholders

JSON (mcpServers)Claude CodeCodexOpenCodeCursorVS CodeGemini CLIGoose

{
  "mcpServers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

[mcp_servers.crawl-census]
url = "https://crawlcensus.com/mcp"

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "crawl-census": {
      "type": "remote",
      "url": "https://crawlcensus.com/mcp",
      "enabled": true
    }
  }
}

{
  "mcpServers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  },
  "deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=crawl-census&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vY3Jhd2xjZW5zdXMuY29tL21jcCJ9"
}

{
  "servers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

{
  "mcpServers": {
    "crawl-census": {
      "httpUrl": "https://crawlcensus.com/mcp"
    }
  }
}

extensions:
  "crawl-census":
    enabled: true
    name: "crawl-census"
    type: streamable_http
    uri: "https://crawlcensus.com/mcp"
    timeout: 300

Quality 79 / 100good

protocol75 · weight 25

warn
protocol.modernnewest supported version is 2025-06-18; 2026-07-28 not supported, older versions are deprecated until 2027-07-28
n/a
protocol.statelessonly applies to 2026-07-28 servers
pass
protocol.transportstreamable HTTP
n/a
protocol.list_ttlonly applies to 2026-07-28 servers

authorizationn/a · weight 15

n/a
auth.prm, auth.as_metadata, auth.cimdno remote uses OAuth
n/a
auth.secret_in_urlno templated URL

tool hygiene71 · weight 30

pass
tools.descriptionsevery tool has a description
pass
tools.description_lengthdescriptions are concise
pass
tools.schemasevery tool has an object input schema
fail
tools.annotationsno tool declares readOnlyHint or destructiveHint, so clients cannot tell safe calls from risky ones
fail
tools.directory_hintsno tool declares all four of readOnlyHint, destructiveHint, idempotentHint and openWorldHint; missing: agent_profile (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); census_facts (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); census_stats (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); …
pass
tools.token_costabout 1,024 tokens to load every tool
pass
tools.api_dumptools are not a one-to-one API dump

stability100 · weight 15

pass
stability.changesno tool changes in 30 days
pass
stability.rug_pullno tool changed its meaning under the same name

dependenciesn/a · weight 15

n/a
deps.known_vulns, deps.mcp_sdk_version, deps.resolvableno npm or PyPI package

7 tools, about 1,024 tokens to load them all · tool set ff86810bd9ba · checked 2026-10-20 13:49 UTC

Tool changes

No tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.

Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.

Trust 93 / 100

provenance90

liveness98

freshness85

hygiene100

+repository +namespace-matches-repo +website +reachable +uptime-95% ~updated-58d-ago

Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.

Endpoints

remote	auth	reachable	uptime 30d	p50	protocol	last ok
`https://crawlcensus.com/mcp` streamable-http	none	yes	95%	263 ms	2025-06-18	2026-10-20

Tools (7)

`agent_profile`	What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
`census_facts`	Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
`census_stats`	Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
`crawl_preflight`	Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
`scan_site`	Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
`site_report`	Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
`submit_domains`	Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

Schemas: list_tools.

Similar MCP servers

Official server.json

{
  "name": "io.github.taylorsmithgg/crawl-census",
  "title": "Crawl Census",
  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
  "remotes": [
    {
      "url": "https://crawlcensus.com/mcp",
      "type": "streamable-http"
    }
  ],
  "version": "1.0.0",
  "repository": {
    "url": "https://github.com/taylorsmithgg/crawl-census-client",
    "source": "github"
  },
  "websiteUrl": "https://crawlcensus.com/for-crawlers",
  "description": "Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?"
}