Search / dev.workers.pathwren.www/robots-policy-lint
protogridR0· no sign-in{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
v1.4.0 · active · website · descriptor JSON · versions
{
"mcpServers": {
"robots-policy-lint": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
}
}
}claude mcp add --transport http robots-policy-lint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots[mcp_servers.robots-policy-lint]
url = "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"robots-policy-lint": {
"type": "remote",
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots",
"enabled": true
}
}
}{
"mcpServers": {
"robots-policy-lint": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
}
},
"deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=robots-policy-lint&config=eyJ0eXBlIjoiaHR0cCIsInVybCI6Imh0dHBzOi8vd3d3LnBhdGh3cmVuLndvcmtlcnMuZGV2L2MvbWNwLXJlZ2lzdHJ5LW9mZmljaWFsL21jcC9yb2JvdHMifQ%3D%3D"
}{
"servers": {
"robots-policy-lint": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
}
}
}{
"mcpServers": {
"robots-policy-lint": {
"httpUrl": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
}
}
}extensions:
"robots-policy-lint":
enabled: true
name: "robots-policy-lint"
type: streamable_http
uri: "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"
timeout: 300
protocol.modernsupports 2026-07-28 (server/discover)protocol.statelessanswers without a sessionprotocol.transportstreamable HTTPprotocol.list_ttltool list is cacheable (ttlMs 3600000)auth.prm, auth.as_metadata, auth.cimdno remote uses OAuthauth.secret_in_urlno templated URLtools.descriptionsevery tool has a descriptiontools.description_length4 tools have descriptions over 1,024 characters, which crowds the contexttools.schemasevery tool has an object input schematools.annotations9 of 9 tools declare readOnlyHint or destructiveHinttools.directory_hintsevery tool declares readOnlyHint, destructiveHint, idempotentHint and openWorldHinttools.token_costabout 3,533 tokens to load every tooltools.api_dumptools are not a one-to-one API dumpstability.changesno tool changes in 30 daysstability.rug_pullno tool changed its meaning under the same namedeps.known_vulns, deps.mcp_sdk_version, deps.resolvableno npm or PyPI packageNo tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.
Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.
+dns-namespace +website +reachable +uptime-96% ~updated-43d-ago -no-repository
Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.
| remote | auth | reachable | uptime 30d | p50 | protocol | last ok |
|---|---|---|---|---|---|---|
https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots streamable-http | none | yes | 96% | 257 ms | 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05 | 2026-10-19 |
am_i_allowedread-only | {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"am_i_allowed","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and the answer is what this host's own robots.txt says to YOU — the group that matches your user-agent, the line that decided it, that group verbatim, and whether you are allowed each of three paths, evaluated by the same RFC 9309 engine as check_path_allowed. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers what this host's own robots.txt says to YOU — the group that matches your token, the line that decided it, and whether you are allowed the path you are on — its six siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev; whoami is answered by all seven and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"am_i_allowed","arguments":{}}} returns your matched group and the line it was matched on, plus an allowed/disallowed verdict for /, /data/agents.json and /mcp/robots. |
audit_ai_accessread-only | FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. Example: path='/' with your robots_txt pasted in — the verdict is per crawler, at that path. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/ai-access?robots_txt=<urlencoded>&s=client-dossiers |
check_path_allowedread-only | FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=<urlencoded>&ua=GPTBot&path=/blog&s=client-dossiers |
diff_robots_txtread-only | FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. Example: before='User-agent: *\nAllow: /\n', after=your edited file, path='/'. |
exampleread-only | {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — lint_robots_txt, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — lint_robots_txt on three groups lifted verbatim from this host's own published /robots/block-ai-training.txt and /robots.txt — our data, not yours, and nothing is fetched — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs lint_robots_txt with {"robots_txt":"# lifted from https://www.pathwren.workers.dev/robots/block-ai-training.txt and returns its real answer. |
lint_robots_txtread-only | FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. Example: robots_txt='User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n' — paste the whole file, it is never fetched for you. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=<urlencoded>&s=client-dossiers — or POST the file as the raw body to the same URL. |
merge_policyread-only | FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. Example: stance='block-ai-training', robots_txt='User-agent: *\nAllow: /\n'. |
no_arguments_lint_this_hosts_robots_txtread-only | TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} to https://www.pathwren.workers.dev/mcp/robots — the answer is the full RFC 9309 lint of this host's OWN robots.txt and ai.txt, run for real on the bytes it serves — every finding with its severity, its line number and the fix, plus an allowed/disallowed verdict per crawler token per path with the exact rule and line that produced it, and the check that the two files still agree with each other. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. The other zero-argument call on this server is am_i_allowed, same empty arguments, which answers what this host's own robots.txt says to YOU — the group that matches your user-agent, the line that decided it, that group verbatim, and whether you are allowed each of three paths, evaluated by the same RFC 9309 engine as check_path_allowed. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} returns both files parsed and linted, their findings with line numbers and fixes, a verdict table naming the rule that decided each row, the roll-up of which indexed AI crawlers this host's file actually stops, and the agreement check between robots.txt and ai.txt. |
whoamiread-only | {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/robots and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: which group in this host's own robots.txt matches YOUR user-agent, the line that decided it, and whether you are allowed the paths you are on, evaluated by the same RFC 9309 engine as check_path_allowed. Takes no arguments. Safe to call. Deterministic. Touches no third party. Nothing is fetched to build it: every fact comes from the headers on your own request or from a file this host already publishes. Only this server answers the last part — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev. Example: arguments={} returns your user-agent, your address, the class we book you as, whether we have seen you here before, and what this host's own robots.txt says to YOU — the group that matches your token, the line that decided it, and whether you are allowed the path you are on. |
Schemas: list_tools.
{
"name": "dev.workers.pathwren.www/robots-policy-lint",
"_meta": {
"io.modelcontextprotocol.registry/publisher-provided": {
"auth": "none",
"docs": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp.html",
"tools": [
{
"name": "whoami",
"search": "who is calling: user-agent, address, our verdict - no arguments"
},
{
"name": "lint_robots_txt",
"search": "lint a robots.txt you paste"
},
{
"name": "check_path_allowed",
"search": "test whether a crawler may fetch a path (RFC 9309)"
},
{
"name": "audit_ai_access",
"search": "see which AI crawlers your robots.txt actually stops"
},
{
"name": "diff_robots_txt",
"search": "diff two robots.txt files by effect, not by text"
},
{
"name": "merge_policy",
"search": "add a ready-made block-ai-training stance to your file"
},
{
"name": "example",
"search": "run the worked example — no arguments"
}
],
"example": {
"curl": "curl -s -X POST https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}}'",
"note": "whoami takes no arguments at all and answers about the request you just sent - user-agent, address, what our classifier makes of it. Every tool here accepts {} and answers with its own documented worked example — nothing of yours is read, nothing is fetched from a third party. Send the request below verbatim.",
"request": {
"id": 1,
"method": "tools/call",
"params": {
"name": "whoami",
"arguments": {}
},
"jsonrpc": "2.0"
},
"endpoint": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots",
"transport": "streamable-http",
"verified_live_at": "2026-09-02T08:49:09Z"
},
"pricing": "free",
"keywords": [
"robots.txt",
"robots.txt tester",
"robots.txt checker",
"rfc 9309",
"block ai training",
"disallow",
"crawl-delay",
"seo",
"opt out of ai",
"markdown",
"markdown docs",
"free",
"no api key",
"no auth"
],
"markdown": {
"note": "Everything this server answers from is also readable without JSON-RPC and without an MCP client: every page of the host is a plain markdown file at the same address with .md — also .mdx, <page>.html.md, <page>.html.mdx, or send Accept: text/markdown to the .html page. No key, no JavaScript, CC0. The registry caps `description` at 100 characters, which is why this sentence lives here.",
"index": "https://www.pathwren.workers.dev/c/mcp-registry-official/sitemap.md",
"example": "curl -s https://www.pathwren.workers.dev/c/mcp-registry-official/mcp-robots.md",
"canonical": "the HTML page; each mirror carries Link: rel=\"canonical\" back to it"
},
"publisher": "Pathwren — an independent, non-commercial automated project",
"dataLicense": "CC0-1.0",
"authentication": "none — no key, no account, no OAuth, no rate limit"
}
},
"icons": [
{
"src": "https://www.pathwren.workers.dev/c/mcp-registry-official/icon.png",
"sizes": [
"256x256"
],
"mimeType": "image/png"
}
],
"title": "Robots.txt Policy Lint",
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
"remotes": [
{
"url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots",
"type": "streamable-http"
}
],
"version": "1.4.0",
"websiteUrl": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp-robots.html",
"description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"whoami\",\"arguments\":{}}} free no key"
}