# RobotsGate

> Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

- name: dev.workers.mike-tusa.robotsgate/robotsgate
- version: 0.3.1
- connection class: R0 (autonomous)
- trust: 81/100 flags: no-repository
- quality: 100/100 (new)
- owner: not claimed
- descriptor: https://api.protogrid.dev/v1/servers/dev.workers.mike-tusa.robotsgate%2Frobotsgate
- tools: https://api.protogrid.dev/v1/servers/dev.workers.mike-tusa.robotsgate%2Frobotsgate/tools

**An agent can connect right now, no human step.** Remote endpoint, no authentication, reachable on the last probe.

## Connection (mcpServers)

```json
{
  "mcpServers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp",
      "headers": {
        "Authorization": "Bearer ${ROBOTSGATE_TOKEN}"
      }
    }
  }
}
```

Placeholders to fill: `${ROBOTSGATE_TOKEN}`.

## Trust

- hygiene: 80
- liveness: 96
- freshness: 100
- provenance: 55
- drivers: +dns-namespace +website +reachable +uptime-91% +updated-3d-ago -no-repository
- Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed.

## Quality 100/100 (new)

### protocol: 100 (weight 25)

- pass `protocol.modern`: supports 2026-07-28 (server/discover)
- pass `protocol.stateless`: answers without a session
- pass `protocol.transport`: streamable HTTP
- pass `protocol.list_ttl`: tool list is cacheable (ttlMs 3600000)

### authorization: n/a (weight 15)

- n/a `auth.prm`, `auth.as_metadata`, `auth.cimd`: no remote uses OAuth
- n/a `auth.secret_in_url`: no templated URL

### tool hygiene: 100 (weight 30)

- pass `tools.descriptions`: every tool has a description
- pass `tools.description_length`: descriptions are concise
- pass `tools.schemas`: every tool has an object input schema
- pass `tools.annotations`: 4 of 4 tools declare readOnlyHint or destructiveHint
- pass `tools.directory_hints`: every tool declares readOnlyHint, destructiveHint, idempotentHint and openWorldHint
- pass `tools.token_cost`: about 1,204 tokens to load every tool
- pass `tools.api_dump`: tools are not a one-to-one API dump

### stability: n/a (weight 15)

- n/a `stability.changes`, `stability.rug_pull`: tool history recorded for 2 of the 7 days needed

### dependencies: n/a (weight 15)

- n/a `deps.known_vulns`, `deps.mcp_sdk_version`, `deps.resolvable`: no npm or PyPI package

- tools: 4, about 1,204 tokens to load them all
- tool set hash: 1704eb6c05d2a79e007f600024819e57
- badge: [![protogrid quality](https://protogrid.dev/badge/dev.workers.mike-tusa.robotsgate/robotsgate.svg)](https://protogrid.dev/servers/dev.workers.mike-tusa.robotsgate/robotsgate)
- Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed.

### Tool changes

_No tool definition changes recorded._

## Remotes

- https://robotsgate.mike-tusa.workers.dev/mcp (streamable-http, auth none, reachable true, uptime30d 0.90909094)

## Packages

_none_

## Tools (4)

- `check_site`: Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
- `generate_robots`: Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
- `list_crawlers`: Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
- `validate_robots`: Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.