protogrid

Search / io.github.mark1russell7/polite-crawler

polite-crawler

L0· runs locally

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

v0.1.4 · active · repository · website · descriptor JSON · versions

Runs on your machine: something must execute the package.Only local packages are published (npm, PyPI, OCI, …). The registry never executes them, so tools are unknown until you run it.
Quality–new
Trust96of 100
Packages1
Tools seen0

Connect

8 clients · secrets stay placeholders
{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "[email protected]",
        "mcp"
      ]
    }
  }
}
claude mcp add polite-crawler -- npx -y [email protected] mcp
[mcp_servers.polite-crawler]
command = "npx"
args = ["-y", "[email protected]", "mcp"]
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "polite-crawler": {
      "type": "local",
      "command": [
        "npx",
        "-y",
        "[email protected]",
        "mcp"
      ],
      "enabled": true
    }
  }
}
{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "[email protected]",
        "mcp"
      ]
    }
  },
  "deeplink": "cursor://anysphere.cursor-deeplink/mcp/install?name=polite-crawler&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInBvbGl0ZS1jcmF3bGVyQDAuMS40IiwibWNwIl19"
}
{
  "servers": {
    "polite-crawler": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "[email protected]",
        "mcp"
      ]
    }
  }
}
{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "[email protected]",
        "mcp"
      ]
    }
  }
}
extensions:
  "polite-crawler":
    enabled: true
    name: "polite-crawler"
    type: stdio
    cmd: "npx"
    args:
      - "-y"
      - "[email protected]"
      - "mcp"
    timeout: 300

This is a local package: the command runs on your machine.

Quality – / 100new

Not scored: fewer than three checks apply, usually because no remote answered a credential-free probe or the server is a local package.

protocoln/a · weight 25
  • n/a
    protocol.modern, protocol.stateless, protocol.transport, protocol.list_ttlno remote answered a protocol handshake (not probed yet, unreachable, or local-only)
authorizationn/a · weight 15
  • n/a
    auth.prm, auth.as_metadata, auth.cimdno remote uses OAuth
  • n/a
    auth.secret_in_urlno templated URL
tool hygienen/a · weight 30
  • n/a
    tools.descriptions, tools.description_length, tools.schemas, tools.annotations, tools.directory_hints, tools.token_cost, tools.api_dumptool list unknown (behind auth, not probed, or empty)
stabilityn/a · weight 15
  • n/a
    stability.changes, stability.rug_pulltool list unknown
dependenciesn/a · weight 15
  • n/a
    deps.known_vulns, deps.mcp_sdk_version, deps.resolvabledeps.dev does not know [email protected]

Tool list unknown · checked 2026-10-19 17:25 UTC

Tool changes

No tool definition changes recorded. A server's first observation is its baseline; later probes record what changes.

Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.

Trust 96 / 100

provenance90
livenessn/a
freshness100
hygiene100

+repository +namespace-matches-repo +website ~liveness-unmeasured +updated-1d-ago

Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed. How it is computed.

Dependencies

[email protected] npmthis version is not known to the dependency resolver

Dependency graphs from deps.dev; advisories from OSV.dev, including the GitHub Advisory Database and the PyPI Advisory Database; all CC BY 4.0. The graph is what a clean install of this version resolves today; nothing was installed or run. How it is checked.

Endpoints

packageregistryversionruntimesecrets
polite-crawlernpm0.1.4npx

Tools (0)

No tools observed. Local packages are never executed by the registry.

Schemas: list_tools.

Similar MCP servers

Official server.json
{
  "name": "io.github.mark1russell7/polite-crawler",
  "title": "polite-crawler",
  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
  "version": "0.1.4",
  "packages": [
    {
      "version": "0.1.4",
      "transport": {
        "type": "stdio"
      },
      "identifier": "polite-crawler",
      "runtimeHint": "npx",
      "registryType": "npm",
      "registryBaseUrl": "https://registry.npmjs.org",
      "packageArguments": [
        {
          "type": "positional",
          "value": "mcp"
        }
      ],
      "environmentVariables": [
        {
          "name": "POLITE_CRAWLER_DIR",
          "format": "filepath",
          "description": "The folder of the store. The default is ~/.polite-crawler."
        },
        {
          "name": "POLITE_CRAWLER_CONTACT",
          "description": "An email address or URL for the From header of each request."
        },
        {
          "name": "POLITE_CRAWLER_ALLOW_PRIVATE",
          "format": "boolean",
          "description": "Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them."
        },
        {
          "name": "POLITE_CRAWLER_ALLOW_USER_PROVIDED",
          "format": "boolean",
          "description": "Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm."
        },
        {
          "name": "POLITE_CRAWLER_SIGN_KEY",
          "format": "filepath",
          "description": "The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT."
        },
        {
          "name": "POLITE_CRAWLER_SIGNATURE_AGENT",
          "description": "Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set."
        }
      ]
    }
  ],
  "repository": {
    "url": "https://github.com/mark1russell7/crawler",
    "source": "github",
    "subfolder": "packages/polite-crawler"
  },
  "websiteUrl": "https://mark1russell7.github.io/crawler/",
  "description": "A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes."
}