# Octocrawl

> Web pages as clean Markdown and JSON for AI agents, in fewer tokens, with an Evidence Record.

- name: io.github.77777R7/octocrawl
- version: 0.7.0
- connection class: R0 (autonomous)
- trust: 97/100
- quality: 83/100 (good)
- owner: not claimed
- descriptor: https://api.protogrid.dev/v1/servers/io.github.77777R7%2Foctocrawl
- tools: https://api.protogrid.dev/v1/servers/io.github.77777R7%2Foctocrawl/tools

**An agent can connect right now, no human step.** Remote endpoint, no authentication, reachable on the last probe.

## Connection (mcpServers)

```json
{
  "mcpServers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
```


## Trust

- hygiene: 100
- liveness: 100
- freshness: 100
- provenance: 90
- drivers: +repository +namespace-matches-repo +website +reachable +uptime-100% +updated-3d-ago
- Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed.

## Quality 83/100 (good)

### protocol: 88 (weight 25)

- pass `protocol.modern`: supports 2026-07-28 (server/discover)
- pass `protocol.stateless`: answers without a session
- pass `protocol.transport`: streamable HTTP
- warn `protocol.list_ttl`: tools/list declares ttlMs 0, so clients re-fetch it on every use

### authorization: 100 (weight 15)

- pass `auth.prm`: Protected Resource Metadata (RFC 9728) published
- pass `auth.as_metadata`: authorization server metadata reachable
- pass `auth.cimd`: Client ID Metadata Documents supported
- n/a `auth.secret_in_url`: no templated URL

### tool hygiene: 100 (weight 30)

- pass `tools.descriptions`: every tool has a description
- pass `tools.description_length`: descriptions are concise
- pass `tools.schemas`: every tool has an object input schema
- pass `tools.annotations`: 2 of 2 tools declare readOnlyHint or destructiveHint
- pass `tools.directory_hints`: every tool declares readOnlyHint, destructiveHint, idempotentHint and openWorldHint
- pass `tools.token_cost`: about 2,466 tokens to load every tool
- pass `tools.api_dump`: tools are not a one-to-one API dump

### stability: 25 (weight 15)

- warn `stability.changes`: 13 tool changes in 30 days
- fail `stability.rug_pull`: 1 tools kept their name but changed most of their description in 30 days; review before trusting them

### dependencies: n/a (weight 15)

- n/a `deps.known_vulns`, `deps.mcp_sdk_version`, `deps.resolvable`: deps.dev does not know @octocrawl/mcp@0.7.0

- tools: 2, about 2,466 tokens to load them all
- tool set hash: a5d1aa20ae35282ede7eddf586fa1984
- tools last changed: 2026-10-19T08:08:39.405Z
- badge: [![protogrid quality](https://protogrid.dev/badge/io.github.77777R7/octocrawl.svg)](https://protogrid.dev/servers/io.github.77777R7/octocrawl)
- Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed.

### Tool changes

- 2026-10-19 `scrape` changed (inputSchema)
- 2026-10-18 `map` changed (inputSchema)
- 2026-10-17 `scrape` changed (outputSchema)
- 2026-10-17 `map` changed (description, inputSchema, outputSchema)
  - before: List a site's URLs without fetching each page: the start URL, the links on its page (read on the http lane alone; no browser) and the entries of the sitemaps the site declares (robots.txt Sitemap: lines, else /sitemap.xml), inside one deadline. Every URL is in the crawl's scope (the start host and its www twin, the start URL's path subtree, assets left out, similar URLs folded) and allowed by its host's robots.txt unless ignoreRobotsTxt is set; what was left out is counted. A title is never fetched: the start page's own, an anchor's text or a sitemap's news title. At the deadline the answer is what was found, status partial (failed when nothing), stoppedBy timeout. Compact by default ({ id, status, stoppedBy, links: [{ url, title?, description?, robots? }], warning?, agentHints?, counts }; robots only on a link robots.txt keeps out, under ignoreRobotsTxt); debug=true returns the full map with each link's evidence (via, sitemapFile, lastmod, robots), the sources read and the refusals. One page body is read at most: a site without a sitemap maps only its start page's links; crawl reads further pages.
  - after: List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.
- 2026-10-17 `scrape` changed (description, inputSchema) **meaning changed**
  - before: Fetch one URL through the Octocrawl coverage ladder. Compact by default; set debug=true for the full audit. The result's warnings name what its content cannot vouch for: robots_overridden (robots.txt disallows the URL; a local server fetched it because you named it), or client_rendered_suspected when the HTTP page looks like a shell its scripts fill in and the browser rung found nothing better. Its agentHints, when present, say what to change next time (a login wall, a robots.txt rule, a gate, a wait).
  - after: Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
- 2026-10-17 `map` changed (title, inputSchema, annotations)
- 2026-10-17 `scrape` changed (title, inputSchema, annotations)
- 2026-10-16 `scrape` changed (description)
  - before: Fetch one URL through the Octocrawl coverage ladder. Compact by default; set debug=true for the full audit. The result's warnings name what its content cannot vouch for: robots_overridden (robots.txt disallows the URL; a local server fetched it because you named it), or client_rendered_suspected when the HTTP page looks like a shell its scripts fill in and the browser rung found nothing better. Its agentHints, when present, say what to change next time (a login wall, a robots.txt rule, a gate, a wait). metadata.scrapeId names the call's record for get_scrape.
  - after: Fetch one URL through the Octocrawl coverage ladder. Compact by default; set debug=true for the full audit. The result's warnings name what its content cannot vouch for: robots_overridden (robots.txt disallows the URL; a local server fetched it because you named it), or client_rendered_suspected when the HTTP page looks like a shell its scripts fill in and the browser rung found nothing better. Its agentHints, when present, say what to change next time (a login wall, a robots.txt rule, a gate, a wait).
- 2026-10-15 `scrape` changed (inputSchema)
- 2026-10-12 `scrape` changed (inputSchema)

Full history: https://api.protogrid.dev/v1/servers/io.github.77777R7%2Foctocrawl/changes

## Dependencies

- npm @octocrawl/mcp@0.7.0: unknown

Dependency graphs from deps.dev; advisories from OSV.dev, including the GitHub Advisory Database and the PyPI Advisory Database; all CC BY 4.0.

## Remotes

- https://mcp.octocrawl.dev/mcp (streamable-http, auth none, reachable true, uptime30d 1)
- https://mcp.octocrawl.dev/mcp-oauth (streamable-http, auth oauth2, reachable false, uptime30d 0.9375)

## Packages

- npm @octocrawl/mcp@0.7.0 secrets: W2L_API_TOKEN

## Tools (2)

- `map`: List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.
- `scrape`: Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.