# OpenMark

> Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.

- name: ai.openmark/openmark
- version: 1.0.0
- connection class: R1 (autonomous)
- trust: 80/100 flags: no-repository
- quality: 73/100 (new)
- owner: not claimed
- descriptor: https://api.protogrid.dev/v1/servers/ai.openmark%2Fopenmark
- tools: https://api.protogrid.dev/v1/servers/ai.openmark%2Fopenmark/tools

**An agent can connect on its own if it already holds `OPENMARK_TOKEN`.** Remote endpoint behind a static API key or header. The registry never sees the value; the agent fills the placeholder.

## Connection (mcpServers)

```json
{
  "mcpServers": {
    "openmark": {
      "type": "http",
      "url": "https://openmark.ai/mcp",
      "headers": {
        "Authorization": "Bearer ${OPENMARK_TOKEN}"
      }
    }
  }
}
```

Placeholders to fill: `${OPENMARK_TOKEN}`.

## Trust

- hygiene: 80
- liveness: 93
- freshness: 100
- provenance: 55
- drivers: +dns-namespace +website +reachable +uptime-83% +updated-2d-ago -no-repository
- Derived from observable signals (official registry feed and our own credential-free probes); no code audit performed.

## Quality 73/100 (new)

### protocol: 75 (weight 25)

- warn `protocol.modern`: newest supported version is 2025-06-18; 2026-07-28 not supported, older versions are deprecated until 2027-07-28
- n/a `protocol.stateless`: only applies to 2026-07-28 servers
- pass `protocol.transport`: streamable HTTP
- n/a `protocol.list_ttl`: only applies to 2026-07-28 servers

### authorization: n/a (weight 15)

- n/a `auth.prm`, `auth.as_metadata`, `auth.cimd`: no remote uses OAuth
- n/a `auth.secret_in_url`: no templated URL

### tool hygiene: 71 (weight 30)

- pass `tools.descriptions`: every tool has a description
- pass `tools.description_length`: descriptions are concise
- pass `tools.schemas`: every tool has an object input schema
- fail `tools.annotations`: no tool declares readOnlyHint or destructiveHint, so clients cannot tell safe calls from risky ones
- fail `tools.directory_hints`: no tool declares all four of readOnlyHint, destructiveHint, idempotentHint and openWorldHint; missing: prepare_attachments (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); list_models (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); get_account (readOnlyHint, destructiveHint, idempotentHint, openWorldHint); …
- pass `tools.token_cost`: about 1,897 tokens to load every tool
- pass `tools.api_dump`: tools are not a one-to-one API dump

### stability: n/a (weight 15)

- n/a `stability.changes`, `stability.rug_pull`: tool history recorded for 1 of the 7 days needed

### dependencies: n/a (weight 15)

- n/a `deps.known_vulns`, `deps.mcp_sdk_version`, `deps.resolvable`: no npm or PyPI package

- tools: 8, about 1,897 tokens to load them all
- tool set hash: c9b2a3245b0c1e0d5c300ed77b93ef7e
- tools last changed: 2026-10-19T14:32:25.526Z
- badge: [![protogrid quality](https://protogrid.dev/badge/ai.openmark/openmark.svg)](https://protogrid.dev/servers/ai.openmark/openmark)
- Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed.

### Tool changes

- 2026-10-19 `watch_task` changed (description, inputSchema)
  - before: Watch the task of a completed job: OpenMark re-tests it on new or cheaper models as they ship (no credits used) and emails the user only when one beats the best model of that job.
  - after: Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.
- 2026-10-19 `get_results` changed (description, inputSchema)
  - before: Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency) plus a recommendation for the chosen objective and quick picks for the others.
  - after: Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.
- 2026-10-19 `run_benchmark` changed (description, inputSchema)
  - before: Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id; poll get_results. Spends the user's credits.
  - after: Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).
- 2026-10-19 `estimate_cost` added
  - Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
- 2026-10-19 `prepare_attachments` changed (description)
  - before: Get an upload_uid and the exact command to upload images, PDFs or text documents for a task. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to run_benchmark.
  - after: Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.
- 2026-10-19 `list_models` changed (description)
  - before: Current OpenMark model ids with list prices ($ per 1M tokens). Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.
  - after: Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.

Full history: https://api.protogrid.dev/v1/servers/ai.openmark%2Fopenmark/changes

## Remotes

- https://openmark.ai/mcp (streamable-http, auth api_key ${OPENMARK_TOKEN}, reachable true, uptime30d 0.8333333)

## Packages

_none_

## Tools (8)

- `estimate_cost`: Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
- `get_account`: The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.
- `get_results`: Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.
- `get_task_format`: How to write an OpenMark task: the YAML structure, every scoring mode, and tips. Call this before writing a task.
- `list_models`: Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.
- `prepare_attachments`: Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.
- `run_benchmark`: Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).
- `watch_task`: Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.