Ai Model Experiments

Quality – / 100not scored

Not scored: fewer than three checks apply, usually because no remote answered a credential-free probe or the server is a local package.

Tool list unknown · tools last changed 2026-10-10 · checked 2026-10-20 16:30 UTC

Tool changes

• 2026-10-10experiment_topupremoved

  How to add prepaid credits for experiments (and your current balance). Payment is x402 — USDC on Base, paid in-band by any wallet-equipped agent: POST https://gateway.pipeworx.io/credits/topup?amount_usd=10 responds HTTP 402 with payment requirements; retry with PAYMENT-SIGNATURE to settle and the credits land instantly. Example: experiment_topup({})

• 2026-10-10experiment_cancelremoved

  Cancel a running experiment: pending cells are skipped (not billed); in-flight cells finish and bill. Example: experiment_cancel({ experiment_id: "..." })

• 2026-10-10experiment_listremoved

  List your experiments, newest first, with status and spend. Example: experiment_list({ limit: 10 })

• 2026-10-10experiment_statusremoved

  Progress of an experiment: cell counts by state (pending/running/ok/error/skipped), spend so far vs cap, and whether it is complete. Poll this after experiment_create (every few seconds). Example: experiment_status({ experiment_id: "..." })

• 2026-10-10experiment_createremoved

  Model Lab: run one prompt across many AI models at once and compare their outputs, cost, and latency side by side. Create and start an experiment: run each prompt against each model (× reps), collecting output, tokens, latency, and billed cost per cell. PREPAID: requires Pipeworx credit balance ≥ max_spend_usd (top up via experiment_topup); bills actual provider cost × 1.5 with a $0.10 minimum per experiment. ASYNC: returns experiment_id immediately — execution starts within ~1 minute; poll experiment_status until complete, then call experiment_results. Do NOT wait synchronously. Set summary:false to skip the AI-written model-comparison summary. Example: experiment_create({ name: "tone test", prompts: ["Rewrite formally: ..."], models: ["anthropic/claude-haiku-4.5", "openai/gpt-5-mini"], reps: 2, max_spend_usd: 2 })

• 2026-10-10experiment_estimateremoved

  Dry-run cost estimate for an experiment BEFORE creating it — cell count and estimated billed cost range (at our 1.5× pricing) for prompts × models × reps. Free to call, no side effects, does not need credit balance. Same spec shape as experiment_create. Example: experiment_estimate({ prompts: ["Summarize: ..."], models: ["anthropic/claude-sonnet-4.5", "openai/gpt-5"], reps: 2 })

• 2026-10-10resolution_diffremoved

  Field-by-field diff of TWO markets' settlement clauses (one from each of `a` and `b`; either can be Polymarket or Kalshi) — runs resolution_audit on both sides and compares source, settle time, precision, and evidence standard. Returns `equivalent`: "true" only when both sides parsed with enough confidence to compare AND no field conflicts; "false" when a specific conflict was found (differing_fields names which — e.g. ["source","settle_time"] for a Polymarket Bitcoin market settling on Binance's 1-minute candle at noon ET versus a Kalshi KXBTCD market settling on CF Benchmarks' BRTI 60-second average at 5pm EDT — SAME asset, DIFFERENT contract); "unclear" when one or both sides could not be confidently parsed (an absence of evidence is not evidence of equivalence — read raw_clause yourself in that case). Only flags a field as differing when BOTH sides gave a SPECIFIC comparable answer — a named source (e.g. "Associated Press, Fox News, NBC") against a generic one (e.g. Kalshi's "consensus of media organizations") is treated as the same evidence standard, not a conflict, since that is standard election-market boilerplate on both venues. Use this before sizing a polymarket_kalshi_spread pair as a real cross-venue arb, or standalone to sanity-check any two markets you suspect settle on different things.

• 2026-10-10resolution_auditremoved

  Extract the settlement clause of a single Polymarket or Kalshi market: who publishes the settling number (source), the clock time + timezone it is taken at, the precision of the computation (e.g. "1-minute candle close" vs "60-second trailing average" vs "election outcome"), the evidence standard (official_source | consensus_reporting | any_credible_report | unspecified), and void_handling (cancellation/postponement settlement — reused verbatim from bet_research's cancellation_rule detector, not re-derived). Parses Polymarket's `description` field (fetched via polymarket_market) or Kalshi's `rules_primary` + `rules_secondary` fields (fetched via kalshi_market) with regex + a small vocabulary — no LLM pass, so an unusual clause reports confidence:"low" rather than a guess. Pass `market` as a Polymarket slug/URL or a Kalshi market ticker (e.g. "KXBTCD-26SEP1317-T66999.99"); a Kalshi EVENT ticker (e.g. "KXBTCD-26SEP1317") also works — it picks one representative market under that event, since the settlement mechanism is normally shared across all strikes/legs in one event. Use this before treating a polymarket_kalshi_spread row as a real arbitrage: two ladders that look alike can settle on different sources, at different times, with different precision — this tool is how you check. Pair with resolution_diff to compare two markets directly. KNOWN GAP: idiosyncratic phrasing that doesn't match the vocabulary returns confidence:"low" and evidence_standard:"unspecified" rather than an LLM-guessed answer.

• 2026-10-10kalshi_weather_edgeremoved

  Prices Kalshi daily high-temperature markets against the NWS forecast for the market's OWN settlement station, and measures whether that forecast actually beats the market. Two modes. LIVE (default): returns the full strike ladder for one city and settlement date with market_prob (mid), forecast_prob, and edge_pp per strike, plus the settlement clause verbatim. BACKTEST (`backtest_days: N`): scores an archived gridded forecast against the market on settled days and returns brier_market vs brier_forecast with a plain-English `verdict`, so the edge is MEASURED rather than asserted. READ THE WARNINGS — they are not boilerplate. (1) These markets DO NOT settle on the NWS. They settle on The Weather Company (weather.com) at a Kalshi station code such as CLINYC, which the response quotes verbatim; so part of every edge_pp is NWS-vs-Weather-Company disagreement about the same day at the same station, which is not mispricing and not tradeable. `settlement_vs_forecast_basis_f` from backtest mode is that part as a number. (2) The station is DERIVED from the settlement clause, never from the city name: Chicago settles at MIDWAY and New York at CENTRAL PARK, so a city-centre forecast would misprice a whole ladder. A station that cannot be resolved yields rows with no forecast and a reason, never a guessed coordinate. (3) forecast_prob assumes a normal distribution around the NWS high whose width is ASSUMED, not fitted (stated in `distribution_assumption`) — run backtest mode to see whether it is calibrated. (4) edge_pp is gross: no Kalshi fees, no bid-ask. MEASURED RESULT, AND IT IS NOT THE FLATTERING ONE: on the first backtest (KXHIGHNY, 13 settled days to 2026-09-11, 58 market observations) the MARKET beat the forecast — Brier 0.1008 for the market against 0.1594 for the archived gridded forecast, lower being better. So on that sample there is NO forecast edge to sell, and a large edge_pp is more likely to be the model disagreeing with a better-informed market than an opportunity. The measured settlement-vs-forecast basis was 1.7F mean absolute over 8 pinnable days, slightly warm-biased, which is a big share of a typical edge_pp on a 2-degree bracket. Re-run backtest_days before believing any edge; if a later sample reverses this, the numbers say so. NWS is US-only, so the ~30 international Kalshi weather series (London, Paris, Tokyo) return market prices with forecast_unavailable rather than a forecast. Precipitation series are listed but not yet priced. Cities: nyc, chicago, los angeles, miami, austin, houston, denver, philadelphia — or pass `series_ticker` for any other (e.g. "KXHIGHTBOS").

• 2026-10-10polymarket_edge_trackerremoved

  Edge persistence and decay telemetry built from daily polymarket_edges snapshots. Answers "how long has this edge existed and is it shrinking?" — a fresh wide edge and a 3-week-old wide edge are different trades (the latter is wide for a reason nobody is willing to take). Args: days (lookback, default 14, max 30), window (snapshot family, default "1wk"). RESPONSE: tracked[] = every opportunity in the LATEST snapshot with its full edge_pp_net time-series across prior snapshots, first_seen, trend (new | widening | stable | decaying) and decay_pp_per_day (both computed on |edge_pp_net| — the value itself is signed by trade direction, negative = SELL YES); expired[] = opportunities that appeared in earlier snapshots but are GONE from the latest (closed, resolved, or arbed away) with their lifespan_days — the median lifespan is your competition clock; snapshot_dates[] = which days actually have data (snapshots are written when polymarket_edges runs on a cache-miss, so gaps mean nobody scanned that day). LIMITS: history depth is bounded by the 60-day snapshot TTL and starts from when snapshotting was enabled; decay numbers come from daily closes of edge_pp_net (net of default slippage AND Polymarket's own taker fee — see polymarket_edges), not intraday.

Showing the latest 10. Full history: list_changes · trend: quality JSON.

Checks run on what our credential-free, read-only probes observe; tools are never called and no code audit is performed. How it is computed.