Documentation menu

Measuring impact with a holdout group

Retrack holds out a slice of runs with no memory and compares success rate, tokens and cost, so the effect of memory is measured, not estimated.

Updated

Most memory products ask you to believe they help. Retrack measures it: a small random slice of your runs gets no memory, and the dashboard compares them with the rest.

How the holdout works

  • By default 10% of runs are held out; change it (0 to 50%) in your project's Settings. Which runs is decided deterministically from the project and the run id, so the split is stable and fair.
  • For a held-out run, recall returns empty memory and memory.holdout is True. Your agent runs exactly as it would without Retrack.
  • Everything else is the same, so the only difference between the two groups is memory.

What is compared

The dashboard's overview (and GET /v1/metrics) compares runs that received memory with held-out runs:

MetricMeaning
Success rateShare of runs with a known outcome that succeeded
LiftMemory success rate minus holdout success rate, in percentage points
Tokens and costAverage per run, and the relative change with memory
Recall hit rateShare of recalls that returned at least one playbook
Recall latencyMedian and 95th percentile

Success rates use runs with a known outcome only, so send outcomes where you can.

When to trust the numbers

Small samples are noisy. Retrack marks the comparison as having enough evidence once at least 20 held-out runs have a known outcome. Before that, treat the numbers as early signs.

Reading the API

curl -H "Authorization: Bearer $RETRACK_API_KEY" \
  "https://api.retrack-ai.online/v1/metrics?days=30"

The response includes memory and holdout blocks (runs, success rate, average tokens and cost), lift_points, token_change, cost_change, enough_evidence and recall statistics. See the HTTP API.