Measured impact: a holdout group for agent memory
A built-in holdout gives a slice of runs no memory, so Retrack shows the real lift in success rate, tokens and cost on your own traffic.
Updated
"Our memory layer improves your agent" is easy to claim and hard to check. Retrack checks it for you, on your own traffic, continuously.
A holdout group, built in
A small random slice of your runs, 10% by default, gets no memory. Everything else about those runs is identical, so the only difference between the two groups is Retrack. The split is deterministic per run, so it is stable and fair, and you can set it from 0 to 50% in your project's settings.
What you see
The dashboard's overview compares runs that received memory with held-out runs:
- Success rate and lift: how many more tasks succeed with memory, in percentage points.
- Tokens and cost per run: whether memory pays for itself.
- Recall hit rate and latency: how often a playbook was found, and how fast.
Retrack marks the comparison as having enough evidence once 20 held-out runs have a known outcome, so you never act on noise.
Honest numbers, including the bad ones
The holdout reports what happened, not what we hoped. In our own tests memory sometimes added a few percent of prompt tokens, because a playbook is extra context; fewer steps and retries paid for most of it. Your dashboard will show your numbers, whatever they are.
Also available by API
GET /v1/metrics?days=30 returns the same comparison for your own reporting. See Measuring impact.