What is a holdout group?
A holdout group is a random share of traffic left without a feature, such as agent memory, so its real effect can be measured against the rest.
Updated
A holdout group is a random share of traffic deliberately left without a feature, such as an AI agent's memory, so its effect can be measured by comparing the two groups.
Why AI agents need one
Agent performance drifts with traffic, models and prompts. Comparing "before" and "after" a change mixes its effect with everything else that changed. A holdout runs at the same time, on the same kind of traffic, so the difference between groups is the feature.
Designing a holdout
- Random and stable: assign by a hash of the run or user id, so the split is fair and repeatable.
- Small but sufficient: 5 to 10% is common; enough held-out outcomes are needed before the comparison means anything.
- Same everything else: held-out runs must differ only in the feature.
- Known outcomes: compare success rates using runs whose outcome was verified.
What to compare
Success rate (the lift in percentage points), tokens and cost per run, latency, and for memory, how often it was recalled.
In Retrack
Every project holds out 10% of runs by default and reports the lift on the dashboard once 20 held-out runs have a known outcome. See Measured, not estimated.