Model migration testing for LLM agents
Replay your agent's memory on a new model before a forced upgrade and get a per-playbook report of what passes, breaks or needs relearning.
Updated
Model upgrades are not optional. Providers deprecate models on fixed schedules, sometimes with only weeks between the notice and retirement, and a newer model does not always handle your agent's specific tasks the same way. Teams find out what broke after the switch.
Retrack makes the switch a checklist.
Replay your agent's memory on the new model
retrack migrate --to claude-sonnet-5 --from claude-haiku-5-5 \
--entry my_agent:run_task --agent support-agentThe command replays every regression case, the ones generated from your playbooks and known failures, through your own agent on the new model. It runs on your side with your model keys, so Retrack never calls the model and your data never leaves your setup for this.
A per-playbook report
| Result | Meaning |
|---|---|
| passes | This procedure still works on the new model |
| needs relearning | The agent finished, but did it differently |
| breaks | The agent raised an error |
| fixed | A known failure no longer happens |
| returns | A known failure still happens |
Playbooks that pass are marked as validated on the new model, so recall trusts them fully the day you switch. The rest is a short list of exactly what to fix first.
Gate the switch in CI
retrack migrate exits with an error when anything breaks or a known failure returns, so a model change cannot ship until its report is clean.
Why it matters
Memory is matched to the model and tool versions it was validated on. Without a migration check, switching models silently lowers trust in every playbook; with one, you know in advance which procedures carry over and which need new runs.
Set it up with the model migration guide.