Documentation menu

Model migration: test a new model first

Replay your agent's memory on a new LLM before a forced model upgrade with retrack migrate, and see per playbook what passes, breaks or needs relearning.

Updated

Model providers retire models on fixed schedules, and a newer model does not always handle your agent's tasks the same way. retrack migrate replays your agent's memory on the new model before you switch, and reports, playbook by playbook, what still works.

What it does

  1. Opens a migration report in Retrack for your agent and the target model.
  2. Runs every regression case (or a sample) through your agent on the new model, on your machine, with your own model keys. Retrack does not call the model.
  3. Sends the results and prints the report.

Playbooks that pass are marked as validated on the new model, so recall ranks them fully as soon as you switch.

Write an entry point that takes a model

Use the same entry point as regression tests, with a third argument for the model:

def run_task(task: str, callbacks: list, model_id: str | None = None) -> str:
    llm = ChatAnthropic(model=model_id or "claude-haiku-5-5")
    graph = build_graph(llm)
    result = graph.invoke({"messages": [("user", task)]}, config={"callbacks": callbacks})
    return result["messages"][-1].content

A two-argument entry point still runs, with a warning, but then the replay cannot actually switch models, and the report says so.

Run a migration

retrack migrate --to claude-sonnet-5 --from claude-haiku-5-5 \
  --entry my_agent:run_task --agent support-agent
OptionMeaning
--toThe model to migrate to (required)
--fromThe current model, for the record
--sample N, --seed SReplay only N cases, chosen with seed S
--entry, --agentAs for retrack test

Reading the report

ResultMeaning
passesThe playbook's case passed on the new model
needs_relearningThe agent finished, but an assertion failed: the procedure changed
breaksThe agent raised an error
fixedA known failure no longer happens on the new model
returnsA known failure still happens

The command exits with 1 when anything breaks or a known failure returns, so you can use it as a gate before switching models in CI. Reports are also in the dashboard under Migrations, and through GET /v1/migrations/{id}.

Migration results never change the regression cases' last results, which always describe your current model.