Documentation menu

Regression tests for AI agents in CI

Turn your agent's playbooks and known failures into regression tests and run them with retrack test in GitHub Actions or any CI before you ship.

Updated

Every lesson your agent learned is also a test. Retrack turns active playbooks and recurring failures into regression cases, and retrack test runs them against your agent, so a prompt change, a new tool or a refactor cannot quietly undo what the agent learned.

Where the cases come from

SourceCaseAssertions
A playbook becomes activeThe task of its first source runThe agent completes, and calls the playbook's tools in order (other calls allowed in between; conditional steps are left out)
A failure signature happens twiceA task that hit itThat tool does not fail with that error again. These cases are critical

Cases update when a playbook gets a new version and retire with it. You can see them, with their last result, in the dashboard under Regression.

Write an entry point

retrack test calls a function you provide that runs one task and passes Retrack's callbacks to your agent:

# my_agent.py
def run_task(task: str, callbacks: list) -> str:
    result = graph.invoke(
        {"messages": [("user", task)]},
        config={"callbacks": callbacks},
    )
    return result["messages"][-1].content

Runs made by the test command are never stored as runs and never become memory; only the results are sent.

Run the tests

export RETRACK_API_KEY="rtk_live_..."
retrack test --entry my_agent:run_task --agent support-agent
OptionDefaultMeaning
--entryrequiredmodule:function that runs one task
--agentrequiredThe agent name you use with Retrack
--threshold0.9Minimum pass rate
--model-idnoneModel under test, recorded with the results

Exit codes: 0 passed; 1 the pass rate is below the threshold, or a critical case that passed last time now fails; 2 usage or connection error.

Run them in CI

Any CI that can run Python can run the tests. A GitHub Actions workflow:

# .github/workflows/retrack.yml
name: Agent regression tests
on: [pull_request]
jobs:
  retrack:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt retrack-ai
      - run: retrack test --entry my_agent:run_task --agent support-agent --threshold 0.9
        env:
          RETRACK_API_KEY: ${{ secrets.RETRACK_API_KEY }}

Add RETRACK_API_KEY as a repository secret. The workflow fails the pull request when the agent regresses.

What each assertion checks

  • completes: the entry point returned without raising.
  • tool_sequence: the expected tools were called in this order.
  • no_failure: a given tool did not fail with a given error class.
  • output_contains: the answer contains the expected text.

Assertion types the command does not know are skipped, not failed, so older connectors keep working as new types arrive.