Verifying runs: outcomes and feedback
How Retrack decides whether an agent run worked: tests, business events and user feedback, an AI judge as fallback, and why hard signals matter.
Updated
Retrack learns only from runs it can verify. A run that looked fine but was actually wrong must never become a playbook, so outcomes come from the strongest evidence you can give.
Sending an outcome
Call retrack.feedback whenever you learn how a run went, during the run or long after:
retrack.feedback(outcome="success", source="test") # the latest run
retrack.feedback(outcome="failure", source="event", run_id=run_id)source | Use it for | Confidence |
|---|---|---|
test | An assertion or test in your code | 1.0 |
event | A business event: ticket resolved, payment settled, PR merged | 0.9 |
feedback | A person's rating or thumbs-up | 0.8 |
outcome is "success" or "failure". You can add a detail dictionary (it is redacted before storage). With runs in parallel, always pass run_id.
How the outcome is decided
- The strongest source wins:
testovereventoverfeedbackover the AI judge. Within one source, the latest signal wins. - If no signal arrives, Retrack's AI judge reads the run once and decides. Its confidence is capped at 0.7, or 0.5 for runs that touched untrusted input.
- The judge compares the final answer with the steps: a claimed success after a failed key step is a failure. A hand-off to a human is
unknown. - A crash in a tool or in your agent's code is recorded as a failure. A crash of the model call (a provider outage) is not your agent's mistake, so it gets no outcome.
Why hard signals matter
Only hard signals (test, event, feedback, confidence 0.8 or more) from trusted runs can promote a playbook to active on their own. Judge verdicts add support, but a person must approve before judge-only playbooks reach your agent.
If a later signal contradicts what Retrack learned from a run, for example a test fails after the judge said success, what that run taught is taken back.
Good sources of signal
- Support agents: the ticket was resolved without reopening; the customer rated the answer.
- Ops and back-office agents: the order, refund or record is in the right state afterwards.
- Coding agents: tests pass; the pull request was merged.
- Data agents: the output passed schema or row-count checks.