Skip to content

feat: per-turn agent trace table for latency and outcome tracking #1012

Description

@cgcardona

Background

Inspired by Kit-Daemon's trace-based learning pattern. We store ACAgentRun rows but capture almost nothing about what happens inside a run at turn granularity.

Problem

We cannot answer:

  • Which issues take the most turns to complete?
  • Which roles fail (cancelled/error) most often?
  • What is the p50/p95 LLM response latency for local vs. cloud?
  • Is Qwen 9B meaningfully slower than 14B for reviewer tasks?

All of this data flows through agent_loop.py and is silently discarded.

Proposed solution

Add a lightweight ACAgentTrace table that records one row per LLM turn:

id            UUID PK
run_id        FK -> ACAgentRun
turn_number   int
role          str  (developer, reviewer, etc.)
model         str  (which model was actually called)
input_tokens  int
output_tokens int
cache_read    int
cache_write   int
latency_ms    int  (wall-clock time for the LLM call)
stop_reason   str  (tool_use, stop, max_tokens)
tool_calls    int  (count of tool calls in this turn)
recorded_at   timestamp

agent_loop.py already has all of this data at the point it processes each LLM response -- it just does not persist it.

Acceptance criteria

  • ACAgentTrace SQLAlchemy model in agentception/db/models.py
  • Alembic migration
  • persist.py: record_agent_trace() async function
  • agent_loop.py: call record_agent_trace() after each LLM response
  • queries.py: get_run_traces(run_id) and get_model_latency_stats() queries
  • Unit tests for persist and queries
  • mypy clean

Future use

This table feeds the complexity-based routing decision (issue #1011) and the degradation detection monitor (separate issue). It is also the foundation for any future prompt optimization work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions