Protege

Appearance

Tracing ​

Tracing captures a durable snapshot of every inference turn — the exact request sent to the model, the response it produced, and the model and settings that shaped it — so you can review what your agents are actually doing and build a labeled dataset for fine-tuning. It's off by default; nothing is recorded until you opt in.

Turning it on ​

ruby
# config/initializers/protege.rb
config.tracing = { enabled: true }

With tracing on, the harness emits a per-turn event that the tracing subscriber snapshots into a Protege::Trace row. Each row is a self-contained training example in the making: the wire request (resolved system prompt, history, the user turn, the tool catalog, and any attachment bytes inline), paired with the response and the inference settings, frozen at that instant.

Rows can be large

Multimodal turns store attachment bytes inline, so a trace of a message with a photo carries that photo's bytes. Enable tracing deliberately, and prune old rows if the table grows — traces have no foreign keys, so the table can be truncated freely.

Reviewing traces ​

Traces are browsable in the dashboard under /protege/traces, newest first. Opening one shows the full turn: the correlation id (which ties every turn of a single run together), the model, the settings, and the pretty-printed request and response JSON.

From the show view you apply a review label — good, bad, or neutral — with an optional note. Labeling is how you separate signal from noise: a fine-tuning corpus is the traces you marked good, and the bad ones tell you where an agent went wrong. Until reviewed, a trace sits unlabeled.

Searching traces ​

The trace list has a search (/protege/traces/search) backed by Protege::TraceSearch. You can filter by:

  • correlation_id — pull up every turn of one run. This is the workhorse: grab the correlation id from the introspection panel or a log line, and see exactly what the model saw and produced on each turn of that run.
  • model — narrow to one model.
  • request / response — free-text search against the serialized JSON, e.g. find every turn whose prompt or output mentions "refund".
  • annotation — search your review notes.
  • label — filter by verdict.

Every filled field is ANDed, so terms combine to narrow the same row. Text matching is case-insensitive.

Limitations ​

Tracing is built for in-dashboard review today; a few things aren't there yet:

  • No export. There's no JSONL (or other) export of the labeled corpus — you review in the dashboard, and pulling the data out for a training run is a query you write against the Protege::Trace table yourself.
  • No automatic labeling. Every verdict is a human one; there's no heuristic or model-assisted pre-labeling.