Protege

Appearance

Observing your agent ​

Sundae and Scout are working — but an agent you can't see into is an agent you can't trust. This chapter tours the two ways Protege lets you watch: the live introspection panel for what's happening right now, and tracing for a durable record you can review later.

The dashboard ​

Everything lives under the mount you set up — http://localhost:3000/protege. It's a real, if compact, console:

  • Personas — your agents (Sundae, Scout), their instructions, tool scope, and access rules.
  • Threads — every conversation, with the full message history, where you can also reply.
  • Messages — the flat stream of inbound and outbound mail.
  • Responsibilities & runs — Scout's duties and the history of each execution.
  • Domains — registered email domains and their DNS records.
  • Traces — captured inference turns (when tracing is on; see below).

Remember the mount ships no authentication — fine on localhost, but you must wrap it before production. That's the first item in going to production.

The introspection panel ​

Open a thread and you'll see a live activity feed alongside it — the introspection panel. As Sundae handles a message it streams the model's output, each tool call as a card (the arguments she passed, the result she got), and the run's start and finish markers. When a customer asks about order SCP-1042, you watch lookup_order fire with { "order_number": "SCP-1042" }, see the status come back, then see send_email compose the reply — in real time.

This is the fastest way to understand — or debug — an agent: you see exactly what it saw and did, in order. It's driven by the same events that power hooks, so it costs the run nothing.

The panel is live-only

The feed shows activity as it happens; it isn't persisted, so reloading the page starts it empty. For a durable record, use tracing.

Tracing ​

Where the panel is ephemeral, tracing captures a durable snapshot of every inference turn — the exact request sent to the model, the response it produced, and the settings that shaped it. It's off by default; turn it on in the initializer:

ruby
# config/initializers/protege.rb
config.tracing = { enabled: true }

Now every turn is saved as a trace, browsable under Traces, newest first. Each shows the correlation id (which ties all the turns of one run together), the model, and the full request/response JSON.

Reviewing and searching ​

Traces are built for two things:

  • Review. Open a trace and apply a label — good, bad, or neutral — with a note. Over time this builds a labeled dataset: the good turns are fine-tuning material, the bad ones show you where an agent went wrong.
  • Search. The trace search filters by correlation id (pull up every turn of one run — the workhorse when debugging a specific conversation), by model, by free text against the request or response JSON (e.g. every turn mentioning "refund"), or by review label.

See Tracing for the full field list and the current limitations (no export or auto-labeling yet).

Next ​

You can build agents, scope them, schedule them, and watch them. The last step is shipping The Scoop for real.

→ Going to production.