A few details make a big difference to what a model comparison can tell me.

README · ARCHITECTURE · TECHNICAL · source

The summary is part of the experiment

Each comparison leg currently uses its chosen model for both the summary and the proposed update. That compares the whole pipeline. An update can change because the summary changed, even if the update stage would otherwise behave similarly.

The next comparison will reuse a fixed summary, starting state, and prompt. That makes the update model the part I can vary directly.

The route to the model matters too

The shared engine interface gives the HTTP and CLI paths the same system/user prompt structure. A comparison of the same model over both paths is useful for spotting differences in how requests reach it.

There are still differences to account for. The reviewed CLI wrapper ignores the shared maxTokens argument while the API sends it, and the CLI has its own environment. The comparison notes describe the transport checks and the examples that led to them.

A map needs a second view

A basin weight tells me how the update model relates a conversation to a label. Entropy tells me how evenly the normalized weights are distributed. Neither, by itself, establishes that a conversation has a dynamical attractor.

The planned independent measurement uses concepts and clusters from the conversation data. Looking at both views gives the study something to compare, including cases where the map and the conversation data disagree.

Keep the interesting examples traceable

Local ingestion and comparison runs are grouped by transcript hash and model. To repeat a controlled comparison, I also need the starting state, summary, prompt, and configuration used for that output.

Read the full research plan · Inspect the comparison workflow · All projects