Model Routing — Technical
README · ARCHITECTURE · TECHNICAL · source
1. Two levels of indirection, not one
Problem. Model ids appear at every call site. A model upgrade becomes a find-and-replace across a codebase, and the reason each site chose that model is nowhere.
Decision. CallProfile (intent) → ModelTier (size) → model id, with a
wrangler var able to override any tier per deployment.
Why. The two levels absorb different changes. Profile→tier absorbs "this kind of call should be smarter"; tier→id absorbs "a new model exists". Collapsing them into one map means every model change is also a semantic change to what the call is for.
Tradeoff. Two lookups and a vocabulary to learn before anyone can read a call site. For three profiles and three tiers that is close to the break-even point; the indirection would not pay for itself with only one profile.
Open edge. The profile set is small and specific to this system — chatter,
extraction, analysis are the three kinds of call this system happened to have.
A fourth kind will be forced into one of them or added ad hoc, and there is nothing
enforcing that the set stays coherent.
2. Tiers named by size, never by model
Problem. The obvious names for three tiers are haiku, sonnet, opus.
Decision. Use small, medium, large.
Why. From the source: naming a tier haiku would make it a lie the day
it points at Sonnet, and the indirection is the entire reason this exists. Size
names also carry an unambiguous ordering, which model names do not for anyone who
has not memorised the lineup.
Tradeoff. The names are less informative at a glance. small does not tell a reader what
they will be billed.
Open edge. Size is one dimension, and it stops describing the choice as soon as models differentiate on something other than cost and capability — a reasoning model, a long-context model, a fast-streaming one. The tier vocabulary has no place to put those, and that is a real constraint rather than a hypothetical one.
3. Unknown models are assumed new
Problem. temperature is rejected by newer models with a 400. Some check has
to decide whether to send it.
Decision. supportsTemperature is a denylist matching old families
(haiku-4|sonnet-4|opus-4). Anything unrecognised is treated as new, and
temperature is dropped.
Why. The two failure directions are not symmetric. An allowlist of known ids breaks the application on every model release until a human updates a list. The denylist's failure is sending no temperature to a model that would have accepted it, rather than sending a parameter the model rejects.
Tradeoff. A model that accepts temperature and is not in the list will not get it, and the current path does not report that omission.
Open edge. It is a regex over model ids, which is a guess about a naming convention that is not a contract. The list also has an awkward lifecycle: it grows by one entry per generation that still accepts temperature, and becomes dead code when none do.
4. Selector jobs do no work
Problem. A job that processes "everything that needs it" is proportional to corpus size, and a worker invocation is not.
Decision. summarize_batch and archive_batch run one capped query
(LIMIT 20) and enqueue one worker job per row. They perform no summarization or
archival themselves.
Why. Each worker job handles one item, a failure retries one item rather than the batch, and a single selector run has a bounded cost.
Tradeoff. This adds two queue hops and uses a fixed cap rather than an adaptive one. A system idle for a month catches up over several ticks rather than in one.
Selection order still matters when a backlog grows. That is an operational policy to test, not a guarantee supplied by the queue pattern itself.
5. At-least-once delivery moves retry safety into jobs
Problem. Queue delivery is at least once, so a job can be run again.
Decision. Retry tolerance is handled in the jobs. Current operations use
INSERT OR REPLACE on link tables, find-or-create on branches, and repeatable R2
deletes. The consumer acknowledges success, retries on throw, and logs the attempt
count on every failure.
Why. Pushing retry behavior into the jobs keeps the consumer small and means each job's author owns its own re-run behaviour, which is where the knowledge is.
Tradeoff. It is a convention, not a constraint. Each operation still needs to be checked for safe redelivery, and repeated failures need an operational path beyond the small consumer shown here.
What transfers between the two systems
The workspace backend and Journal Club solved routing independently, and converged on the same two ideas:
- Normalise at the boundary. One module owns the provider call and returns a
shape the rest of the system can rely on. In the worker that is
callClaude; in Journal Club it is theGenerationdataclass. - Keep failures inspectable. The workspace throws an error with provider
details for its caller to handle. The orchestrator instead returns a
Generationwithfinish_reason="error", allowing the experiment to save the failed attempt. These are two different error-handling contracts.
The thing that did not transfer, and should: the worker has no equivalent of
Generation's careful distinction between "not reported" and "zero". That gap is
the subject of the
Systems Logging and Telemetry.