Systems Logging and Telemetry - Architecture
README · ARCHITECTURE · TECHNICAL · source
Where the signals come from
Three extraction paths exist because providers disagree about where the model id
lives: model for most, modelVersion for Google Gemini, meta.model for Cohere,
and inside message_start for Anthropic streaming. Each falls through to
"unknown" rather than throwing — an unidentifiable response is still a response,
and losing it entirely would bias the counts toward the providers that are easy to
parse.
model:gossip only records changes. A thousand identical calls write nothing
to it. That makes it a drift log rather than a request log, and small enough to
keep in one KV value.
Family, not exact id
Counts are keyed by family — opus, sonnet, haiku — rather than by the full
dated id.
claude-opus-4-6 → opus
claude-sonnet-4-20250514 → sonnet
gpt-4o-2024-08-06 → gpt-4o (date suffix stripped)
grok-3 → grok-3
A dated snapshot rolling forward would otherwise fragment one model's history into unrelated buckets, and the question worth asking of these counters is "how much am I spending on large-tier calls", not "how much on the 2024-08-06 build".
Non-Claude ids keep their name with a trailing date suffix stripped, which is a looser rule and is acknowledged as such: it works on the providers here and is a guess about naming conventions that are not contracts.
Normalising a generation
Journal Club's Generation is the richer half of the story, because it must hold
four providers' disagreements without flattening them.
Every optional field distinguishes None from 0. None means the
provider does not report this; 0 means it reported none. When a provider includes
reasoning-token metadata, had_thinking can use it as a graded, provider-native
signal. Those values help interpret a response, but they are not treated as directly
comparable measurements across providers.
Four kinds of "the response is short"
These are different facts and the dataclass keeps them apart:
| Field | Means |
|---|---|
finish_reason == "stop" | The model finished |
finish_reason == "length" → hit_token_cap | The provider cut it off |
truncated_by_us | Our own display limit clipped it |
finish_reason == "error" | The call failed; this is still a row |
raw_chars records length before any truncation of ours, so the measured
length is the model's, not the display layer's. The experiment passes
max_len=None to disable our truncation entirely — the 1900-character cut exists
for Discord, and letting a presentation constraint into a dataset whose measures
include response length would manufacture a finding.
Storage housekeeping
The workspace separates archiving, backup, and cleanup because each job understands a different part of the data. The useful rule is simple: cleanup should act only on objects it can cross-reference confidently. Anything outside that scope should be left alone.
Backups have a different job. A partial backup and a failed backup are different outcomes, so coverage needs to stay visible instead of being flattened into one success signal. The detailed failure-path review stays in the private engineering notes.
Rate limiting
Some model routes need a spend ceiling. The current limiter is intentionally lightweight: it adds useful friction around a small personal system, but it is not an exact quota or a multi-user authorization boundary. Describing it as approximate is part of describing it accurately.
What is deliberately not recorded
The counters and drift history described on this page store operational metadata, not conversation content. Backups are separate and private because their purpose is to preserve application data rather than measure system behavior.