Multi-Provider LLM Orchestrator
Routing models from different providers through a shared reading-room workflow.
A shared workflow for models from different AI providers to read, discuss, and respond to the same paper, with an evaluation harness for comparing prompts.
Jennifer Naomi Nguyen · research tooling · Exploratory pilot · February 2026 – present · built with Claude Code and Codex
README · ARCHITECTURE · TECHNICAL · source
What it does
I built a shared workflow for models from different AI providers. Its first use was a Discord reading group I called Journal Club: four language models read the same paper, take notes in model-specific channels, discuss it, and draft comments.
Watching them interact raised a question I couldn't answer just by watching: does a persona prompt change what a model does, or mainly how it sounds? So the project grew a second part — an evaluation harness for comparing prompt conditions.
From a reading room to an experiment
The pilot uses a 2×2 design: with and without the persona text, and with and without an additional identity anchor. The task framing stays the same across all four conditions.
That was a familiar kind of problem from immunology: keep the task in the control, so the comparison is about the part you actually changed. Papers and repeated generations are separate parts of the design, too.
Decisions and gotchas
Saving the raw responses paid off twice. The initial run recorded 24 Claude errors, unevenly spread across conditions. The debugging notes traced them to a parser that expected text in the first returned block, so I could rerun only the Claude arm against the same saved articles.
The same records showed why an identity measure missed introductions like "Grok here." I corrected the matching rule and recomputed the measurements without paying to regenerate the responses. Quoted identity statements can still produce false positives, so annotated examples are the next useful check for that instrument.
The everyday plumbing mattered too. Reusing provider clients and setting timeouts helped keep long runs moving. Splitting long Discord messages preserved the end of a draft, including its approval footer.
The technical page explains these choices.
How it works
The Discord bot and experiment harness share a codebase but have different jobs. The bot manages the reading-room interaction. The harness generates responses under defined conditions and keeps the raw responses alongside their measurements.
The architecture shows the two paths. The source excerpts cover prompt composition, the measurement correction, and a few practical details from running the club.
Current status
The combined analysis set contains 359 non-error responses: the initial GPT and Grok arms plus the Claude rerun. A read-only recount confirmed that total and the shared set of ten articles. Persona-present conditions produced longer responses for Claude and GPT on all ten articles; Grok's pattern was smaller and less consistent. These are descriptive findings from this stimulus set.
The pilot compares response style in single turns. The next run is designed around multi-turn discussion and immunology papers, bringing the evaluation closer to the conversations that prompted the project. It has not been run yet.
Participating models
The reading room brings together OpenAI's GPT, Anthropic's Claude, Google's Gemini, and xAI's Grok. The documented 359-response pilot used GPT, Claude, and Grok; Gemini was not included in that run.
Explore the project
| Page | What you'll find |
|---|---|
| Architecture | Components, workflows, data boundaries, and validation |
| Technical | Prompt design, measurement choices, and practical gotchas |
| Source | Selected code excerpts and the detailed run record |
The cleaned-up Multi-Provider LLM Orchestrator repository is public, with a v0.1.0 release, setup instructions, and 20 offline tests. The original configured implementation and study records remain private. Release tests use synthetic responses; they are separate from the historical pilot.
Attractor — Context Research explores a different research question around memory and conversation patterns. Routing and Systems Logging and Telemetry take a closer look at the infrastructure supporting this work.