--- title: "Maestro — Multi-model orchestration | Harshith Nayaka L" description: "Multi-model orchestration engine: a conductor routes one task across thinker, worker and verifier roles, returning a verified answer with a full decision-log." canonical: "https://harshith-nayaka-l-portfolio.vercel.app/work/maestro" last-updated: "2026-08-22T04:29:48Z" author: "Harshith Nayaka L" content-type: "text/markdown" html-version: "https://harshith-nayaka-l-portfolio.vercel.app/work/maestro" --- # Maestro — Multi-model orchestration > Get frontier-quality answers out of free models by orchestrating them: a conductor routes one task across thinker, worker, and verifier roles, and shows its work at every step. Case study by Harshith Nayaka L, AI Engineer (Full-Stack), Bengaluru, India. Canonical page: https://harshith-nayaka-l-portfolio.vercel.app/work/maestro - **Type:** Open-source orchestration engine - **Pattern:** Conductor + Thinker / Worker / Verifier - **Stack:** Python + FastAPI, deployed - **Status:** Live & deployed ## The problem A single free model is uneven: strong on some tasks, unreliable on others, and impossible to fully trust because you can't see how it reached an answer. The obvious workaround, pay for a bigger model, isn't the interesting one. The interesting claim, backed by Sakana's TRINITY and Mixture-of-Agents research, is that intelligent orchestration beats raw model size. Maestro is a glass-box, open-source rebuild of that idea: not cheaper tokens (the models are already free), but better answers from them, with the reasoning made visible instead of hidden. ## What I built A Conductor model reads a task and assigns Thinker, Worker, and Verifier roles across a pool of free LLMs, then a Synthesizer produces the final answer. Crucially, the Verifier is always a different model family than the Worker, which mitigates the well-documented 10–25% self-preference bias in LLM-as-judge. Every step appends to a structured, replayable decision-log, the plan, the routing rationale, each model's output, the verifier's verdict, token and latency cost. That log is the actual product: it's what a black-box orchestration layer can't give you. It's engineered to run in the real world on free tiers: a per-model token-bucket limiter enforcing both RPM and TPM, exponential backoff with jitter on 429s, and diversified fallback chains so one failed call never crashes a run. Models are swapped by editing one config file, never the orchestration code. It deploys publicly without leaking your quota: API-key auth, per-client rate limiting (globally consistent via Upstash Redis on serverless), security headers, input hardening, and strict CORS, with a startup self-audit that warns on unsafe production config. ## Pipeline - **Task:** Task in (A single prompt / problem) - **Conduct:** Conductor (Plan + routing rationale) - **Reason:** Thinker (Strategy for the answer) → Worker (Produces the answer) - **Verify:** Verifier (Different model family; 1 bounded retry) - **Deliver:** Synthesizer (Final answer) → Decision-log (Every step, replayable) ## The judgment calls **The judge is never the same family as the worker** LLM-as-judge has a measured 10–25% self-preference bias, so a model grading its own family's output is compromised. Maestro's verifier is always a different model family than the worker, and a failed check triggers exactly one bounded retry rather than an open-ended loop. **The decision-log is the product** Every run emits a complete JSON log: the plan, why each model was routed where, each step's output and verdict, and the token/latency totals. You can replay and audit exactly how an answer was built, which is the whole point of a glass-box design. **Built for free-tier limits, honestly** Groq's free tier binds on tokens-per-minute, not requests. Maestro reserves estimated tokens before a call so it defers instead of getting 429'd, backs off with jitter when it does, and falls back across model families. Long-context steps route to Gemini's roomier budget. **Swap models without touching code** The model pool, role chains, and rate limits live in one config file; the orchestration logic never hard-codes a model ID. When the Groq catalog rotates, or a Llama model is retired, you edit config, not Python. **Deployable without leaking your quota** API-key auth, per-client rate limiting that stays consistent across serverless instances via Upstash Redis, security headers, input hardening, and a production self-audit that warns on wildcard CORS or mock mode left on. It's meant to be put on the public internet safely. ## What it changed **What it demonstrates:** Systems thinking about LLMs: routing, verification across model families, honest cost accounting, rate-limit engineering, and a security posture, all in service of making unreliable free models produce trustworthy, auditable output. **Honest disclosure:** Maestro's conductor is prompt/rule-based, not a trained coordinator like Fugu's evolved model. It's a faithful re-creation of the concept, made open and transparent, not a claim to have reproduced the trained artifact. The benchmark harness is deliberately built to report where orchestration doesn't help. **Status:** Open-source (MIT) and deployed live on Vercel, with a mock provider so the full flow, dashboard, and tests run offline with no API keys. ## Built with Python, FastAPI, Groq (Llama / Qwen / gpt-oss), Google Gemini, Pydantic, Token-bucket rate limiting, Upstash Redis, Vercel / Railway, n8n ## Links - [Live app](https://maestro-psi-neon.vercel.app/) - [View on GitHub](https://github.com/HarshithNayakaL/Maestro)