--- title: "Maestro — Multi-model LLM orchestration | Harshith Nayaka L" description: "Multi-model orchestration engine: a conductor routes one task across thinker, worker and verifier roles, returning a verified answer with a full decision-log." canonical: "https://harshith-nayaka-l-portfolio.vercel.app/work/maestro" last-updated: "2026-10-12T03:21:37Z" published: "2026-08-03T13:25:26Z" author: "Harshith Nayaka L" author-role: "AI Engineer" author-title: "AI Engineer" author-employer: "DemandNXT (title: AI Engineer - Full Stack)" author-location: "Bengaluru, India" author-availability: "Available for freelance work" author-email: "harshith28124@gmail.com" content-type: "text/markdown" html-version: "https://harshith-nayaka-l-portfolio.vercel.app/work/maestro" --- # Maestro — Multi-model LLM orchestration > Get frontier-quality answers out of free models by orchestrating them: a conductor routes one task across thinker, worker, and verifier roles, and shows its work at every step. Case study by Harshith Nayaka L, AI Engineer, Bengaluru, India. Canonical page: https://harshith-nayaka-l-portfolio.vercel.app/work/maestro - **Type:** Open-source orchestration engine - **Pattern:** Conductor + Thinker / Worker / Verifier - **Stack:** Python + FastAPI, deployed - **Status:** Live & deployed ## The problem A single free model is uneven: strong on some tasks, unreliable on others, and impossible to fully trust because you can't see how it reached an answer. The obvious workaround, pay for a bigger model, isn't the interesting one. The interesting claim, backed by Sakana's TRINITY and Mixture-of-Agents research, is that intelligent orchestration beats raw model size. Maestro is a glass-box, open-source rebuild of that idea: not cheaper tokens (the models are already free), but better answers from them, with the reasoning made visible instead of hidden. ## What I built A Conductor model reads a task and assigns Thinker, Worker, and Verifier roles across a pool of free LLMs, then a Synthesizer produces the final answer. Crucially, the Verifier is always a different model family than the Worker, because LLM judges tend to favour their own kind: in the MT-Bench study GPT-4 gave its own answers a 10% higher win rate and Claude-v1 a 25% higher one (a result its authors call suggestive, not conclusive), and later work found LLM evaluators recognise and favour their own generations. Every step appends to a structured, replayable decision-log, the plan, the routing rationale, each model's output, the verifier's verdict, token and latency cost. That log is the actual product: it's what a black-box orchestration layer can't give you. It's engineered to run in the real world on free tiers: a per-model token-bucket limiter enforcing both RPM and TPM, exponential backoff with jitter on 429s, and diversified fallback chains so one failed call never crashes a run. Models are swapped by editing one config file, never the orchestration code. It deploys publicly without leaking your quota: API-key auth, per-client rate limiting (globally consistent via Upstash Redis on serverless), security headers, input hardening, and strict CORS, with a startup self-audit that warns on unsafe production config. ## Pipeline - **Task:** Task in (A single prompt / problem) - **Conduct:** Conductor (Plan + routing rationale) - **Reason:** Thinker (Strategy for the answer) → Worker (Produces the answer) - **Verify:** Verifier (Different model family; 1 bounded retry) - **Deliver:** Synthesizer (Final answer) → Decision-log (Every step, replayable) ## The judgment calls **The judge is never the same family as the worker** LLM evaluators have been shown to recognise and favour their own generations, so a model grading its own family's output is compromised. Maestro's verifier is always a different model family than the worker, and a failed check triggers exactly one bounded retry rather than an open-ended loop. **The decision-log is the product** Every run emits a complete JSON log: the plan, why each model was routed where, each step's output and verdict, and the token/latency totals. You can replay and audit exactly how an answer was built, which is the whole point of a glass-box design. **Built for free-tier limits, honestly** Groq's free tier binds on tokens-per-minute, not requests. Maestro reserves estimated tokens before a call so it defers instead of getting 429'd, backs off with jitter when it does, and falls back across model families. Long-context steps route to Gemini's roomier budget. **Swap models without touching code** The model pool, role chains, and rate limits live in one config file; the orchestration logic never hard-codes a model ID. When the Groq catalog rotates, or a Llama model is retired, you edit config, not Python. **Deployable without leaking your quota** API-key auth, per-client rate limiting that stays consistent across serverless instances via Upstash Redis, security headers, input hardening, and a production self-audit that warns on wildcard CORS or mock mode left on. It's meant to be put on the public internet safely. ## What it changed **What it demonstrates:** Systems thinking about LLMs: routing, verification across model families, honest cost accounting, rate-limit engineering, and a security posture, all in service of making unreliable free models produce trustworthy, auditable output. **Honest disclosure:** Maestro's conductor is prompt/rule-based, not a trained coordinator like Fugu's evolved model. It's a faithful re-creation of the concept, made open and transparent, not a claim to have reproduced the trained artifact. The benchmark harness is deliberately built to report where orchestration doesn't help. **Status:** Open-source (MIT) and deployed live on Vercel, with a mock provider so the full flow, dashboard, and tests run offline with no API keys. ## Questions this answers **How do you orchestrate multiple free LLMs to get a better answer?** Split the task into roles and check the result with a different model. In Maestro, an open-source orchestration engine built by Harshith Nayaka L, a Conductor assigns Thinker, Worker and Verifier roles across a pool of free LLMs and a Synthesizer produces the final answer; the Verifier is always a different model family from the Worker, and a failed check triggers exactly one bounded retry. Every step goes into a replayable decision-log: the plan, the routing rationale, each output, the verdict, and token and latency cost. **How do you handle Groq free-tier rate limits?** Budget tokens before calling instead of recovering after failing. Groq’s free tier binds on tokens per minute rather than requests, so Maestro, built by Harshith Nayaka L, reserves estimated tokens through a per-model limiter that enforces both requests and tokens per minute, backs off exponentially with jitter on a 429, and falls back across model families. Long-context steps route to Gemini’s larger budget, and models are swapped in one config file rather than in code. ## Built with Python, FastAPI, Groq (Llama / Qwen / gpt-oss), Google Gemini, Pydantic, Token-bucket rate limiting, Upstash Redis, Vercel / Railway, n8n ## Sources 1. [Sakana AI (Xu et al.), TRINITY: An Evolved LLM Coordinator (arXiv 2512.04695)](https://arxiv.org/abs/2512.04695) 2. [Wang et al., Mixture-of-Agents Enhances Large Language Model Capabilities (arXiv 2406.04692)](https://arxiv.org/abs/2406.04692) 3. [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)](https://arxiv.org/abs/2306.05685) 4. [Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations (arXiv 2404.13076)](https://arxiv.org/abs/2404.13076) ## Links - [Live app](https://maestro-psi-neon.vercel.app/) - [View on GitHub](https://github.com/HarshithNayakaL/Maestro)