--- title: "When to run an LLM locally | Harshith Nayaka L" description: "When to run an LLM locally rather than through an API, what it asks of you, and how to build AI apps that still run with no API keys at all." canonical: "https://harshith-nayaka-l-portfolio.vercel.app/guides/running-llms-locally" last-updated: "2026-10-31T06:08:19Z" published: "2026-10-29T08:02:52Z" author: "Harshith Nayaka L" author-role: "AI Engineer" author-title: "AI Engineer" author-employer: "DemandNXT" author-location: "Bengaluru, India" author-availability: "Available for freelance work" author-email: "harshith28124@gmail.com" content-type: "text/markdown" html-version: "https://harshith-nayaka-l-portfolio.vercel.app/guides/running-llms-locally" --- # Running LLMs locally: when to keep AI on your own machine > Keep the model on your own machine when the data shouldn't leave it or the tool has to work offline. AI Notes runs DeepSeek R1 locally through Ollama, so notes never reach an external API and the app works offline once the model is pulled; the paper I co-authored on it reports 87% user satisfaction on summarisation with responses in 1.9 to 3.8 seconds. Below: what local inference buys you, what it asks of you, and how to build AI apps that still run with no keys at all. Guide by Harshith Nayaka L, AI Engineer, Bengaluru, India. Canonical page: https://harshith-nayaka-l-portfolio.vercel.app/guides/running-llms-locally ## What local inference buys you In AI Notes there is no API endpoint receiving your notes, because there is no external call at all. Summarising, improving writing, expanding an idea, generating questions and extracting keywords all happen on the user's own machine. Privacy isn't a policy promise there; it's the architecture. It also removes two dependencies: once the model is pulled, the AI features keep working with no internet, and they don't depend on a server being up or a bill being paid. From the case study: [AI Notes](https://harshith-nayaka-l-portfolio.vercel.app/work/ai-notes/index.md). ## What it asks of you The model has to be pulled once and runs on the user's machine, so that machine sets the ceiling on what you can run. AI Notes is deliberately small, plain HTML, CSS and JavaScript with notes stored in the browser, and it's a focused build proving the local pattern rather than a feature-complete notes product. Local inference fits private, single-user tools; a service many people share is a different decision. From the case study: [AI Notes](https://harshith-nayaka-l-portfolio.vercel.app/work/ai-notes/index.md). ## Keep the actions local too Local inference is half of it if the tool also acts. Personal MCP OS keeps execution on the machine: its core binds to loopback and requires a token on every endpoint, the phone reaches it through an adb reverse tunnel so there is no LAN listener and no cloud relay, and credentials are encrypted at rest with Windows DPAPI and Android Keystore. Nothing exports cookies, passwords or auth headers, so an MCP client can work the machine without any of it leaving or passing through a hosted service. From the case study: [Personal MCP OS](https://harshith-nayaka-l-portfolio.vercel.app/work/personal-os-mcp/index.md). ## Build it to run with no keys at all Cannon boots, builds, passes its full test suite and serves a working demo with no provider key and no database: a scripted model stands in for inference, a deterministic lexical embedder for embeddings and a process-local store for Postgres. That lets CI run true end-to-end tests with zero secrets against the production code path instead of a mocked shortcut, and real inference and storage become opt-in upgrades. Maestro ships a mock provider so the full flow, dashboard and tests run offline. BrandForge can be reviewed from local fixtures with no keys and no network, and the fixtures say they're fixtures so a demo can't pass for a real result. SPECTRA's deterministic half still runs without a Gemini key. From the case studies: [Cannon](https://harshith-nayaka-l-portfolio.vercel.app/work/cannon/index.md), [Maestro](https://harshith-nayaka-l-portfolio.vercel.app/work/maestro/index.md), [BrandForge](https://harshith-nayaka-l-portfolio.vercel.app/work/brandforge/index.md), [SPECTRA](https://harshith-nayaka-l-portfolio.vercel.app/work/spectra/index.md). ## When you do call a hosted model, keep the keys off the client In Nova, the provider key stays in a serverless function and the browser never holds a credential; chat threads live in local storage rather than on a server. BrandForge keeps both model credentials in its API process only, and Octo locks its paid provider on the server behind an access word that never reaches the browser bundle. From the case studies: [Nova](https://harshith-nayaka-l-portfolio.vercel.app/work/nova-ai/index.md), [BrandForge](https://harshith-nayaka-l-portfolio.vercel.app/work/brandforge/index.md), [Octo](https://harshith-nayaka-l-portfolio.vercel.app/work/octo/index.md). ## Self-hosting the app and self-hosting the model are separate choices Octo is a self-hosted, single-workspace app that runs locally on Node's built-in SQLite or on Vercel with Neon Postgres, yet it calls hosted models: gpt-6-astra through the OpenAI Agents API, and Gemini 3.8 Flash. Where the app runs and where the model runs are two decisions. Make the first on who needs to use it, and the second on whether the data can leave the machine. From the case study: [Octo](https://harshith-nayaka-l-portfolio.vercel.app/work/octo/index.md). ## Questions this answers ### Is a local LLM fast enough to be useful? For focused tasks, yes. AI Notes, built by Harshith Nayaka L, runs DeepSeek R1 locally through Ollama; the paper he co-authored on it reports response times of 1.9 to 3.8 seconds and 87% user satisfaction on summarisation. ### When should you run an LLM locally instead of through an API? When the data shouldn't leave the machine or the tool has to work offline. Harshith Nayaka L's AI Notes keeps inference on the user's machine for exactly those reasons; a service many people share, or a model your hardware can't run, points to a hosted API instead. ### How do you test an LLM app without API keys? Give every external dependency a working stand-in. Cannon, built by Harshith Nayaka L, boots, builds and passes its full test suite with no provider key or database, using a scripted model, a deterministic embedder and a process-local store, so CI tests the real code path with zero secrets.