--- title: "CraftConnect — Multi-modal artisan assistant | Harshith Nayaka L" description: "Voice-and-photo assistant on Google Gemini letting artisans run an online storefront by talking, not typing. Gen AI Exchange Hackathon 2025 semi-finalist." canonical: "https://harshith-nayaka-l-portfolio.vercel.app/work/craftconnect" last-updated: "2026-10-12T03:21:37Z" published: "2026-06-06T01:30:29Z" author: "Harshith Nayaka L" author-role: "AI Engineer" author-title: "AI Engineer" author-employer: "DemandNXT (title: AI Engineer - Full Stack)" author-location: "Bengaluru, India" author-availability: "Available for freelance work" author-email: "harshith28124@gmail.com" content-type: "text/markdown" html-version: "https://harshith-nayaka-l-portfolio.vercel.app/work/craftconnect" --- # CraftConnect — Gen AI Exchange Hackathon 2025 > Let an artisan stand up an online storefront by talking and showing a product, instead of typing forms in a language that isn't theirs. Case study by Harshith Nayaka L, AI Engineer, Bengaluru, India. Canonical page: https://harshith-nayaka-l-portfolio.vercel.app/work/craftconnect - **Role:** Product lead, full frontend, co-built backend - **Context:** Gen AI Exchange Hackathon 2025 - **Scale:** 270,000+ developers nationally - **Status:** Built & deployed on Google Cloud ## The problem Most artisans don't lose sales because their craft isn't good enough. They lose them because getting a product online means typing English product descriptions, setting prices, writing tags, and fighting a form-heavy interface that assumes you already know e-commerce. That barrier is the whole problem. The skill is in the hands; the friction is in the keyboard. ## What I built CraftConnect removes the keyboard. An artisan photographs a product and describes it out loud in their own language. From that, the system produces a complete, structured listing: a title, a written description, suggested categories and tags, and a storefront page, ready to publish. It's multi-modal by necessity, not for show. The photo carries information the voice doesn't (material, colour, form), and the voice carries information the photo can't (story, intended use, price intent). The pipeline fuses both into one validated listing. I led the product, built the entire frontend, co-built the backend, and owned the decisions that actually mattered: which AI services to use where, and what the end-to-end user journey should feel like. ## Pipeline - **Capture:** Voice input (Artisan's own language) → Product photo (One or more images) - **Understand:** Speech-to-text (Transcribe + detect language) → Vision analysis (Gemini reads the product) - **Compose:** Fuse signals (Voice intent + visual attributes) → Listing generation (Gemini, schema-constrained) - **Verify:** Validate fields (Title, price, tags present & sane) → Localize (Buyer-facing language) - **Publish:** Storefront page (Ready to go live) ## The judgment calls **Voice and vision are treated as two witnesses, not one** The photo and the spoken description are analyzed separately, then reconciled. When they agree, confidence is high. When they conflict (the voice says 'silk' but the image reads cotton), the system surfaces it rather than silently guessing. That's a deliberate reliability choice, not an accident of the model. **Generation is schema-constrained, not free-text** The model isn't asked to 'write a listing'. It's asked to fill a defined structure: title, description, category, tags, price band. Constraining the output shape is what makes it usable by the rest of the app instead of being a paragraph someone still has to parse. **Language is decoupled from logic** The artisan speaks one language; the buyer may read another. Localization happens as its own step at the end, so the understanding pipeline never has to care what language it started in. One pipeline, many markets. **AI service selection was a judgment call, not a default** I chose where to spend model capability and where a cheaper, narrower service was enough. Speech-to-text, multimodal reasoning, and generation are different jobs with different cost and latency profiles, and they were wired accordingly. ## What it changed **What changed:** A task that previously required literacy in English, e-commerce, and form-filling collapses into 'show it and say it'. The artisan's expertise stays in the craft, not the keyboard. **Recognition:** Built and deployed for the Gen AI Exchange Hackathon 2025, a national event with 270,000+ developers, and reached the semi-finals. **Honest scope:** This was a hackathon build, not a production marketplace. The value it proves is the interaction model and the multi-modal pipeline behind it, both of which hold up. ## Questions this answers **How can AI help artisans sell their products online?** By removing the keyboard. In CraftConnect, which Harshith Nayaka L led and which reached the semi-finals of the Gen AI Exchange Hackathon 2025, an artisan photographs a product and describes it aloud in their own language, and the system produces a complete structured listing — title, description, categories, tags and a storefront page — ready to publish. The expertise stays in the craft instead of in English product copy and e-commerce forms. **How do you combine voice and image input in a multimodal AI app?** Treat them as two witnesses rather than one input. In CraftConnect, built by Harshith Nayaka L’s team, the photo and the spoken description are analysed separately and then reconciled: when they agree, confidence is high, and when they conflict — the voice says silk, the image reads cotton — the system surfaces it instead of guessing. The listing is then generated against a fixed schema, and localisation runs as its own final step so the understanding pipeline never depends on the input language. ## Built with Gemini (multimodal), Google Cloud Speech-to-Text, Google Cloud, React, Node.js, Structured / schema-constrained output ## Links - [View on GitHub](https://github.com/HarshithNayakaL/craftconnect)