Case study · LLM / RAG · Sep 2026

Phone Advisor: a local RAG and multi-agent system

Answers questions about Samsung phones and writes grounded reviews and comparisons, using only local open-source models. Built end to end: scraper, database, retrieval, agents, API and tests.

Context

Built solo as a technical assessment, in about three days

Stack

Python, requests + BeautifulSoup, SQLAlchemy (MySQL, PostgreSQL or SQLite), sentence-transformers (MiniLM), FAISS, Qwen2.5-1.5B-Instruct, CrewAI, FastAPI

Links

Demo (recorded answers) · Code on GitHub

Problem

People ask phone questions in very different shapes: "which has the best battery?", "compare the S23 and S24", "what camera does the S24 have?". A single vector search handles the last one well and the first two badly, because embeddings cannot rank numbers or guarantee that both phones in a comparison are retrieved.

Architecture

Key decisions

• Route before retrieving. Superlatives become SQL rankings over parsed numeric columns (battery mAh, refresh rate, weight), with tie-breaks. Comparisons fetch both records. Only open-ended lookups use vector search.
• Chunk by aspect. Each phone is split into 8 aspect passages (display, camera, battery and so on), 120 passages in total. Retrieval keeps only passages for the asked aspect, which stopped a display refresh rate from being reported as a camera spec. A regression test guards it.
• Ground the small model. Qwen2.5-1.5B skipped the CrewAI tool call and occasionally invented specs, such as a wrong camera resolution or a non-existent chipset. The mitigation was to inject the verified database record into every agent task, so the writer works from facts rather than memory. It is not yet measured by an evaluation set (see below).
• Degrade gracefully. If no GPU or model is available, answers fall back to templates built from the same retrieved facts, and a native orchestrator replaces CrewAI. A health endpoint reports which backend is live.
• Parse, don't guess. Spec parsers return nothing rather than a guess, with range checks, so bad values never enter the rankings.

Result

• 15 phones, 858 specification rows and 53 price listings scraped into a three-table schema that runs on MySQL, PostgreSQL or SQLite.
• 11 FastAPI endpoints with Pydantic validation, Swagger docs and a browser UI.
• 49 unit tests across parsers, routing, retrieval, agents and the HTTP contract, all passing without a GPU.
• Docker image (non-root, SQLite seeded on first start) and GitHub Actions CI that runs the tests, the evaluation and an image build on every push.

Evaluation

I built a labelled set of 188 questions (lookups, prices, comparisons, rankings, recommendations, nicknames such as "S23U", and off-topic questions), each checked against the database, and split it in half. I fixed routing using only the dev half, then measured once on the held-out test half.

* 93.3% with all scope rules; 53.3% without three rules that happen to match only test questions, so the honest held-out gain is at the lower end. With the local LLM, answers whose every number is backed by the retrieved data rose from 86.4% to 93.5%. Full results

Limitations and next steps

• Intent routing and the scope check are word lists; a small classifier evaluated on the same set is the next step.
• Only one LLM run per side was made, and the model samples, so LLM-mode numbers carry noise.
• The public demo shows recorded answers; a live hosted version needs paid hosting, so it runs locally with Docker for now.