ccRAG: Credit-Card Q&A

A bilingual (English / 中文) question-answering system for 54 credit cards from Chase, American Express and Bank of America. Ask in either language and get a short answer in which every figure cites its source: the bank’s own product page, cardmember agreement or benefits guide, or a dated blog post. Retrieval runs on LlamaIndex and Qdrant, a local 9B model writes the answers, and the whole system is served to the public from one home GPU. Every design choice was settled by a pre-registered experiment on a hand-reviewed benchmark.

Try it live

Private repository. Feel free to reach out if you’d like to see the code.

Highlights

• Retrieval was the lever: small-to-big chunking (512-token leaves, 2,048-token parents) and a card-name filter built from a hand-reviewed alias table (Chinese nicknames, abbreviations like CSP and BCE) raised test Recall@5 from 0.62 to 0.93. For Chinese questions whose facts exist only on English bank pages, it rose from 0.34 to 0.97.
• Without retrieval the model is useless: asked closed-book, the local Qwen3.5-9B got 57 of 61 test questions wrong (judge score 0.10 / 2). With retrieval it scores 1.69 / 2, within 0.2 of Claude Sonnet, at $0 per query.
• Honest evaluation: a 122-question benchmark with gold answers and verbatim evidence quotes, reviewed question by question. Choices were made on a dev split and the test split was used once. Rules were written before running, and every comparison has a paired-bootstrap 95% CI.
• Negative results kept: hybrid search, a cross-encoder reranker, official-first ordering and thinking mode were each tested; none helped significantly. The reranker cost 7–20× latency, and thinking mode left 11 of 61 answers empty.
• LLM judge calibrated against a human: a blind Claude Opus judge on a 0–2 rubric agreed with my own blind ratings at quadratic-weighted κ 0.63, never more than one level apart.
• Running in public: a hardened server behind a Cloudflare Tunnel with:
  • a request queue that shows each visitor’s place in line
  • per-IP rate limits
  • prompt-injection guards that wrongly refused none of the 122 benchmark questions
  • a visitor and feedback log viewed on a stats page only I can reach

How It Works

1. Crawl & parse — headless Firefox renders the banks’ product pages, including hidden tabs and screen-reader text. Agreements and benefit PDFs are kept as-is, and blog posts in English and Chinese are added. The result is 237 documents converted to Markdown.
2. Index — small-to-big chunks embedded with multilingual BGE-M3 into Qdrant: 2,961 passages.
3. Find the card — an alias table maps names like “CSP” or “蓝宝石” to cards and limits the search to them. An ambiguous name triggers “which card do you mean?” instead of a guess.
4. Answer, cited — the 5 best passages go to the model with labels (official or blog, date, language), and the prompt says the official figure wins. Citations are renumbered in order of use, and only cited sources are shown.

Results

Test split (61 questions), run once after every choice was made on dev

Where the local model still loses is deciding which source to trust. On traps where a blog advertises a higher offer from another channel, Qwen scored 0.6 / 2 and Sonnet 2.0 / 2.

Screenshots

The wallet: drag a card into the question bar (or double-click it) to limit the search to that card

A comparison answer; hovering a citation shows the source passage with the figure highlighted

"Sapphire" fits two cards, so the app asks instead of guessing

The same app in Chinese: an answer about a dragged-in card, with official and blog sources

Running It in Public

• Separate from development: the public server serves a released snapshot (page, card art and its own index copy). Work in progress never goes live, and the public server never competes with experiments for the vector store’s lock.
• Fixed and bounded:
  • one model and fixed settings; anything the browser sends is ignored
  • 500-character questions and 800 output tokens
  • one answer at a time, at most 6 waiting
  • per IP, 6 questions a minute and 60 an hour
  • errors reach the browser as codes only
• Prompt injection:
  • The public prompt refuses off-topic requests and ignores instructions hidden in the question.
  • Of 9 attacks (role play, “print your system prompt”, essays, repetition, harmful requests), 8 were refused outright.
  • The 9th asked the model to append a link. A server-side filter now drops any sentence naming a site that no source mentions.
• Analytics without accounts:
  • An anonymous per-browser ID counts unique visitors.
  • Visits, questions, answers and 👍/👎 go to a local log.
  • A stats page shows them; it is reachable only by me.
• Runs itself: systemd services start the site, the tunnel and the stats page at boot. No port is opened and the home IP stays hidden.

Technical Summary