Skip to content
← Projects

RAG Chat

Chat with your own documents: hybrid search (vector + full-text + trigram) fused with RRF, LLM reranking, an intent router (including SQL over spreadsheets), three-layer conversation memory, streamed answers with citations.

The problem

A language model does not know your private documents and tends to make things up without context. RAG Chat makes the model read the relevant passages before answering, and shows which passage each claim came from.

It has to handle what a plain "chunk, embed, top-k" RAG usually gets wrong:

  • questions typed without Vietnamese tones or misspelled, and product codes or names an embedding does not understand;
  • questions that need calculation over a spreadsheet (sums, averages, rankings), which search cannot answer;
  • follow-ups like "and the second one?", which need conversation memory;
  • public chat that must stay within a budget.

Ingestion

Ingestion runs in a background worker (Redis + ARQ), never inside the upload request. If Redis is down, upload fails with a clear error instead of silently running somewhere else.

  1. Parse: file type detected from magic bytes and checked against the extension; user-supplied file names are not trusted.
  2. Chunk by file type:
    • PDF, DOCX, HTML, Markdown, TXT: by headings when present, otherwise by sentences, about 700–800 tokens with 100 overlap. A heading always travels with the first chunk of its section.
    • Excel, CSV: one summary chunk (columns, types, statistics) plus row groups with the header repeated. Prose tables such as FAQs are detected automatically and get one row per chunk.
  3. Optional: contextual retrieval (a one-sentence context prefix per chunk) and parent–child (search small chunks, let the model read a ~2500-token parent).
  4. Embed with text-embedding-3-small (1536 dimensions), cached by content hash.
  5. Index in one transaction: vectors into pgvector with HNSW, text into an accent-stripped tsvector column.
  6. Summarise each document in one paragraph, used for "what is this document about" questions.

Each vector width has its own table. pgvector cannot build HNSW on vector above 2000 dimensions, so 3072-dimension models use halfvec.

Answering a question

question ├─ memory: trim history to 3000 tokens · recall older turns similar to the question ├─ understand: rewrite into a standalone question + keywords · route by intent ├─ search: dense (30) + full-text (30) + trigram (20) ─▶ RRF ─▶ LLM rerank ─▶ keep 6 ├─ answer: system prompt · summary · memories · history · numbered context ─▶ stream └─ after: store citations + debug · rolling summary · store memory

Hybrid search

Each branch fails where another works:

  • Dense (vector) catches paraphrases, but has no notion of a product code.
  • Full-text catches exact words, but misses other phrasings.
  • Trigram catches misspellings and missing tones, and needs no word segmentation, which PostgreSQL cannot do for Vietnamese.

The three lists are fused with Reciprocal Rank Fusion: each branch contributes weight / (60 + rank). RRF uses ranks, not scores, because a cosine of 0.62 and a ts_rank of 0.0004 are not comparable, while "third place" is.

A cheap model then scores each passage 0–10, in parallel batches of 10. For long passages the reranker sees the lines that best match the question, not just the beginning. Passages under the threshold are dropped; if none remain, the answer is "not found" rather than a guess.

Intent router

IntentWhenHandling
chitchatgreetings, thanksno search
semantic_searcha fact or number present in the documentshybrid search + rerank
structured_queryneeds calculation over many rowsthe model writes DuckDB SQL run on the sheet, plus search
document_summarywhat the documents are aboutper-file summaries
multi_hopcomparing two things in two placessplit into 2–3 sub-questions, search each

Generated SQL must parse as exactly one SELECT before it runs.

Three-layer conversation memory

  1. Working memory: the last 6 turns verbatim, capped at 3000 tokens; over the cap the oldest whole turn is dropped.
  2. Question rewriting: reads history, summary and memories to turn "what you said at the start" into concrete nouns before searching.
  3. Long term: a rolling summary of older turns, and episodic memory that embeds every turn and recalls the old ones similar to the new question.

Every memory layer is best-effort: if it fails, only memory is lost, the answer is still produced.

Prompt

The prompt order is fixed: [system prompt][summary][recalled memories][history][context + original question]. The system prompt is the agent's prompt verbatim, with nothing inserted before it, so the prompt prefix stays identical across turns and OpenAI's prompt cache applies. Context passages are numbered [1][2]…, and each number in the UI opens its source.

The answering model reads the user's original question; the rewritten one is only used for search.

Models and cost

No model names are hard-coded. Logic asks for a tier and models.yaml answers:

TierModelUsed for
cheapgpt-4o-minirewriting, routing, reranking, SQL, summaries
maingpt-4.1-minigenerating the answer
stronga reasoning modeladmin only

Reasoning models are banned from the cheap tier, since their reasoning tokens are billed but invisible; the registry enforces this at startup. Every model call is logged in rag_usage_logs with its purpose and cost. There is a monthly budget, rate limits per cookie and per IP, and a token cap per public question.

Access and security

  • Public chat, no login. Each browser gets a random HttpOnly cookie; the database stores only its sha256. Conversations and memories are always filtered by that key and never leak to another visitor (enforced by a test).
  • Creating agents, uploading, changing settings, viewing costs: admin only. PDF and Office parsers are attack surface, and every ingested file costs embedding money.
  • Model output is rendered by a hand-written Markdown renderer with a limited element set. HTML in an answer is shown as text, never executed.

Debugging

Every answer stores retrieval_debug: intent, rewritten question, hits per branch, rerank scores, executed SQL, and memory state. When an answer is wrong, check in order: was the intent right → what was the question rewritten to → did any branch find the right passage → did the reranker drop it.

The whole pipeline, step by step: Inside RAG Chat: the journey of a question.

Architecture

ingest: file → parse (magic bytes) → chunk by file type → embed → pgvector (HNSW) + tsvector
ask: recall memories → rewrite question → intent router
   → dense + full-text + trigram → RRF → LLM rerank → numbered context → SSE stream
models chosen by tier (cheap / main / strong) in models.yaml, every call logged with its cost