Skip to content
← Blog

Inside RAG Chat: the journey of a question, from uploaded file to answer

A minimal RAG needs three steps: cut documents into chunks, embed them, take the top-k chunks closest to the question and hand them to an LLM. That version works for a demo and breaks the moment real users arrive:

  • people type "hoc phi" without Vietnamese tones;
  • people ask "total Q3 revenue", and the model adds up whatever numbers happen to sit in the three chunks it found;
  • people ask "and the other class?", which no embedding can understand without the history.

RAG Chat is what came out of fixing those failures one by one. This post walks through the whole pipeline in the order the data flows. For each step: what it does, its default numbers, and, where there is one, the real bug that made it exist.

Two separate flows

Flow A, ingestion runs in the background in rag-worker, once per file Flow B, answering runs in the api on every chat turn, streamed to the browser over SSE

They never share a process. Parsing and embedding a long PDF can take minutes; doing it inside the upload request would pin a web worker that should be serving someone's chat.

One convention runs through everything: the code never names a model. Logic asks for a tier, and config/models.yaml answers:

TierCurrent modelUsed for
cheapgpt-4o-minirewriting, routing, reranking, SQL, sub-questions, summaries, titles
maingpt-4.1-minigenerating the answer
embeddingtext-embedding-3-small, 1536 dimschunks, questions, memory

Every single model call is logged in rag_usage_logs with its purpose and cost.


Flow A: a file enters the store

upload ─▶ store original ─▶ Redis queue ─▶ rag-worker

  1. PARSE detect type from magic bytes
  2. CHUNK split according to file type
  3. CONTEXTUALIZE (optional) prepend one context sentence [cheap]
  4. PARENT–CHILD (optional) group child chunks into parents
  5. EMBED cached by content hash [embedding]
  6. INDEX one transaction: delete old, write chunks, write vectors
  7. FILE SUMMARY one paragraph for the whole document [cheap]

1. Parse: never trust the file name

The file type comes from its magic bytes, the first few bytes of the content, checked against the extension. A .pdf that is really something else is rejected. PDF and Office parsers are attack surface, so only the admin can upload.

2. Chunk: one strategy per document type

Prose (PDF, DOCX, HTML, Markdown, TXT) is split by headings when there are any, otherwise by sentences, into 700–800-token chunks with 100 tokens of overlap. Two small-sounding details once hurt quality without raising a single error:

Overlap used to be zero and nobody noticed. The first version split recursively only until each piece fit chunk_size. With 700/100, a 700-token paragraph was one indivisible piece, and overlap carried whole pieces, so the real overlap was zero. The fix splits down to min(chunk_size, chunk_overlap) first and packs back up. Measured overlap afterwards is 82–83 tokens. It is never exactly 100, because overlap carries whole sentences: half a repeated sentence means nothing.

A lone heading is garbage. A 3-token chunk holding only ## Costs matches no query, and the chunk with the real content loses its title. A heading now always travels with the first chunk of its section, and a section with a heading but no content is dropped.

Spreadsheets (XLSX, XLS, CSV) get one summary chunk per sheet (columns, types, statistics) plus row-group chunks with the header repeated in each. One bug here is worth retelling:

An FAQ spreadsheet contained the question "class sizes". A user asked exactly that. The system answered: not found.

Several Q&A pairs had been packed into one chunk. Its embedding was diluted by the other questions, and the reranker only read the start of the chunk, so it never saw the matching row. The system now tells two kinds of table apart. If more than half the rows have a cell of 20 words or more, it is a prose table and every row becomes its own chunk. Numeric tables are still packed at about 500 tokens per chunk.

3–4. Two options, off by default

  • Contextual retrieval: a cheap model writes one sentence placing the chunk within the whole document, prepended before embedding.
  • Parent–child: search small chunks for precision, but let the model read a ~2500-token parent for context. Parents never cross a top-level section and never merge table rows.

Both are off by default because both cost more: the first is one model call per chunk, the second makes the prompt several times longer.

5–6. Embed and index

Embeddings are cached by content hash, so re-indexing an almost unchanged document costs almost nothing. Writing happens in one transaction: delete the old chunks, write rag_chunks, write the vectors. The store is never half old, half new.

Each chunk is stored twice:

  • as a vector in pgvector with an HNSW index;
  • as a tsvector for full-text search, with Vietnamese tones stripped, so "hoc phi" matches "học phí".

One technical limit shaped the whole schema: pgvector cannot build an HNSW index on vector above 2000 dimensions. Every width therefore has its own table (rag_emb_v1536, rag_emb_h3072…), and 3072-dimension models use halfvec, the half-precision type, which can be indexed.

7. Summarise the file

Finally a cheap model writes a one-paragraph summary of the whole document, for questions no top-k can answer, like "what is this document about?".


Flow B: one question through 12 steps

Say a user has asked a few things about tuition, then types:

and grade 11?

Here is everything that happens to it.

  1. CHECKS budget · agent has a system prompt · read history ── memory ──
  2. LAYER 1 trim history to 3000 tokens
  3. RECALL find old turns similar to the question [embedding] ── understand ──
  4. REWRITE standalone question + keywords [cheap]
  5. ROUTER classify the intent [cheap] ── search ──
  6. PER INTENT SQL · sub-questions · file summaries [cheap]
  7. HYBRID SEARCH dense + full-text + trigram ─▶ RRF
  8. RERANK score 0–10, keep 6 [cheap]
  9. EXPAND PARENTS (if parent–child is on) ── answer ──
  10. ASSEMBLE numbered [1][2]…
  11. PROMPT fixed order
  12. GENERATE stream token by token [main] ── afterwards ──
  13. STORE citations · debug · title · summary · memory

Step 0: checks before spending money

Is there budget left this month? Does the agent have a system prompt? An agent without one refuses to answer; there is no fallback prompt. Then the user's message is stored and the last 6 turns are read.

Steps 1–2: memory

Layer 1, working memory: recent turns verbatim, capped at 3000 tokens. Over the cap, the oldest whole turn is dropped, never half a turn. The turn right before the question is always kept, because a pronoun in the new question almost certainly points at it.

Episodic recall: every finished Q&A turn is embedded and stored. The new question is compared with turns that have already left the layer-1 window, and up to 3 with cosine similarity of 0.35 or more are recalled. That is how "the tuition you mentioned at the start" still resolves on turn 20 without resending 20 turns.

This runs before the rewrite, so the rewrite can read the memories.

Step 3: rewriting the question

"and grade 11?" embeds to a vector that resembles nothing. A cheap model reads the history, the rolling summary and the recalled memories, and returns two things:

  • question: a standalone sentence, e.g. "What is the tuition for grade 11?";
  • keywords: 3–8 terms and synonyms that a matching passage would literally contain.

It runs on the first question too, because people rarely use the document's own words: "class size of 10A1" has to find a table whose column is called "No. of students".

Each version of the question goes somewhere different:

StepUses
vector search, trigram, rerankthe rewritten question
full-textquestion + keywords
answering modelthe user's original words, with history

The answering model reads the original. The rewrite is a search tool, and it can get nuance wrong.

Step 4: the router

A cheap model classifies the intent:

IntentWhenHandling
chitchatgreetings, thanks, "what can you do"no search
semantic_searcha fact or number present in the documentshybrid search + rerank
structured_querymust compute over many rowsSQL on the sheet + search
document_summarywhat the documents are about, which files existper-file summaries + search
multi_hopcomparing two things in two placessplit into sub-questions

chitchat skips the whole search: no embedding, no search, no 3000 tokens of context. On a public chat where many messages are "hello", that is the biggest single saving.

The hardest boundary is between semantic_search and structured_query. "How many students are in each class?" contains "how many" but is semantic_search, because the numbers already sit in the document. Only summing, counting, averaging or ranking across rows is structured_query. The router's prompt has examples for exactly that boundary.

Step 5: intent-specific work

structured_query → SQL. "Total Q3 revenue by region" is not a search problem. Treated as one, the model sums whatever numbers landed in three chunks and produces an answer that is wrong but looks right. Instead, the sheet is loaded into DuckDB, a cheap model writes one SELECT, the system runs it and answers from the result.

Model-written SQL is still treated as hostile input:

  • the model sees only this document's tables and columns;
  • the statement must parse as exactly one SELECT, checked with DuckDB's own parser, not a regex;
  • every table and column name must be on the allow-list;
  • the connection is read-only, over an in-memory database built from this document alone;
  • a LIMIT is enforced, and the query has a time limit.

multi_hop → sub-questions. The question is split into 2–3 sub-questions; each is searched and reranked on its own, then merged. Every sub-question keeps at least its best result, so one cannot crowd out the other.

document_summary → per-file summaries written in step 7 of flow A, placed ahead of the retrieved passages.

Step 6: hybrid search, three branches

Each branch fails exactly where another works:

BranchGood atBad atCandidates
dense (vector)paraphrases: "operating costs" vs "operating expenses"product codes, names30
full-textexact wordsother phrasings30
trigramtypos, missing tones, no word segmentation neededmeaning20

Trigram matters especially for Vietnamese: PostgreSQL cannot segment Vietnamese words, and trigram does not need to.

The three lists are fused with Reciprocal Rank Fusion:

RRF(d)=∑b∈brancheswb60+rankb(d)\mathrm{RRF}(d) = \sum_{b \in \text{branches}} \frac{w_b}{60 + \mathrm{rank}_b(d)}

RRF uses ranks, not scores. A cosine of 0.62 and a ts_rank of 0.0004 cannot be compared, but "third place here" and "third place there" can. The weights wbw_b come from one knob, semanticBalance (default 0.5): 0 is keywords only, 1 is vectors only.

Step 7: reranking, where quality is decided

RRF says how often a passage surfaced. It does not say whether it answers the question. The two disagree often, which makes reranking the biggest quality lever after hybrid search itself.

OpenAI has no rerank endpoint, so the reranker is a cheap model acting as judge:

  • up to 30 candidates, in batches of 10, with batches running in parallel, because any latency here is added straight to the user's wait;
  • only about 350 tokens per passage. For long passages, those are the lines that best match the question plus the first line (the table or section name), not the opening;
  • the model returns 0–10 scores under a strict JSON schema;
  • the top 6 are kept, anything under 0.3 (3/10) is dropped. If nothing remains, the answer is "not found" rather than a guess.

Instructions and schema sit at the front of the prompt, identical every time, so the prompt cache applies.

Choosing the reranker model was an expensive lesson. The cheap tier used to be gpt-4.1-nano, which is cheaper. For the same passage, one that answers the question directly, at temperature 0, nano scored 10, 1, 10, 10, 10 when judged alone, and 5/10 inside a batch of 10. gpt-4o-mini scored 10 all five times and 9–10 in the batch. A low rerank score means the right passage is dropped and the user gets "not found", so paying about 1.5× more for the auxiliary steps is worth it.

Steps 8–9: assembling the context

With parent–child on, child chunks are swapped for their parents. Passages are then numbered in document order, not score order. Score order reads like shuffled notes; document order reads like a document, and the model summarises it better.

The context has its own token budget: about half the window for retrieved passages, the rest split between the system prompt, the summary, history and a reserve for the answer. When it does not fit, the lowest-ranked passage is dropped whole, never cut in the middle. Half a passage produces a citation pointing at text the reader cannot find.

Step 10: a prompt in fixed order

[system prompt verbatim][conversation summary][recalled memories][layer-1 history][context + ORIGINAL question]

The order is deliberate: what stays still comes first, what changes comes last. OpenAI's prompt cache only applies when the first 1024 tokens are identical across calls. The agent's system prompt is therefore placed verbatim at the very start, with nothing inserted before it, and a test guards against anyone inserting something there by accident. The consequence: caching only kicks in when the agent's prompt is at least 1024 tokens long.

Step 11: streaming

The main model generates the answer and each token is pushed to the browser over Server-Sent Events. Along the way the browser also receives retrieval_step events ("Searching relevant passages", "Computing directly on the table"…), so the user sees what the system is doing instead of a spinner.

The answer is rendered by a hand-written Markdown renderer supporting a limited set of elements. This text is generated from documents an outsider may have influenced, so any HTML in it is shown, never executed. Every [2] in the answer is a button opening the exact source passage.

Step 12: after the answer

The answer is stored with its citations and retrieval_debug. Then three side jobs:

  • a title for the conversation, first turn only;
  • a rolling summary once the conversation passes 6 turns, folding only the new messages into the previous summary (at most 400 tokens) instead of summarising from scratch;
  • embedding the finished turn into episodic memory.

All three are best-effort. If they fail, only memory is lost; the answer has already been streamed and saved. Finally the browser receives a sources event (the citations) and a usage event (tokens in, tokens out, cached tokens).


How many model calls one chat turn makes

CallCountNotes
embedding2–3question for recall, question for search, finished turn for memory (cached)
cheap3–5rewrite, router, rerank (once per 10 passages), maybe SQL or sub-questions
cheap (occasionally)0–2title, rolling summary
main1the answer

The expensive model is called once. Every other bit of "thinking" goes to the cheap tier, and the cheap tier is banned from using reasoning models, because reasoning tokens are billed but invisible. The registry enforces this at startup: a wrong config and the API does not start.

Strangers sharing one system

Chat is public, with no login. Each browser gets a random HttpOnly rag_guest cookie, and the database stores only its sha256. Conversations, messages and memories are tied to that key and every query filters on it. A test guarantees one visitor's memories can never end up in another visitor's prompt.

With no accounts, cost limits live at the system level: a monthly budget, rate limits per cookie and per IP, a token cap per question, and the strong tier reserved for the admin.

When an answer is wrong

Every answer carries retrieval_debug, visible to the admin in the Sources panel. The usual order of checks:

  1. intent: did the router classify it correctly?
  2. query, keywords: what was the question rewritten into?
  3. branchHits: did any branch find the right passage?
  4. hits after rerank: was the right passage dropped?

If the right passage is missing from branchHits, the problem is chunking or the rewritten question. If it is there but not kept, the problem is the reranker or the threshold. Those two questions pinpointed the cause of almost every bug in this post.

Looking back

No step in this pipeline is complicated on its own. The hard part is that each one fixes a very specific kind of failure, and many of those failures raise no error at all: zero overlap, an FAQ row hidden inside a chunk, an erratic reranker, a prompt under 1024 tokens that means the cache never applies. They only show up as "not found", or on the bill.

The next step is a knowledge graph for the questions that passage search cannot reach. The plan is in GraphRAG: a plan for adding a knowledge graph to RAG Chat.