Machine Learning Q&A: Teaching RAG to Read Math
Try it: Q&A about Machine Learning
1. What the app does
It's a chatbot for Machine Learning cơ bản, a 422-page Vietnamese machine learning textbook. You ask a question, it answers with citations, and most importantly, the formulas come out exactly as they appear in the book.
To make that work, the math has to survive every step:
- Parse: convert the PDF to Markdown, formulas to LaTeX, tables to Markdown tables.
- Chunk: split by chapter and section, never through an equation or table (870 chunks).
- Retrieve: find the most relevant chunks, with their sources.
- Answer: gpt-4o replies in Vietnamese, writes math in LaTeX, and cites its sources.
- Render: the interface turns LaTeX into real math notation.
2. Why I built it
First, I wanted to take on a problem the usual approach can't handle. Most RAG tutorials quietly assume your documents are plain text. But the documents people most need help with, like textbooks, papers and exam sheets, are full of math. Ask an agent or LLM about them and the formulas either vanish or come back in the wrong format. I wanted a system that treats formulas as the most important content, not something to throw away.
The second reason is bigger. I'm training a model to solve math problems. It needs a lot of clean math data, and eventually it will need to look up definitions and theorems while it works. This project is the first building block: the same pipeline that runs the chatbot also turns math PDFs into clean LaTeX.
3. The story behind it
The book is packed with linear algebra, probability and optimization. The first time I asked about it through a regular "chat with your PDF" tool, the answers sounded confident, but the math was wrong, missing, or broken.
At first I blamed the model. Digging deeper, I found the real problem was in the very first step. PDF text extractors read characters by their position on the page, so any nicely typeset formula falls apart. Here's the mean formula (3.35) from page 60 after a standard extractor gets through with it:
N
1 X 1
x̄ = xn = x1, (3.35)
N n=1 NAnd here's what the model actually needs to see:
\bar{x} = \frac{1}{N}\sum_{n=1}^{N} x_n = \frac{1}{N}\mathbf{x}\mathbf{1}The squared norm ‖z‖²₂ turns into kzk22. Fractions, sums, matrices and subscripts all get flattened into meaningless text. At that point, no model can help, because the original formula is simply gone. That was the biggest lesson for me: to get the right answer, you fix the root cause, not the prompt. So I rebuilt the whole pipeline around LaTeX.
4. How it helps
- Students working through the book can ask "what does formula 3.35 mean?" and get an answer in the same notation as their textbook, with a pointer back to the source so they can keep reading.
- People building RAG on technical documents can reuse the approach: fix PDF parsing first, keep formulas whole when chunking, and render them on the way out.
- Me, because this is the data and retrieval layer for the math-solving model I'm training.
5. My decisions and tradeoffs
I weighed and made each of these calls myself, and every one of them came at a cost.
| Decision | Why | Tradeoff |
|---|---|---|
| Convert the PDF to Markdown + LaTeX instead of extracting plain text | Formulas stay exact and searchable | Slower parsing |
| Chunk by section and never split an equation or table | Half a formula is worse than none | Uneven chunk sizes, some longer than ideal |
| Retrieve text, not page images | Cheaper search, precise citations | Figures and plots are lost; only captions remain |
| Have the model answer in LaTeX and render it in the UI | Answers match the book's notation | The prompt and the front end must agree on one LaTeX format |
| Use gpt-4o for answers | Handles Vietnamese well and reads LaTeX reliably | API cost for every question |
What's next
There's still a lot I want to do better. I want to bring figures back with a multimodal step. I want to add a plain-language description to each chunk, since a question like "integration by parts" matches prose far better than raw LaTeX. Above all, I want to connect this pipeline to the math-solving model, so it can look up the right theorem before it starts solving.
Every time I fix a small failure, a broken formula here, a table cut in half there, I understand a little more about how machines actually read documents. That's my favorite part of this project, and it's why I want to keep pushing it further.