Skip to content
← Blog

GraphRAG: a plan for adding a knowledge graph to RAG Chat

What RAG Chat does today

RAG Chat searches by passage: three branches (dense, full-text, trigram) fused with RRF, an LLM reranker, and an intent router. The router already has multi_hop (splitting into sub-questions) and document_summary (reading per-file summaries).

That works when the answer sits in a few passages that resemble the question. It is weak on two kinds of questions.

Questions that follow relations. "Which projects share a supplier with project X?" The passage about X's supplier and the passage about some other project Y may be in two different documents, and the second does not resemble the question at all. Today's multi_hop splits the question before searching, so it has to guess the second hop in advance. Here the second hop depends on the result of the first.

Corpus-wide questions. "What are these documents broadly about, and how do they relate?" No top-k set of passages contains the answer. document_summary summarises each file on its own and never sees the links between files.

The idea

During ingestion, extract entities (people, organisations, products, concepts…) and the relations between them to build a graph. At question time, instead of only looking for passages that resemble the question, find the entities the question mentions and follow the edges to neighbouring entities. The passages behind those edges become context, even when they look nothing like the question.

Two main references:

  • Microsoft GraphRAG (Edge et al., 2024, From Local to Global): build the graph with an LLM, partition it into communities with the Leiden algorithm, and write a summary for each community. Corpus-wide questions are answered from those summaries.
  • LightRAG: lighter, retrieving at two levels (specific entities and broad themes), and updating the graph per document instead of rebuilding it.

Intended design

Principle: add, don't replace. The graph becomes a new retrieval branch, going through the same RRF, reranking, citations and cost logging as the existing branches.

At ingestion

PARSE ─▶ CHUNK ─▶ EMBED ─▶ INDEX (unchanged) └─▶ EXTRACT ENTITIES + RELATIONS [cheap, per chunk] ─▶ MERGE DUPLICATE ENTITIES ─▶ WRITE NODES + EDGES (each edge points to its source chunk) ─▶ (later) COMMUNITIES + COMMUNITY SUMMARIES

  1. Extraction. The cheap tier reads each chunk and returns JSON under a fixed schema: a list of entities (name, type, short description) and a list of relations (source, target, relation type, description). Output that breaks the schema is dropped, not guessed at.
  2. Merging duplicates. "ABC Company", "ABC" and "abc corp" must be one node. First pass: normalise names (strip tones, lowercase), compare embeddings of name plus description, and merge only when the type matches and similarity is high. A wrong merge damages the graph more than a missed one, so the threshold will be strict.
  3. Provenance on every edge. Each edge stores the id of the chunk it came from. This is the most important part: it means an answer that went through the graph can still cite the original passage, exactly as today.
  4. Community summaries come later, once local search works well.

Where to store it: PostgreSQL first, Neo4j if needed

The first choice is two new tables in PostgreSQL (nodes and edges), not a second database:

  • same transaction as rag_chunks: re-indexing a document deletes and rewrites its nodes and edges in one go, so they never drift apart;
  • 1–2 hops are doable with a recursive CTE;
  • entity names can use the trigram and full-text indexes that already exist;
  • one less service to run, back up and secure.

If heavy graph algorithms turn out to be needed later (many hops, communities over a large graph), that is when Neo4j gets considered.

At question time

  1. The router gets an intent for corpus-wide questions; multi_hop questions also use the graph branch.
  2. Entity linking. Find nodes matching entities in the rewritten question (names by trigram, meaning by embedding).
  3. Expand 1–2 hops from those nodes, with a cap on edges and a preference for edges seen in several chunks.
  4. A fourth RRF branch. The source chunks of the edges found become one more ranked list, fused with dense, full-text and trigram. Everything after stays the same: reranking, the threshold, numbered context.
  5. Debug output. Which entities were recognised and which edges were walked go into retrieval_debug next to the existing fields, so debugging works the way it does now.

Cost and switching it on

Extraction costs one extra LLM call per chunk at ingestion, which is the most expensive part. So:

  • the graph is a per-agent option, off by default, like contextualRetrieval and parentChild today;
  • every extraction call is logged in rag_usage_logs with its own purpose and counts against the monthly budget like any other call;
  • at chat time, entity linking and graph walks are SQL queries, adding no model calls beyond the existing ones.

Measuring before believing

It does not get switched on for everyone just because it sounds good. The evaluation plan:

  1. Build a question set in three groups: single-fact lookups, multi-hop relations, and corpus-wide questions, each with the correct passages marked.
  2. Run the same set with the graph on and off, over the same documents.
  3. Compare: share of correct passages reaching the context, answer correctness, cost and latency per turn.
  4. Keep it only if the multi-hop and corpus-wide groups clearly improve and single-fact lookups do not get worse.

Risks already in view

  • Bad extraction means a bad graph, and that error is harder to spot than a wrongly retrieved passage. The graph must be viewable in the admin pages.
  • Duplicate or wrongly merged entities break relations or connect the wrong things.
  • Updates: editing a document must delete exactly the nodes and edges that came only from it, keeping nodes other documents also mention.
  • Prompt injection into the graph: documents may contain instructions aimed at the extraction model. Entity descriptions must be treated as data, never as instructions, the same way passages are handled now.

Progress will be posted on the GraphRAG project page.