When GraphRAG earns its extra complexity
Vector search finds similar passages. A graph can connect people, events, and dependencies across many passages. The useful question is when you need both.
After working on production retrieval systems, I stopped asking which embedding model would fix every bad answer. A better embedding can improve search. It cannot create a relationship that the system never represented.
Standard retrieval augmented generation is good at finding passages that look similar to a question. It struggles when the answer depends on facts scattered across many documents: who owns a system, what depends on it, which decision came first, and how one event affected another.
A knowledge graph can make those relationships explicit. GraphRAG combines that structure with text retrieval. It can be a large improvement for the right questions, but it also adds indexing work, cost, and another system that can be wrong.
What vector search does well
The usual RAG pipeline splits documents into chunks, turns those chunks into embeddings, and searches for vectors close to the user’s question. It is a strong baseline because it works across messy language without requiring a carefully designed database first.
Ask about a policy whose wording is close to the source paragraph and vector search will often find it. Ask for a product feature described in one support article and it usually performs well. Hybrid search, which combines semantic and keyword signals, makes the baseline stronger again.
The limit appears when similarity is not the same as connection. A passage about a supplier may not use the same language as a passage about the system that supplier supports. A vector index can retrieve either passage, but it does not naturally know that both belong to one chain of evidence.
What the graph adds
A graph stores entities and relationships. A person owns a service. A service depends on a database. A decision replaced an older decision. Those edges give retrieval a path it can follow instead of asking similarity to do all the reasoning.
Microsoft Research introduced GraphRAG for questions over private text collections where users need an overview of a whole dataset. Its pipeline extracts a graph, groups related parts into communities, and produces summaries that can help answer broad questions. That is a different job from finding the single passage closest to a query.
This is where a graph earns its keep:
- The answer needs several documents, not one good chunk.
- Entity names change or have aliases.
- Ownership, sequence, or dependency matters.
- Users ask broad questions about themes across a collection.
- The product needs to show a traceable path from an answer to connected evidence.
The last point is useful, but it needs careful wording. A graph can make the retrieval path easier to inspect. It does not guarantee that every extracted relationship is true. The extraction model can still create a bad edge, and a summary can still leave out an important fact.
What it costs
GraphRAG is not a free upgrade to every vector database. The system has to identify entities, create relationships, group the graph, and often write summaries before the first user asks a question. That indexing stage can take much more time and model usage than normal chunking and embedding.
Updates are harder too. If one source document changes, the system may need to revisit the edges and summaries built from it. A simple vector index can often replace a few chunks. A graph pipeline needs a clear plan for what becomes stale.
There is also an evaluation problem. A plausible graph looks impressive even when the important relationships are missing. Test it with real multi-step questions and inspect the evidence path, not only the final prose.
The architecture I would start with
I would not begin a new product with a graph unless its questions clearly need one. Start with clean source documents, good chunking, metadata filters, and hybrid retrieval. Build a small evaluation set from the questions users actually ask.
Look at the failures. If the missing answers consistently require joining entities, following dependencies, or summarising the whole collection, add a graph for that part of the problem. Keep the original text index. The two systems are useful together: text search gives broad recall, while graph paths provide structure for the questions that need it.
The important shift is not that every RAG system should become GraphRAG. It is that retrieval should represent the shape of the question. Similarity works for similar language. Relationships work for connected facts. Production systems often need both, but only after the simpler system has shown where it falls short.