When GraphRAG beats vector RAG on complex retrieval

GraphRAG is a workload-specific retrieval architecture

GraphRAG changes the RAG architecture decision for questions that depend on relationships across the corpus. Conventional RAG splits documents into chunks, embeds them, retrieves a small set of chunks similar to the query, and gives those passages to the model. For “What was our Q3 refund policy?”, that workflow can retrieve the relevant passage directly. For “What are the recurring themes across two years of customer complaints?”, the answer may be spread across many passages that individually look only weakly related to the question.

That difference makes GraphRAG useful for a narrower class of retrieval problems. GraphRAG, introduced in work discussed by Microsoft Research, substitutes or augments isolated-text retrieval with a graph of corpus entities and their relationships. Engineering teams therefore need to identify which queries require graph structure and which can stay on the cheaper, simpler retrieval path. The original Microsoft work and four independent benchmark studies make that decision workload-dependent rather than showing a general graph advantage.

The architectural difference is the ability to connect evidence across passages

The workload distinction starts with what vector similarity can retrieve. When two passages contain facts linked through a shared entity, embedding each chunk independently does not expose that connection as an explicit retrieval path. Similarity search also covers only a small fraction of a corpus, which creates a structural problem for questions about themes or patterns spread across the collection. Chunk boundaries compound the problem because relationships and hierarchy needed for complex reasoning can disappear when documents are divided.

That retrieval problem drives Microsoft Research’s approach. Microsoft Research says baseline RAG “struggles to connect the dots.” The same work says it performs poorly when asked to “holistically understand summarized semantic concepts over large data collections.” The first problem is connecting distributed evidence; the second requires selecting and organizing enough evidence to represent an entire corpus.

GraphRAG addresses both problems by moving substantial semantic processing ahead of the query. During indexing, an LLM first processes every chunk and extracts entities, relationships, and claims. Those elements become a weighted knowledge graph, after which the Leiden algorithm performs community detection, which groups connected graph elements into a hierarchy of topics. GraphRAG then produces a natural-language summary in advance for each community, so the index contains explicit relationships and higher-level summaries for later retrieval.

Those indexed communities change how a query gathers evidence. Relevant communities produce partial answers in the “map” step; those partial answers are ranked and merged during the “reduce” step; and the model generates its answer from the resulting structured context. For a corpus-wide complaint analysis, this process gives the model evidence from multiple relevant parts of the collection and a systematic way to combine it. The query can therefore reach across the corpus without depending entirely on the nearest individual chunks.

Other graph systems use explicit relationships differently. HippoRAG combines a graph with Personalized PageRank, a graph-walk mechanism that ranks relevant nodes by moving through connected parts of the graph, to retrieve passages. Its retrieval process differs from Microsoft GraphRAG’s precomputed community summaries, but both make corpus relationships available when choosing context. The architectural distinction is whether retrieval can follow those relationships when one passage is insufficient.

Those relationships require extra work before they can improve retrieval. GraphRAG moves semantic processing earlier in the pipeline through entity and relationship extraction, graph construction, clustering, and summarization. Any retrieval or reasoning gain consequently brings additional indexing work and operational complexity. The benchmark question is whether that investment buys enough quality for the queries a system actually receives.

Across benchmarks, the graph advantage grows with reasoning depth

Microsoft’s global-query experiments show a workload where that investment can pay off. Microsoft compared GraphRAG head-to-head with naïve RAG on questions requiring understanding of million-token datasets as a whole, and an LLM judged the answers for comprehensiveness, diversity, and empowerment. Microsoft develops GraphRAG and related tooling, so it has a commercial stake in adoption of the approach; its results should be read with that interest visible. The comparison figures appear in the scorecard below alongside independent benchmark results.

The global-query experiment matters because it tests the problem created by limited vector retrieval. A request for corpus-wide themes cannot reliably assume that a few individually similar chunks represent the whole collection. Microsoft also found that GraphRAG’s highest-level summaries consumed up to 97% fewer tokens than processing the source text directly. Precomputing structure can therefore reduce the source material entering this kind of query, even though building that structure carries an indexing cost.

Multi-hop benchmarks test the same retrieval problem at a finer level. MuSiQue, HotpotQA, and 2WikiMultiHopQA require answers that depend on multiple reasoning steps, so finding one relevant passage does not necessarily provide enough evidence. Their Recall@5 results measure whether supporting material appears among the five retrieved results, making retrieval quality visible before generation begins. Generation cannot reason over evidence that retrieval never supplies.

HippoRAG provides a related test with a different graph-retrieval design. It reports up to a 20% accuracy improvement on multi-hop QA while costing 10–20× less and running 6–13× faster than iterative retrieval methods. Those efficiency figures compare HippoRAG with iterative retrieval rather than ordinary vector retrieval, so they answer a different architecture question from a direct graph-versus-vector comparison. They show that exploiting graph relationships can avoid repeatedly sending a model through an expensive sequence of retrieval steps.

A controlled 2025 study from Michigan State and Meta reduces another source of variation. It compared conventional RAG with four GraphRAG families using identical chunking, embeddings, and generation, and no method won universally; graph and conventional retrieval were complementary. Holding those surrounding components fixed makes task type more visible as a factor separating the results. The study’s single-hop and multi-hop outcomes appear in the scorecard.

GraphRAG-Bench, associated with ICLR 2026, makes task type the question directly: “In which scenarios do graph structures provide measurable benefits?” Its task-level results distinguish simple facts from complex reasoning and contextual summarization. That division matters because it tests whether graph structure changes performance as evidence becomes more distributed and interconnected. The scorecard keeps those measurements separate because the studies use different metrics and comparison designs.

The strongest GraphRAG claim depends on the evaluation method

The task pattern also changes how the Microsoft headline win rates should be interpreted because many GraphRAG comparisons use another LLM to evaluate generated answers. An independent audit of LLM judges found systematic position bias: reversing which answer appears first can alter the win rate by more than 30 points. The audit also identified length bias and trial bias, where repeated identical comparisons can produce conflicting judgments. These effects make small differences in subjective answer preference difficult to treat as stable measurements.

The scale of that evaluation problem appears in one audited example. A popular method initially reported a 66.7% win rate, but correction reduced it to about 39%, below the 50% break-even level. A judge-based result can therefore move from apparently convincing superiority to losing more comparisons than it wins. Comprehensiveness margins deserve reference-based checks when architecture decisions depend on them.

Direct retrieval and task-accuracy measurements rely less on that subjective ordering. HippoRAG reports what is characterized as “+20% multi-hop accuracy,” while the multi-hop benchmark results are described as “+15-to-30-point recall jumps,” with those benchmarks measuring whether retrieval finds relevant evidence or whether the final answer is correct. Those measures give stronger support for workload selection than a narrow preference margin because they depend less on an LLM judge ranking two generated responses. The case for graphs is strongest where task-specific measurements support the same type of workload identified by global-query experiments.

The evaluation caveat therefore narrows the architecture claim. It reduces confidence in treating every reported win rate as equivalent evidence, particularly for subjective qualities such as comprehensiveness. Multi-hop accuracy, Recall@5, controlled factual lookup, and task-specific GraphRAG-Bench results give a firmer basis for deciding when graph structure helps. Once the quality claim becomes workload-specific, the additional cost should be evaluated on the same basis.

Graph quality has an indexing bill

That cost begins with the preprocessing needed to create graph relationships. An LLM has to extract entities and relationships across the corpus, and one analysis estimates roughly $48 to construct the index for a moderate corpus using GPT-4o as the pricing reference. Conventional vector indexing avoids that semantic-extraction pipeline. When simple-fact benchmarks show little graph benefit, applying the graph cost across every corpus has weak quality justification for workloads dominated by those questions.

Microsoft’s LazyGraphRAG changes those economics by deferring extraction until query time. Microsoft claims its cost is around 0.1% of the original approach; because Microsoft develops these GraphRAG approaches and benefits commercially from their adoption, that cost claim carries the same vendor interest as its other GraphRAG claims. Deferral changes when the work occurs, so teams still need to account for the resulting query path alongside quality requirements. Lower preprocessing cost changes the trade-off without removing it.

The trade-off is clearest for “what’s the phone number on page 3.” The requested evidence is directly located and needs no relationship traversal, community synthesis, or corpus-wide reasoning. Graph extraction adds processing without supplying a capability that query requires, while the controlled study and GraphRAG-Bench results provide no factual-lookup advantage to offset the extra work. At sufficient query volume or corpus size, indexing cost, latency, and operational complexity can outweigh a marginal quality change.

That simple lookup shows why cost changes the architecture decision itself. If graph retrieval were reliably superior on every query, its added expense could be evaluated against a universal quality improvement. The measured improvement instead depends on the task, so the expense should depend on which path the task needs. Query-time selection follows from that asymmetry.

Route queries by the evidence they require

A retrieval router can preserve the strengths of both paths by choosing a retrieval method for each query. Multi-hop, global, and sensemaking questions are the clearest candidates for graph retrieval, especially when the user needs comprehensive answers representing multiple perspectives. Richly interconnected corpora such as research libraries, case files, incident histories, and knowledge bases create relationships that graph retrieval can use. These workloads align with the tasks where the benchmarks show larger gains.

Single-fact lookup points toward conventional chunks because its evidence can often be retrieved directly. Small or flat corpora also provide fewer useful relationships to encode, while teams with tight constraints on cost, latency, or operational simplicity have additional reasons to keep the vector path. Conventional text-chunk retrieval remains a practical architecture for the workload it handles well. Graph capability can then be invoked where the query requires connections beyond one directly relevant passage.

A hybrid system can make that choice per query or fuse evidence from both retrieval methods. Across the systematic studies, combining graph and chunk retrieval consistently beats either approach alone. Fusion can preserve directly relevant passages while adding graph-selected evidence where relationships matter, while routing can avoid invoking the extra graph path when direct lookup is sufficient. The research supports this architectural principle without establishing a universal classifier or threshold for implementing the router.

Without a universal threshold, each team still faces a workload-specific engineering problem: determine from its own query distribution which requests require cross-passage reasoning or global synthesis. A support knowledge base dominated by isolated policy and contact-detail questions has different economics from an incident archive used to trace recurring causes across systems and time. Query composition determines how often the extra graph structure can earn its cost. Architecture should follow that composition.

The bottom line

GraphRAG is not a replacement for vector RAG. It is an additional retrieval architecture for workloads where answers depend on relationships across documents, entities, or the corpus as a whole. The benchmark evidence points in the same direction: the value of graph structure tends to increase as the evidence becomes more distributed and the reasoning becomes more complex.

For decision-makers, that makes query composition more important than headline benchmark gains. An organization dominated by factual lookup may gain little from paying for graph construction, maintenance, and additional operational complexity. One that regularly asks multi-hop, investigative, or corpus-wide questions has a stronger case for that investment.

The practical architecture is therefore often hybrid. Keep conventional retrieval for questions it handles efficiently, and route relationship-heavy or global questions toward graph retrieval when the expected quality gain justifies the cost. Evaluate both paths against representative business queries using retrieval accuracy, answer quality, latency, and total cost rather than relying on a single benchmark.

The decision is ultimately less about choosing GraphRAG or vector RAG than deciding where each belongs. Graph structure earns its place when relationships are part of the information the business needs to retrieve.