Experiment: Graph RAG outperforms standard RAG on multi-hop questions, but frontier models lead

Over the last year, I’ve lost count of the number of conversations I’ve had about AI agents, RAG, semantic layers, and knowledge graphs. Whether you’re reading analyst reports, vendor architectures, or technical blogs, knowledge graphs increasingly show up as a key ingredient for helping AI systems understand not just information, but the relationships between pieces of information.

I had no reason to doubt that. But I wanted to understand it a bit more concretely.

Specifically, I wanted to know whether adding a knowledge graph would materially improve the quality of answers for a retrieval use case, and whether the additional engineering effort was justified when compared with more traditional RAG approaches (or simply dumping the source material directly into the context window of a frontier model).

So, I built four different approaches, ran them against the same documents and questions, and compared the results.

The outcome was clear: the frontier model produced the strongest answers overall. At first glance, that might sound like a disappointing result for Graph RAG. It wasn’t. In fact, the more I looked at the outputs, the more interesting the trade-offs became. Graph RAG consistently outperformed traditional RAG on some types of questions, struggled on others, and highlighted something that is often missing from the discussion: the value of a graph depends heavily on the problem you’re trying to solve.

This article isn’t really about which approach won. It’s about understanding when the additional complexity of Graph RAG is justified, and when it probably isn’t.

The Experiment

Before diving into the results, it’s worth explaining what I was trying to achieve.

This wasn’t intended to be a rigorous benchmark. The dataset was deliberately small, consisting of two documents (I used two recently published articles by Anthropic on the topic of AI Security) and a set of evaluation questions. My goal wasn’t to identify a universally superior architecture. I wanted to understand how different retrieval approaches behave when given access to the same information, and more specifically, whether the additional complexity of a knowledge graph translates into better answers. And in the process, I also wanted to gain some hands-on experience of running RAG architectures, knowledge graphs and SLMs on my own personal laptop.

To keep the comparison as fair as possible, I used the same core components across all retrieval-based systems. I used Microsoft’s Phi-4 language model (14 billion parameters) for the answer-generation model and ran it locally on my Thinkpad through Ollama. I used the all-MiniLM-L6-v2 embedding model (which converts text into numerical vectors that can be compared for similarity) for powering semantic search. For the knowledge graph and graph-based retrievals, I used Neo4j to store and query entities and relationships extracted from the source material.

For comparison, I also included a frontier model approach using Claude Haiku 4.5 (which, to note, is not one of Anthropic’ s top-tier models like Claude Opus or Claude Sonnet, but is advertised as a fast, cost-efficient small model), where both source documents were provided directly to the model without any retrieval layer. This allowed me to compare retrieval-based approaches against a large-context model when the entire corpus comfortably fits within the available context window.

I should also mention that I hadn’t done any of this before. Throughout the project, I used Claude as a collaborator at every stage — brainstorming the overall approach, coming up with a detailed list of steps, and guiding me through all the installations on my personal laptop. Claude also wrote the multiple Python programmes involved, acted as my personal instructor to help me understand the concepts and the individual pieces of code, and helped with debugging and root cause analysis when things got stuck or took an unexpected turn.

The four systems are summarised below.

System A — Plain RAG

This is the architecture most people think of when they hear the term RAG (Retrieval-Augmented Generation). Documents are broken into chunks and converted into vector embeddings. When a user asks a question, the system retrieves the most relevant chunks and provides them to the language model as context. The model then generates an answer using only the retrieved passages. This served as the baseline against which all other approaches were compared.

Under the Hood: I used LangChain’s RecursiveCharacterTextSplitter (which attempts to preserve paragraph and sentence boundaries where possible) to split the documents into 500-character chunks with a 50-character overlap. Each chunk was then converted into a 384-dimensional embedding using the all-MiniLM-L6-v2 model and stored in memory. At query time, the question was embedded using the same model and cosine similarity was calculated against all chunk vectors using NumPy. The five most relevant chunks were retrieved and supplied to Phi-4 (running locally via Ollama) along with the original question as context

System B — Graph Only

Where System A treats documents as collections of passages, this approach treats them as collections of facts. Rather than retrieving text, the system retrieves structured knowledge — concepts and the relationships between them — stored in a graph database.

I built the graph by extracting entity-relationship triples from the same source documents using Phi-4, then storing the entities and relationships in Neo4j as a structured graph. Each concept node was also given a 384-dimensional vector embedding, allowing semantic search to find the most relevant entry points into the graph at query time. Instead of retrieving passages, the system embeds the question, finds the most similar nodes via vector search, and retrieves those concepts along with their relationships as context for the LLM.

In theory, this should help uncover connections that traditional vector search might miss, particularly when information is spread across multiple sections or documents. In practice, I found this to have a significant limitation. While the graph can tell the LLM which concepts and relationships are connected and how, without the original text, the LLM lacked the detail needed to answer accurately, and filled the gap with hallucination.

Under the Hood: I used Phi-4 to extract knowledge triples from each document chunk in the form of subject–relationship–object. Initially I let the model choose its own relationship types freely, which produced an explosion of one-off labels that were too inconsistent to be useful for graph traversal. I switched to a constrained vocabulary of pre-defined relationship types, which brought the graph under control and made the relationships consistent enough to reason over. Each triple was written to Neo4j with each concept node also given a vector embedding using all-MiniLM-L6-v2, so it could be found by semantic search at query time.

At query time, I embedded the question using the same model and used Neo4j’s vector index to find the eight most semantically similar nodes. From those anchor nodes, I traversed the graph two hops outward, collecting the surrounding relationship network. This subgraph (concepts and their connections, but no original document text) was then handed to Phi-4 as context.

System C — Graph RAG

This approach combines the strengths of both previous systems. Rather than replacing document retrieval with a graph, the graph is used to provide additional structure and context. The language model receives both the relevant passages from the source documents and the relationships extracted from the knowledge graph.

Conceptually, this was the system I expected to perform best from the outset. The source text provides grounding and factual detail, while the graph helps expose connections between concepts that may not be obvious from isolated chunks of source text. As the results later show, this combination generally outperformed both Plain RAG and Graph-Only retrieval.

Under the Hood: The retrieval process runs two pipelines in parallel. On one side, vector search retrieves the five most relevant document chunks, providing the grounding text. On the other, the graph retrieval process finds the eight most semantically similar concept nodes and traverses two hops outward, collecting the surrounding relationship network.

Both are combined into a single prompt supplied to Phi-4, with the model instructed to use both sources when generating its response and to cite source material where appropriate. The graph contributed structured relational context layered on top of the raw passages rather than replacing them.

System D — Full Context Frontier Model

The final approach removed retrieval entirely. Rather than searching for relevant information, both source documents were provided directly to Claude Haiku and the model was asked to answer the same questions using the complete corpus as context. This effectively eliminates retrieval errors because no information needs to be selected or ranked before the answer is generated. It also highlights an important caveat when comparing retrieval architectures with modern frontier models: if the entire knowledge base fits comfortably inside the context window, retrieval may not provide much benefit at all.

Under the Hood: Both source documents were provided in full to Claude Haiku with no chunking, vector search, graph traversal, or retrieval layer of any kind. The model was prompted directly with the complete document corpus alongside the evaluation question, and generated its answer using the entire source set as context.

This approach performed best overall, but only because the document corpus was small enough to fit comfortably within the available context window. At greater scale (hundreds of documents rather than two), this approach becomes impractical, and retrieval architectures become necessary again.

Evaluation Approach

Each system was asked the same set of questions against the same source documents.

Responses were evaluated across four dimensions:

Accuracy: is the answer factually correct? Completeness: Did it cover all relevant aspects of the question? Reasoning: Did it connect information logically and coherently? Provenance: Could claims be traced back to the source material?

The questions themselves were designed to test different aspects of retrieval and reasoning:

Q1: Multi-hop reasoning across multiple documents. Q2: Cross-document synthesis and retrieval completeness. Q3: Attribution, traceability, and provenance

To maintain consistency, I used Claude Haiku as an LLM judge, asking it to score each response across four dimensions on a scale of 1 to 10. The instructions were deliberately simple — I only specified the four dimensions and the scale and let Claude do the rest.

There is an obvious limitation here: Claude Haiku was also the model used in System D, which means it was effectively marking its own homework. I did consider using a second model as judge but settled on Claude Haiku for simplicity.

What I found most interesting was not the absolute numbers, but the recurring patterns in how different retrieval architectures behaved — where they succeeded, where they struggled, and what that reveals about when Graph RAG actually adds value.

What the scores reveal

The headline result was clear: the frontier model produced the strongest answers overall. But the overall averages hide a more interesting story. Looking at each question individually reveals where Graph RAG added value, where it struggled, and why the “best” architecture depends heavily on the problem you’re trying to solve.

Q1 — How does the four-question risk framework relate to SDLC security controls?

The first question was designed to test multi-hop reasoning across documents.

The “four-question risk framework” appears in one document, while the SDLC security controls appear in another. Answering the question requires more than simply retrieving a relevant passage. The model needs to identify concepts in one document, understand their meaning, and then connect them to related concepts in a completely different source.

This is exactly the kind of problem where Graph RAG is often claimed to provide an advantage. The graph explicitly captures relationships between concepts, making it easier to traverse connections that may be separated across documents.

The results broadly supported that hypothesis.

Plain RAG struggled. The relevant chunks were retrieved, but they did not provide enough context for the model to confidently connect the framework and the SDLC controls. Rather than hallucinating, the model admitted it could not determine the answer. While technically safer, that also made it the least useful response.

Graph RAG performed much better. By combining document passages with the relationship structure extracted from the graph, the model was able to identify meaningful connections between the two sources and explain how the controls relate to the framework. The answer was still not perfect, but it demonstrated a level of cross-document reasoning that the baseline RAG implementation could not consistently achieve.

The frontier model produced the strongest answer overall. Because both documents were available in the context window simultaneously, it could reason across the full source material without needing any retrieval step.

The interesting takeaway isn’t that the frontier model won. I already knew it had access to more context. The more important observation is that Graph RAG substantially outperformed Plain RAG on a question that required connecting information across documents. This is one of the clearest examples from the experiment where the graph added genuine value rather than additional complexity.

Q2 — What controls appear in both the CISO guide and the SDLC post?

The second question tested cross-document synthesis and retrieval completeness.

Unlike the first question, this wasn’t primarily a reasoning problem. The answer was explicitly present in both documents. The challenge was retrieving enough relevant information from each source and combining it into a complete answer.

This is the kind of question that many RAG implementations struggle with. The model doesn’t need to infer much, but it does need access to all of the relevant evidence. Missing even a few key passages can lead to an incomplete answer.

This turned out to be the weakest area for all three local retrieval systems.

Plain RAG identified some of the overlapping controls but failed to retrieve the complete set. Graph Only performed even worse, introducing controls that appeared in neither document. Graph RAG performed slightly better than the baseline in the quality of its reasoning, but it still failed to identify all of the common controls.

The frontier model produced a very different result. Because both documents were available in full, it correctly identified all the controls that appeared in both sources.

What surprised me most was that Graph RAG provided little advantage over Plain RAG here. Going into the experiment, I expected the graph structure to help identify controls that appeared in both documents.

In practice, the limitation seemed to be less about reasoning and more about coverage. The answer required identifying every control that appeared across both source documents. While the relevant information may well have existed somewhere within the retrieved graph and document corpus, only a subset of chunks, nodes, and relationships were surfaced to the model at query time. As a result, some of the evidence needed to construct a complete answer never made it into the prompt.

This highlighted an important distinction between reasoning problems and coverage problems. Graphs can help a model connect concepts that it has already retrieved. They are less effective when the primary challenge is ensuring that every relevant piece of information has been retrieved in the first place.

My suspicion is that increasing the retrieval breadth, for example retrieving more chunks, expanding the graph traversal, or improving entity resolution, would have improved the result. But that also reinforces the broader point: Graph RAG does not eliminate the retrieval problem. It adds structure and reasoning capability on top of retrieval — but if the relevant chunks or graph nodes never make it into the prompt in the first place, because of retrieval limits or similarity thresholds, the graph has nothing to reason over.

For me, this was one of the clearest examples of where having the entire corpus available in the context window remains a powerful advantage when the document set is small enough to fit.

Q3 — Which document introduced the concept of sandboxed execution?

The final question tested attribution and provenance.

Unlike the previous questions, this wasn’t a reasoning or synthesis problem. The task was simply to identify which source document first introduced a particular concept.

This produced the most surprising result of the experiment. The Graph Only approach, which struggled badly in Q2, performed almost as well as the frontier model.

Looking at the graph helped explain why. This wasn’t a synthesis problem. It was a provenance problem. Once the relevant concept had been located, the graph contained enough contextual information for the model to correctly identify the source document.

What surprised me was how well the Graph Only approach performed. In the previous question it struggled because the model lacked the grounding text needed to answer accurately. Here, that limitation mattered far less. The task wasn’t to explain or synthesise information. It was simply to trace a concept back to its origin.

The other interesting observation is how closely grouped the scores are. Unlike Q1 and Q2, this question didn’t require broad retrieval coverage or multi-hop reasoning. It was a single attributable fact, and all four approaches were able to answer it reasonably well.

So When Does Graph RAG Actually Add Value?

The experiment revealed three distinct patterns:

• Multi-hop reasoning: Graph RAG provided a clear advantage when the answer required connecting concepts across documents.
• Retrieval completeness: Graph RAG offered little benefit when the primary challenge was retrieving all relevant information.
• Provenance and traceability: Graph-based approaches performed surprisingly well when the task was identifying where information came from.

The lesson I took away wasn’t that Graph RAG is always better than traditional RAG, but rather, different retrieval architectures solve different problems, and the right choice depends on the nature of the questions being asked.

The table below draws on the evaluation results, and wider research to compare the four approaches.

What I’d do differently

Going into this project, I assumed that the hard part would be getting the graph retrieval working. However, it turned out that much of the effort ended up in areas I hadn’t thought much about beforehand: ontology design, entity resolution, graph maintenance, and operational considerations. If I were building the system again, these are the changes I’d make first.

Starting with an ontology (even a simple one)

The biggest source of graph noise was relationship proliferation. Even with a constrained extraction prompt, the model initially produced dozens and dozens of one-off relationship types. This made the graph harder to query, analyse, and maintain because concepts that should have been connected ended up linked through slightly different relationship labels.

To address this, I eventually moved to a small, constrained vocabulary of relationship types. This significantly improved the consistency of the graph and made graph traversal results much easier to interpret. It’s possible that a larger frontier model would have produced more consistent relationship types from the outset than the local Phi-4 model I used. However, I suspect the problem would not disappear entirely. Regardless of model capability, there is still value in defining a relationship vocabulary that reflects the questions the graph is intended to answer.

Taking a step back, the lesson wasn’t really about choosing the “right” relationship labels. In a production scenario, I would expect to start with some form of domain ontology, even if it’s a relatively simple one that defines the entities, relationships, and information that matter most to the business. This provides a consistent foundation for extraction and retrieval, allowing concepts and relationships to be mapped to meaningful domain constructs rather than proliferating into dozens of subtly different variants. The ontology doesn’t need to be perfect, but it helps ensure the graph evolves into a meaningful representation of the business domain rather than simply becoming a collection of connected data.

Importance of entity resolution

A similar challenge appeared at the entity level. Just as relationship types proliferated without a constrained ontology, concepts themselves were often represented in multiple ways across the graph.

For example, I ended up with three separate nodes for what was essentially the same concept: “prompt injection”, “injection attack”, and “prompt injection attack”. Rather than being treated as a single concept, the graph spread the associated relationships and context across multiple nodes, making it harder to build a complete picture.

In a production environment, I would expect some form of entity resolution to address this — for example by comparing embeddings for a new entity against existing ones and merging them when they exceed a similarity threshold. Ultimately, the same principle applies here as it does with relationship types: a graph can only support effective retrieval and reasoning if concepts are represented consistently.

A flawed evaluation can lead to the wrong conclusions

The importance of a robust evaluation framework became clear early in the project. In my first test runs, the evaluation script was truncating the source documents before passing them to the evaluator, which resulted in some correct answers being marked as unsupported because the relevant evidence appeared outside the truncation window.

More broadly, I kept the evaluation deliberately lightweight, effectively outsourcing the scoring to Claude Haiku against a rubric covering accuracy, completeness, reasoning, and provenance. While sufficient for exploring the relative strengths and weaknesses of different retrieval approaches, it was far from perfect, particularly as Claude was effectively evaluating its own answers in System D.

For the purposes of this experiment, that was a trade-off I was willing to accept. However, in a production environment I would expect the evaluation approach to be far more rigorous, with representative test cases, predefined expected outcomes, independent evaluation mechanisms, and a clear methodology for measuring performance over time.

The lesson is simple: if the evaluation is flawed, it becomes very easy to draw the wrong conclusions about the architecture.

Additional details — live UI output

The comparison UI was built using Gradio and runs locally alongside the knowledge graph. The screenshots below show two additional example questions run through all four systems with scores generated by Claude Haiku in real time.

Example Q1 “What are the top takeaways from each article?”

Example Q2 “ What controls should organisations implement before deploying AI agents?”

The Code

The full implementation is at github.com/arijitghoshal222/graph-rag-experiment — Neo4j graph construction, triple extraction, all four RAG systems, the evaluation framework, and the Gradio UI. The README walks through setup from scratch.