LLM-as-a-Judge to evaluate graph-based RAG

CONTEXT

While working on a research project for the University of Sydney, I faced the task of establishing whether a custom-built graph-based RAG pipeline actually retrieved better answers than two published baselines, GraphRAG and HippoRAG. The dataset was a set of meeting transcripts, and the goal for these RAG systems was not only to retrieve facts from a single meeting, but the rationale and explanation behind decisions taken across the whole transcript corpus.

Without a measurement instrument, the comparison would have come down to reading each system's answers and forming an impression. Human scoring is reliable but does not scale. On the other hand, an LLM-as-a-Judge framework scales, but has to be shown to agree with human judgement before its numbers mean anything.

PROBLEM

First, no previous QA benchmark had been created to evaluate this transcript corpus in this way. Earlier benchmarks built on this dataset tested retrieval within a single meeting, not across meetings. A new benchmark had to be created from scratch.

Second, each question had to be designed to evaluate a different RAG capability, because a different question validates a different type of reasoning. Some questions test a specific fact ("what decision was made in this meeting"), others test change over time ("how did a decision evolve across four meetings"), and others test whether the model hallucinates ("was this decision ever resolved or not?").

Third, a decision had to be taken on the evaluation criteria. The obvious strategy (eg. asking a strong model to rank answers from 0 to 10) was not the solution. Previous research on this kind of evaluation found that 10-point LLM scores collapse into three distinct bands relative to human scores. A different approach was required.

SOLUTION

• Create a QA benchmark from scratch. I designed 32 cross-meeting questions spanning six question types. Each question was tagged with its hop distance (the number of reasoning steps needed to reach the answer) and annotated with its ground-truth evidence span, the specific passages supporting the correct answer.
• Design a specific LLM evaluation criteria. The LLM judge received the question type, hop distance, ground-truth answer and criteria specific to each question type, so that it could reach a conclusion. I designed the criteria around a coarse 0/1/2 scale, chosen deliberately against finer rubrics. The judge was asked to write its justification before its score, so the verdict followed from an explicit comparison.
• Validate the results against human scoring. The answers to all 32 questions were also scored by hand, and then the results were compared. The judge and the human rater agreed on the headline result: GraphRAG performed best with a 0.75 LLM score and 1.03  human score, and disagreed only mildly on the two closer systems.  The final human filter allow me to gain confidence over the LLM Judge results.

OUTCOME

• #1 system overall was GraphRAG. Its greatest strength was creating synthesis via summarization at index time, while the other systems evaluated defer synthesis to generation.
• 29x longer are GraphRAG's answers compared to HippoRAG's, and 3x longer than the custom-built pipeline. Good performance comes at a cost in verbosity and tokens.
•  1 framework validated against human scoring and honest about its limits. All three systems ran the same base model and were evaluated on the same multi-type benchmark and criteria, which allowed me to explain performance rather than trust a single aggregate.