# comparethemarket-context-retrieval-ai-code-review-gkg-rag-2026-03-06

## Veille

Empirical study by the **Compare the Market** engineering team (Meerkat Careers, UK) evaluating four approaches to **context retrieval for AI code review**: Baseline (no additional context), **RAG** (vector search), **GKG** (GitLab Knowledge Graph, AST-based knowledge graph), and **GKG+RAG** (hybrid). Evaluation on **79 real merge requests** with **MLflow on Databricks**. Striking result: **RAG performs worse than the baseline** on almost every metric — vector noise is counterproductive for code review. **GKG outperforms RAG by +21%** in inline comments coverage (0.696 vs 0.577) through structural AST understanding (Tree-sitter + Kuzu graph database). Code requires **structural** understanding (callers, signatures, hierarchies), not mere semantic similarity. GKG costs 4× the baseline but delivers measurable improvements; RAG costs 3× with no improvement. Implemented as a **Docker sidecar** in CI/CD wrapping the GKG binary (still in GitLab beta) with a local MCP server.

## Titre Article

Comparing Context Retrieval Approaches for AI Code Review

## Date

2026-03-06

## URL

https://comparethemarketcareers.com/blog/comparing-context-retrieval-approaches-for-ai-code-review/

## Keywords

Compare the Market, Meerkat Careers, AI code review, context retrieval, RAG, GKG, GitLab Knowledge Graph, AST, Tree-sitter, Kuzu, code knowledge graph, MLflow, Databricks, merge requests, empirical evaluation, inline comments coverage, semantic vs structural similarity, Docker sidecar CI/CD, local MCP server, counterproductive vector noise, automated code review, AI review cost-efficiency, RAG worse than baseline, structural code approach

## Authors

Équipe Engineering Compare the Market (Meerkat Careers, UK — site de comparaison d'assurances et services financiers).

## Ton

**Profile**: Technical article published on the careers blog of Compare the Market (Meerkat Careers brand, UK), a consumer price-comparison company. An internal engineering format shared publicly — the tone is that of a team that ran a rigorous evaluation and shares the results without embellishment. Target audience: software engineers, DevOps, software quality teams looking to tool their code review with AI, architects evaluating RAG vs structural approaches for code.

**Style**: Empirical technical register, data-first. The article proceeds methodically: hypothesis → experimental setup (4 approaches, 79 MRs, MLflow/Databricks) → metrics → results → cause analysis → recommendations. No hype, no marketing promises: RAG's negative results are presented head-on, without any attempt to soften them. The style is that of an internal engineering report made public — factual, structured, with metrics tables. Authority comes from the data, not from rhetoric.

**Epistemic position**: empirically honest. The article does not try to sell GKG as a miracle solution — it acknowledges the 4× overhead and GKG's beta status. The negative result on RAG is the most salient finding: demonstrating that a popular approach is counterproductive in a specific use case is more useful than promoting the alternative. The Docker sidecar architecture is described as a pragmatic workaround (GKG not yet natively integrated into GitLab CI/CD).

## Pense-betes

- **Date / source**: March 6, 2026, Meerkat Careers blog (Compare the Market, UK). Author: engineering team (individual name(s) not available — site blocked 403, entry based on content indexed by search engines and secondary sources).
- **Problem addressed**: what is the best way to provide codebase context to an AI reviewer? Is semantic similarity (RAG) sufficient, or is structural understanding of code (AST/graph) required?
- **4 approaches evaluated**: 1. **Baseline** — no additional context, the AI model only sees the MR diff. 2. **RAG** (Retrieval-Augmented Generation) — code chunking, embeddings, vector search to retrieve semantically similar snippets. 3. **GKG** (GitLab Knowledge Graph) — AST parsing via **Tree-sitter** (`gitlab-code-parser`), structured knowledge graph stored in **Kuzu** (graph database), structural queries (callers, class hierarchy, function signatures). 4. **GKG+RAG** (hybrid) — combination of both approaches.
- **Key metrics across 79 merge requests** (evaluated via **MLflow on Databricks**): | Metric | GKG | RAG | Gap | |----------|-----|-----|-------| | Inline Comments Coverage | **0.696** | 0.577 | **+21%** | | Summary Coverage | **0.681** | 0.664 | +3% | | Issue Coverage | **0.929** | 0.926 | marginal | | Score Accuracy | **GKG better** | RAG worse than baseline | — |
- **Main finding: RAG performs worse than the baseline** on almost every metric. Adding noisy context is counterproductive.
- **4 identified causes of RAG's failure for code review**: 1. **Noise** — vector similarity retrieves code that "looks similar" but is not relevant. 2. **False positives** — RAG finds functions unrelated to the change. 3. **Per-file limitation** — RAG has no understanding of cross-file relationships. 4. **Distraction effect** — additional context can mislead the model rather than help it.
- **Why GKG works**: code review requires **structural** understanding — when reviewing a change in a function, one must know **who calls it**, **what it calls**, and **how it fits into the architecture**. GKG precisely identifies callers, understands signatures, and traces code relationships. This is **structural navigation**, not **semantic similarity**.
- **Technical implementation**: GKG still in beta, not yet available as a native GitLab CI/CD feature → the team built a **Docker sidecar**: a lightweight container wrapping the official GKG binary, mounted alongside the reviewer in the CI pipeline. On every MR pipeline: (1) the sidecar mounts the project source, (2) indexes the full codebase and builds the knowledge graph from scratch, (3) starts the **GKG MCP server** on a local port, (4) exposes a set of tool calls that the AI reviewer connects to.
- **Cost-benefit analysis**:
- GKG: **4× the baseline cost** → measurable and justified improvements.
- RAG: **3× the baseline cost** → results worse than adding nothing.
- GKG+RAG: cumulative cost with no additional benefit vs GKG alone.
- **Conclusion**: if quality is paramount, use GKG. Avoid RAG and GKG+RAG.
- **Watch-file connections**:
- **Strong convergence** with the doctrine *"grep wins when you know what you're looking for, AST wins when you need structural relationships"* — a 2026 trend confirmed by the practices of Cursor, Claude Code, Devin (cf. MindStudio *"Coding Agents Skipped RAG"*).
- Aligns with **Zhutov/QMD** (2026-03-01) on the superiority of structured search over vector search for code.
- Extends **Trivedy/LangChain** (2026-03-10) *Anatomy of an Agent Harness*: the harness (here, the GKG + MCP sidecar) matters more than the model.
- Aligns with **Dropbox/Okumura** (2026-05-28) *"the value comes less from the model itself than the systems surrounding it"* — structural context makes the difference, not the LLM.
- Complements the **MCP** file: GKG exposed via a local MCP server, confirming the MCP sidecar pattern in CI/CD.
- Corroborates **Anthropic Data Science** (2026-06-03): *"raw grep across thousands of SQL files → accuracy moves by only one point"* — the bottleneck is structure, not access.
- Reinforces the anti-naive-RAG thesis of **Seale** (2025-05-30) *Philosophy Eats AI*: ontology (here, the AST) takes precedence over similarity.
- **To be used for**: enterprise AI code-review architecture decisions; argument against RAG-by-default for code; design of agent-augmented CI/CD pipelines; evaluation of code-context tools (GKG, CodeGraphContext, code-review-graph).
- **Limitations**: (a) 79 MRs = modest sample size; (b) a single codebase (Compare the Market) — representativeness bias; (c) GKG still in beta, not yet validated at scale; (d) author(s) not identified — site blocked 403; (e) no comparison with other structural approaches (Augment Context Engine, CodeGraphContext, etc.).
- **LinkedIn shortlink**: the link `lnkd.in/dacPc6fM` (pending since 2026-05-15) points to this article.

## RésuméDe400mots

The **Compare the Market** (Meerkat Careers, UK) engineering team published, on March 6, 2026, an empirical evaluation of four context-retrieval approaches for **AI code review**: Baseline (no additional context), **RAG** (vector search via embeddings), **GKG** (GitLab Knowledge Graph, an AST-based knowledge graph via Tree-sitter and the Kuzu graph database), and a **GKG+RAG** hybrid. The evaluation covers **79 real merge requests**, measured via **MLflow on Databricks**.

The main finding is counterintuitive: **RAG performs worse than the baseline** on almost every metric, including inline comments coverage, summary coverage, and score accuracy. Adding context retrieved via vector similarity is not only useless but **counterproductive** for code review. Four causes are identified: **noise** (vector similarity retrieves code that "looks similar" without being relevant), **false positives**, the lack of understanding of **cross-file relationships**, and a **distraction effect** that misleads the model.

Conversely, **GKG outperforms RAG by +21%** in inline comments coverage (0.696 vs 0.577). The reason is structural: code review requires knowing **who calls a function**, what it calls, and how it fits into the architecture — information that the AST and the knowledge graph capture natively, but that semantic similarity cannot provide. GKG precisely identifies callers, understands function signatures, and traces code relationships.

The implementation is pragmatic: since GKG is still in beta and not yet natively integrated into GitLab CI/CD, the team built a **Docker sidecar container** that wraps the GKG binary, indexes the codebase on every MR pipeline, and exposes the tools via a **local MCP server**. The cost is 4× the baseline, but the improvements are measurable and justified. RAG costs 3× the baseline for worse results.

This study confirms a major 2026 trend: for code, **structural** approaches (AST, knowledge graphs, targeted grep) outperform **vector-based** approaches (semantic RAG). Code is not text — its informational value lies in its **structural relationships**, not in its lexical similarity. Strong convergence with Zhutov/QMD, Dropbox/Okumura (*"the value comes from the systems surrounding the model"*), and the Anthropic Data Science doctrine (*"the bottleneck is structure, not access"*). To be used as empirical reference for AI code-review architecture choices and as a counter-argument to RAG-by-default in the code domain.

## GrapheDeConnaissance

- Compare the Market —a_créé→ évaluation empirique revue de code IA (EVENEMENT, 0.95)
- GKG —surpasse→ RAG (TECHNOLOGIE, 0.97)
- GKG —mesure→ 0,696 inline comments coverage vs RAG 0,577 (+21 %) (MESURE, 0.96)
- RAG —mesure→ performance pire que baseline sur presque toutes les métriques (MESURE, 0.96)
- GKG —utilise→ Tree-sitter (TECHNOLOGIE, 0.95)
- GKG —utilise→ Kuzu (TECHNOLOGIE, 0.94)
- GKG —est_instance_de→ GitLab Knowledge Graph (TECHNOLOGIE, 0.97)
- Compare the Market —affirme_que→ le code exige une compréhension structurelle, pas une similarité sémantique (AFFIRMATION, 0.95)
- RAG —réduit→ qualité de la revue de code IA (bruit vectoriel contre-productif) (AFFIRMATION, 0.94)
- GKG —permet→ identification précise des appelants, signatures, hiérarchies de code (CONCEPT, 0.95)
- évaluation empirique revue de code IA —utilise→ MLflow (TECHNOLOGIE, 0.93)
- évaluation empirique revue de code IA —mesure→ 79 merge requests évaluées (MESURE, 0.95)
- Compare the Market —a_créé→ GKG (TECHNOLOGIE, 0.92)
- GKG —utilise→ serveur MCP local (TECHNOLOGIE, 0.92)
- GKG —mesure→ coût 4× baseline avec améliorations mesurables (MESURE, 0.93)

---
Canonical: https://www.thekb.eu/en/fiches/comparethemarket-context-retrieval-ai-code-review-gkg-rag-2026-03-06/