A Comprehensive Benchmark for Evaluating Context Retrieval in Code Agents
A collaboration between
LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving.
We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution.
Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context.
ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks.
The pipeline extracts file views and spans from agent trajectories, then computes coverage and precision metrics by comparing against human-annotated gold context at multiple granularities.
🏆 Live leaderboard and interactive results: https://contextbench.github.io/
# Install dependencies
pip install -r requirements.txtDownload the ContextBench dataset from Hugging Face:
from datasets import load_dataset
# Load full dataset (1,136 instances)
dataset = load_dataset("Contextbench/ContextBench", "default")
# Or load the verified subset (500 instances)
dataset_verified = load_dataset("Contextbench/ContextBench", "contextbench_verified")
# Save to parquet for evaluation
dataset['train'].to_parquet("data/full.parquet")Or download directly from: 🤗 Hugging Face Dataset
python -m contextbench.evaluate \
--gold data/full.parquet \
--pred path/to/trajectory.traj.json \
--out results.jsonlThe evaluation automatically detects trajectory formats, clones repositories, extracts code symbols, and computes comprehensive metrics across file, symbol, span, and edit-location granularities.
Online documentation: https://euniai.github.io/ContextBench/
Complete documentation is available in the docs/ directory:
- Agent Trajectory Extractors – How to extract trajectories from different agents (Agentless, SWE-agent, MiniSWE, etc.)
- Running Agents on ContextBench – Unified runner (
python -m contextbench.run) for evaluating agents across all benchmark variants - Process Trajectories – Details on trajectory format and parsing
ContextBench/ # Repository root
├── README.md # This file (project homepage)
├── contextbench/ # Python package
│ ├── agents/ # Trajectory extractors for different agents
│ ├── core/ # Repo management, intervals, file I/O
│ ├── extractors/ # Tree-sitter symbol extraction
│ ├── metrics/ # Metric computation
│ ├── parsers/ # Gold, trajectory, and diff parsers
│ └── evaluate.py # Main evaluation entrypoint
├── data/ # Benchmark datasets (Verified, Pro, Poly, Multi)
│ ├── selected_500_instances.csv
│ └── *.parquet
├── docs/ # Documentation and assets
│ ├── source/ # Sphinx RST documentation
│ └── assets/ # Images and figures
├── scripts/ # Utility scripts for running agents
├── agent-frameworks/ # Agent implementation submodules
│ ├── agentless/
│ ├── mini-swe-agent/
│ ├── openhands/
│ └── swe-agent/
└── requirements.txt # Python dependencies
For detailed metrics definitions and implementation, see the Sphinx documentation.
If you use ContextBench in your research, please cite our paper:
@misc{li2026contextbenchbenchmarkcontextretrieval,
title={ContextBench: A Benchmark for Context Retrieval in Coding Agents},
author={Han Li and Letian Zhu and Bohan Zhang and Rili Feng and Jiaming Wang and Yue Pan and Earl T. Barr and Federica Sarro and Zhaoyang Chu and He Ye},
year={2026},
eprint={2602.05892},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2602.05892}
}📄 Paper: arXiv:2602.05892
ContextBench is a collaborative research project between:
- Nanjing University (南京大学)
- University College London
We thank the developers of the agent frameworks evaluated in this benchmark: Agentless, SWE-agent, Mini-SWE-Agent, OpenHands, and Prometheus.
We gratefully acknowledge Mistral AI and Amazon Web Services (AWS) for providing API support that enabled large-scale experiments and evaluations.
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
