A/B test whether pre-gathered context helps coding agents complete tasks
A new GitHub simulator compares how AI coding agents perform with and without pre-gathered organizational context, measuring quality, speed and cost differences.
The context-engine-simulator, developed by Unblocked and published on GitHub, enables developers to test whether providing AI coding agents with pre-gathered organizational context improves their performance. The tool runs parallel tests against real repositories, comparing baseline performance (agents working from task descriptions alone) against context-enhanced performance (agents first gathering relevant context before planning and implementing).
The simulator supports multiple popular coding agent CLIs including Claude Code, OpenAI Codex, Cursor, and Grok, allowing developers to compare how each AI assistant benefits from additional context. This could help organizations determine which coding tools work best with their specific codebases and workflows.
How It Works
Each test execution runs two arms in parallel within isolated git worktrees to prevent interference. The baseline arm follows a three-step process: plan, review, and implement, with results scored against acceptance criteria. The context-enhanced arm adds three additional steps: gather context, extract patterns, and gather plan context before the same review and implement phases.
The context gathering step involves a read-only researcher agent searching the codebase and connected sources like issues, pull requests, chat messages, and documentation to create a focused context bundle. This bundle is then distilled into coding-style conventions and used to plan the task with a head start.
The context arm additionally produces an attribution analysis showing which gathered context actually influenced the implementation.
Setup and Usage
To use the simulator, developers need Node.js version 18 or higher, git, and at least one supported agent CLI installed. Each agent must be authenticated per its own CLI, and the agent's MCP servers must be configured for the target repository before running tests. The simulator shells out to whichever agent CLI is selected.
The tool can be run either with a YAML fixture file or through CLI flags. Fixtures specify the repository path, branch, task description, acceptance criteria, agent type, and model. When run, the simulator generates results including a terminal table, JSON file, and standalone HTML report showing quality scores, timing, costs, and token usage for both test arms.
To ensure accurate measurements, the simulator includes contamination controls. The --disable-mcp flag blocks specific MCP servers in both arms' worktrees, and the tool aborts runs if blocked servers are called, preventing silent skewing of comparison results.
Key Features
Each test runs in throwaway git worktrees that are automatically cleaned up unless --keep-worktrees is set. The simulator calculates costs based on each agent's reported token usage where available. For subscription-billed agents like Grok that report token counts but not per-token prices, costs display as $0.
The tool never commits or pushes changes to repositories, only reading the target repo and working within isolated worktrees. This approach ensures that testing doesn't interfere with the actual codebase while providing comprehensive performance metrics for different AI coding approaches.