GitHub Researcher Automates Analysis of Coding Agents with New AI Tool

A researcher at GitHub's Copilot Applied Science team has created 'eval-agents,' a tool that automates the analysis of coding agent trajectories, effectively eliminating repetitive intellectual toil. By leveraging GitHub Copilot, the tool surfaces patterns across hundreds of thousands of lines of code, enabling faster feedback loops and team-wide collaboration.

'I may have automated myself into a new role—maintaining the tool so my peers can do the same,' said the researcher, who leads the project.

Background

Coding agents are AI systems that solve tasks by generating and executing code. Their performance is measured against benchmarks like TerminalBench2 and SWEBench-Pro, which produce detailed trajectories—JSON files listing every thought and action an agent took.

Each task yields its own trajectory, and a single benchmark run can produce dozens of files, totaling hundreds of thousands of lines. Manually analyzing this data is impossible, requiring scientists to repeatedly use Copilot to find patterns and then investigate a few hundred lines.

What This Means

Eval-agents turns that repetitive loop into an automated process. Scientists can now author new analysis agents easily, share them across the team, and make contributions through coding agents themselves.

'The guiding principle was that engineering and science teams work better together,' the researcher noted. The tool is designed for easy sharing and authorship, leveraging skills from the researcher's time as an OSS maintainer on the GitHub CLI.

For the wider software engineering community, this demonstrates how agent-driven development can automate intellectual toil, freeing experts to focus on creative problem-solving. The result is a dramatically faster development loop for both the individual and the team.

As the researcher concluded, 'By removing the friction of trajectory analysis, we unlock more time for breakthrough research.'