Merged pull requests is a vanity metric
Coding agents push merged-PR counts up whether or not anything improved. The six numbers to report instead, how to baseline them in week one, and the one result that counts.
- KKK Manoj KumarCo-founder
5 min read
The first chart people show after a coding-agent rollout is merged pull requests per week. It always goes up. Agents make a PR cheap to open, so more get opened, and as long as someone keeps approving, more get merged. The chart proves the licences are in use. It says very little about whether the team ships better software sooner.
The public numbers point the same way. Microsoft's study of its own Claude Code and Copilot CLI rollout found roughly 24% more merged PRs among adopters over four months. That's a real effect, and it's also the number you'd expect to move first. Faros, which sells engineering analytics, reports from customer telemetry that high-adoption teams merged 98% more PRs while review time rose 91% and PR size rose 154%, with company-level metrics flat (vendor-reported). The 2025 DORA report puts it in one line: AI adoption raises delivery throughput and lowers stability.
So the merged-PR count goes on the dashboard last, if at all. These are the numbers we put first.
Six numbers that tell you something
| Number | How to measure it | What a bad trend looks like |
|---|---|---|
| Lead time | Issue marked ready to change running in production | Flat while PR count climbs |
| Change failure rate | Deploys that caused an incident, rollback or hotfix | Up a few points after rollout |
| Review wait | PR marked ready to first human review, plus total time in review | Climbing week on week |
| PR size | Median lines changed and files touched | Median doubles |
| Rework and revert rate | PRs reverted, or followed by a fix to the same lines within 14 days | Up for agent-authored PRs only |
| Cost per merged PR | Seat plus token spend for the team, divided by merged PRs | Rising with no lead-time gain |
Start the lead-time clock when the issue is approved. Agent work hides a lot of time before the PR exists: a plan that sat waiting for sign-off, a build that looped on red CI for an hour. If the clock starts at PR open, agent PRs look instant and the comparison with human work is meaningless.
Change failure rate and rework rate are the stability side. Track them separately for agent-authored and human-authored PRs, which means you need a way to tell them apart: a label, a commit trailer or the bot account that opened the PR. Without that split, a bad month for agent PRs disappears into the team average.
Review wait and PR size are the early warnings, and they move weeks before incidents do. When PRs double in size, reviewers either slow down or skim. Both show up later, one in lead time and the other in change failure rate. Engineering managers describe being handed a thousand-line diff across dozens of files by someone who can't say what a file does. That PR lands in the PR size column the day it opens, long before the bug it carries reaches production.
Cost per merged PR keeps the spend conversation honest. Total token spend tells finance the bill. Cost per merged PR tells you whether the bill bought anything, and it lets you compare a team running the largest model for everything with a team that routes work by task.
Baseline in week one
Most teams skip the baseline because the rollout feels urgent, then spend the next quarter arguing about anecdotes. The data already exists. Git has PR open, merge and revert times. CI has run history. The tracker knows when each issue moved to ready. The incident tool knows which deploy caused what. Pull the last eight to twelve weeks before the first agent PR merges.
A few decisions make the baseline usable later. Write down the event that starts lead time. Mark agent-authored PRs from day one, because labelling them after the fact is miserable. Choose a pilot team and a comparison team doing similar work, so a quiet month or a big launch doesn't get credited to the agents. Keep everything at team level. Individual leaderboards turn a measurement into a target, and people find the cheapest way to hit a target.
Then write the goal before anyone starts, in one sentence the team agrees with. A good one reads something like: lead time down, change failure rate within a point of baseline, review wait no higher than today. Self-reported speed-ups stay out of it. Ask engineers how the tools feel, by all means, and keep that survey separate from the numbers.
Throughput up, stability flat
The only result worth reporting is more work delivered while stability holds. Lead time falls or merged work rises, and change failure rate, revert rate and review wait stay where they were. Anything else is a cost moved somewhere less visible. Throughput up with change failure rate up means work moved from the author to on-call. Throughput up with review wait up means it moved to your senior engineers, and they'll tell you about it in their one-on-ones before the dashboard does.
A six-week readout for a pilot team might look like this. The numbers are illustrative.
Pilot team · weeks 1-6 vs 8-week baseline
lead time (median) 6.1d → 4.3d
change failure rate 9% → 10%
review wait (median) 5.2h → 6.8h ← watch
PR size (median lines) 142 → 188
rework / revert rate 4% → 5%
cost per merged PR n/a → $11.40That readout is a qualified yes. Lead time moved, stability held within a point, and review wait is the next thing to fix, probably by capping PR size and attaching evidence to every agent PR so reviewers know where to look. It's a far more useful conversation than "PRs up 40%", and it's one the team can act on in the next sprint.
Put the six numbers on one page, fill in the baseline column before the first agent PR merges, and review it with the team every two weeks. It's the first page we build in a coding-agent rollout.