Google examines how to measure a coding assistant's usefulness before it writes code
Research connected to Jules focuses on finding the right problem and timing an intervention, rather than treating diagnosis and successful repairs as the same result.
Measuring useful assistance
On June 22, 2026, Google published a report about evaluating proactive coding assistants. In a preliminary internal experiment, Hit@5 rose from 33% to 57% when exploration increased from two rounds to three. This measures a correct diagnostic insight appearing among the first five suggestions, not the percentage of all software bugs repaired.
These are early findings, not evidence that an assistant will achieve the same performance across every repository or team.
Autonomy and initiative differ
The associated research paper distinguishes executing a requested task from deciding what deserves attention in the first place. Its proposed perspective includes selecting a relevant problem, grounding it in evidence, deciding when to interrupt and adapting to user feedback.
That distinction exposes a practical concern: more alerts do not necessarily mean more help. An assistant can become distracting if its suggestions have little connection to the team's current objective.
What does SWE-bench measure?
SWE-bench evaluates real software issues collected from GitHub. A proposed patch is applied and the repository's tests are run to assess whether it resolves the issue. That is different from the earlier question of which problem should have been identified.
A useful diagnosis and a working repair are connected stages, but one result cannot simply stand in for the other.
A practical team experiment
Our interpretation is that a small development team can test the principle without building an enormous evaluation project. Select a previously repaired problem, give the assistant the code as it stood before that repair, and examine whether it finds useful evidence.
Then ask more than how many suggestions appeared. Was the diagnosis right? Did it locate the relevant file or behaviour? Could a reviewer verify it easily? How much review time did it consume relative to its value?
Recording unhelpful suggestions also matters. Counting only successes can hide the interruption burden imposed on the developer.
What the result does not establish
A plausible suggestion is not yet repaired, tested software. Changes still need review and testing before they reach users. The post concerns evaluation research; it is not an announcement that every Jules account received a new feature with identical capabilities that day.
Sources
Founder and editor-in-chief of Kurd Developers. Phone and tech news on YouTube, Instagram and TikTok.
More stories
One text can break your phone: Android's 'SMS of Death' explained
A flaw in Android 12 and later lets a specially crafted SMS drop calls and even make a phone unusable. The fix is in October's security patch.
GTA VI lands November 19: prices, editions and what to know
Rockstar has locked the date: Thursday, November 19 on PS5 and Xbox Series X|S. Standard costs $79.99 and Ultimate $99.99.
Apple's first touchscreen MacBook Pro brings OLED and Dynamic Island
The new PROLED XDR display responds to touch, the notch is gone and Dynamic Island takes its place. Prices start at $2,899.
The $1,999 iPhone Duo Has a Black-Bar Problem, and Fixing It Is Harder Than It Looks
Apple's first foldable is a hardware flex. But unfold it and many of your apps shrink into a phone-shaped window floating in black. Here's why the app world wasn't ready, and why developers can't just hit "stretch".