Hill climbing with AI judges: getting coding agents past their first build

My coding agents will build whatever I describe, and it usually runs. Whether it's any good is a separate question, and when I ask the agent that built it, it tells me it's done. It has no score to beat, and in my experience it stops at the first version that works.

Recently I've been giving my agents a score. Other models judge the work blind against a written rubric, a builder fixes what they agree on, and the judges score it again. On a recent feature, the version already in production scored 6.0, the agents' first build scored 7.47, and after two rounds of fixes it scored 8.04. Those are the judges' numbers, and the judges mislead in a few consistent ways. On another run they preferred a version that I rated no better than the original.

One round#

1. A script signs in and takes the same screenshots every time: my build, and other products doing the same job. For a recent feature that was six other products, at 1440×900 on desktop and 390×844 on a phone.
2. A second script shuffles them into a pack where each product has a letter. The key that says which letter is mine stays outside the pack, and the order changes every round. The screenshots still show each product's own name and logo, so it's blind only in the sense that nothing marks one of them as mine.
3. Each judge is a fresh, read-only session on a different model. It gets the pack, the rubric and the same short brief. It isn't told which product is mine, which round it is, what anything scored before or what score I'm hoping for.
4. A scoring script takes the mean of the judges' scores for each category, and the mean of the categories as the overall score. It lists any category where the judges are two or more points apart. Those go in the report, so I can see where they disagree before I trust the mean.
5. The agent running the loop reads the judges' reports side by side and keeps the findings that at least two of them made, whether there are two judges or three. Judges word things differently, so this is matching by meaning and it's a judgement call. A builder agent gets that list, makes the changes, and the next round starts from new screenshots.

The other products' screenshots are frozen after the first capture, so later rounds are measured against the same thing.

The brief is short. This one is illustrative, but it has the parts mine has:

Answer directly. Do not load skills, spawn sub-agents or ask questions.
 
You are judging products A to G from the screenshots in this folder.
Score each product from 1 to 10 on every category in RUBRIC.md.
For each score, give one sentence of evidence that names the screenshot.
Then list the three changes that would most improve each product.
Return JSON: { product, category, score, evidence } and { product, fixes }.

The builder's brief is built from the judges' reports. Again illustrative:

Fixed direction (do not change): [my own decisions about this design]
 
Problems raised by at least two judges, most important first:
1. [the problem, in a sentence, with the screenshots it shows up in]
2. ...
 
Fix the problems. Don't add features nobody listed. Describe what you
changed for each one, then stop.

Write your own rubric#

I write the rubric before round one and don't touch it afterwards. For that feature it had seven categories, each scored 1 to 10, with a sentence or two saying what to count. One was about how quickly someone gets to something useful, and it told the judge to count screens and required inputs, and that a short flow only scores well if it reaches something real. Others covered visual polish, phone ergonomics and trust. This is the copy category as written:

copy. Clarity of copy: every line short, plain and specific; no narration of
the interface, no jargon; it's obvious what each button does.

Yours will need different categories, so this is where I'd spend the time. A few questions to start from:

• What would make a user choose a competitor over this? Each answer is a candidate category.
• What do I already know my agents get wrong? Mine over-explain, so I ended up adding a category for restraint.
• Which products should it be judged against? I use the ones my users would compare it with, plus one from outside the category that does the job well.
• What can I capture the same way every round? For interfaces that's screenshots at fixed sizes. For an API or a document it would be the output itself.

The rubric also sets the stopping rule in advance. Mine has two parts. The target is that my build leads every other product overall, with no category below its round-one score. Without the second half, a builder can buy a higher average by quietly making one thing worse. The exit is a plateau: once the target is met, I stop when the new build gains less than about 0.3 over the previous one, ideally with both judged in the same pack. It's a rule of thumb. The same screenshots can move by that much from one round to the next, so I don't count a smaller gain as progress.

If I change the rubric, that starts a new comparison. Scores from before and after the change don't go on the same chart.

What one run looked like#

The build led from round one, so the lead didn't decide when to stop. Round two raised the overall score, but two categories dipped below their round-one scores, which broke the second half of the rule. Round three recovered both, and gained 0.24 on round two, which is under my plateau threshold, so the run ended there. Trust gained two points over the run, the most of any category. The two that dipped in round two were visual polish and phone ergonomics. Phone ergonomics finished exactly where it started, and so did the category about leading people towards real use.

The reference screenshots, which never changed, are the noise check. The same screenshots of the production version scored 6.00, 5.84 and 6.03, and no other product moved by more than 0.39. Against that, 0.57 over three rounds is a modest gain, and I wouldn't read much into a single round's 0.2. On this run each pack held only the latest build, so the round-to-round comparison leans on those references and the 0.24 is a looser reading than my rule asks for. Putting the previous build in the pack came later.

I haven't measured what a round costs in money, and I didn't time them. The judges and builders run on my existing subscriptions, and the limits I've actually hit were on my own machine: parallel builders filled the disk and most of the memory, so now at most two of them run a dev server at once. One judge worked through a large pack for four hours without returning a verdict. I now cancel any judge that has run for 90 minutes and carry on with the verdicts I have.

Keep the builder and the judges apart#

I don't trust a model's review of its own work in the same session, so the judges never share a context with the builder, and I use at least two judges from different providers. The run above had three: Claude Opus 5.5, GPT-6 Astra and GPT-6.1 Sol. Lately it's usually Opus and Astra.

The separation isn't complete. On one run the builders were Opus and Astra and the judges were Opus and Sol. Astra never scored its own build. Opus did, in a fresh session with nothing to say which build was whose.

I use the same shape for code review, with findings in place of scores: two models from different providers, fresh sessions and several rounds. Tests still come first there. I only use model judges for things I can't measure with a program, which for me is mostly design.

Use whichever models you have. What matters is that there are at least two, that they start fresh each round, and that you check now and then whether one of them is consistently the odd one out.

Where the judges mislead#

Scores only compare inside one pack. On a different run the same screenshots moved by 0.4 to 0.5 between packs, and a reference product's score drifted upwards over the rounds. So I put the previous round's build in the pack next to the new one where I can, and compare those two. When the frozen references move by more than half a point, I report the gap to them and ignore the absolute number.

Judges contradict themselves from round to round. One round a tall text box was "wasteful". The next round a short one was "wasteful" too. If the builder acts on every comment, it oscillates, so a fix needs two judges behind it. It also goes to the builder as a problem, without a number attached. One judge comment turned into a 760-pixel cap on a list, which then cut off its contents.

They reward explanation. Agents add helper text everywhere, and judges mark it up as clarity. I ran OCR over the screenshots and the word count per screen had gone from 114 to 190 in four rounds, while the scores rose. On the next run I added a restraint category to the rubric, and a text-heavy build scored about 1.3 points lower overall than it had without it. Those two scores came from different rubrics, so the word count is the firmer evidence.

They reward features as well. Judges kept recommending an extra comparison view that I didn't want, because it made a simple choice harder. My own decisions now go into the builder's brief as fixed direction.

They judge whatever is in the screenshot. A floating developer-tools button in local captures was reported, round after round, as part of my interface overlapping the content.

Their taste also isn't mine. After 21 rounds on one page, the judges put the hill-climbed version about 1.5 points above the one in production. I gave both 6 out of 10. It was tidier, but I didn't think it was better. Small polish rounds had flattened out, and the loop had only ever been asked to improve the design it started with. That run is where the plateau rule came from. Since then, between some rounds I answer screenshot questions about specific contested choices, such as which side a panel opens on, which is more use to the builder than a score from me. And every so often a round is structural: three bold variants judged together, with no attempt to improve the current design.

When the scores and my own judgement disagree, I work out why, then change the builder's brief for the next round, or the rubric for the next run.

Try it on one screen#

If you want to try it, pick one screen. Capture it and two or three products you'd like to beat, write five categories of your own and a stopping rule, and give the pack to two models that didn't build it. Take the problems both of them raise, have the builder fix those, and judge the old and new versions together. Then rate the before and after yourself, without looking at their scores, and see how far apart you are.