Nobody has fully solved it, but the setups that work all do the same thing: make the checker different from the builder, and make "done" something the agent can't redefine halfway through.
For verification in general:
Write the acceptance criteria before the build starts and keep at least some of them out of the builder's view. That's StrongDM's holdout idea, and you can do a cheap version in any repo by keeping a scenarios file the build agent never gets in its context.
Run checks from a fresh session, ideally on a different model. A verifier that saw the code being written makes the same assumptions the writer did.
Lean on deterministic checks before AI judges: types, lints, tests, a real build. Then check the tests themselves. Mutation testing (break the code on purpose and see if a test fails) is the quick way to catch "assert true" style tests.
Keep changes small. A reviewer can actually check one stage per commit. A 2,000-line diff gets skimmed whether a person or a model reads it.
For bad UI decisions specifically, I think the main problem is that agents edit UI code without ever looking at the result. What helps:
Make it look. Have the agent run the app, take screenshots in a real browser at phone and desktop widths and in light and dark mode, and attach them to the stage. Review the screenshots, not the diff.
Shrink the space of choices. A small design system (color and spacing tokens, a handful of components) plus a short written list of rules like "no new modals," "no new colors," "match existing page X" stops most invented UI. Agents follow concrete prohibitions much better than "keep it clean."
Give it a reference. "Make this look like the threads page" beats any description, and it gives the reviewer something to compare against.
Use visual regression diffs so unrelated pages that changed get flagged automatically.
Keep the human on UI taste. The one thing models are still weak at judging is whether something feels right to use, so that's where a person's review is worth the most.
In Railspike terms, that would mean the Define step lists the scenarios and screenshots required for done, and the Build stage can't be marked finished until they're attached.
#ai #agents #softwarefactory #ui
#
For verification in general:
Write the acceptance criteria before the build starts and keep at least some of them out of the builder's view. That's StrongDM's holdout idea, and you can do a cheap version in any repo by keeping a scenarios file the build agent never gets in its context.
Run checks from a fresh session, ideally on a different model. A verifier that saw the code being written makes the same assumptions the writer did.
Lean on deterministic checks before AI judges: types, lints, tests, a real build. Then check the tests themselves. Mutation testing (break the code on purpose and see if a test fails) is the quick way to catch "assert true" style tests.
Keep changes small. A reviewer can actually check one stage per commit. A 2,000-line diff gets skimmed whether a person or a model reads it.
For bad UI decisions specifically, I think the main problem is that agents edit UI code without ever looking at the result. What helps:
Make it look. Have the agent run the app, take screenshots in a real browser at phone and desktop widths and in light and dark mode, and attach them to the stage. Review the screenshots, not the diff.
Shrink the space of choices. A small design system (color and spacing tokens, a handful of components) plus a short written list of rules like "no new modals," "no new colors," "match existing page X" stops most invented UI. Agents follow concrete prohibitions much better than "keep it clean."
Give it a reference. "Make this look like the threads page" beats any description, and it gives the reviewer something to compare against.
Use visual regression diffs so unrelated pages that changed get flagged automatically.
Keep the human on UI taste. The one thing models are still weak at judging is whether something feels right to use, so that's where a person's review is worth the most.
In Railspike terms, that would mean the Define step lists the scenarios and screenshots required for done, and the Build stage can't be marked finished until they're attached.
#ai #agents #softwarefactory #ui