Text column edges

When you do decide to delegate

Sometimes, it does make sense to hand over the problem to an agent: it could be too big, not valuable enough, time could be too short, or you just want to get a first prototype out to explore a problem end-to-end.

This one is probably an essay by itself: adapting old-school best practices when you have infinitely fast stochastic intelligence on top. The idea here is to reduce the risk in making changes, flattening the vibe-limit curve and making it possible to make bigger changes safely.

Beyond giving the agent some means to verify itself against the goal you're aiming for, I find it very useful to:

Observability: Invest heavily in observability, both black-box and white-box, and make sure to keep a direct handle on the metrics instead of solely relying on the agents' view. As a habit I prefer to triangulate onto a behavior with multiple sources of metrics, and that becomes even more important with systems rapidly becoming opaque.
Non-functional requirements: Don't forget to observe resource utilization and other non-functional requirements: these can be canaries for unexpected behavior.
Deterministic validation: Build mechanical straitjackets for the model: have held-out pieces of code or deterministic tools for validation and observations as a hold-out against the model misunderstanding or glitching.
Shadow Traffic: Make it possible and cheap to run a deployed candidate against shadow traffic with no side effects: keep it as similar to production as possible, and this can be the final gate to prevent surprises.
Style guides: Give the model a style guide based on your existing code and work. I've found Claude's output far more to my taste after having the models generate a style guide based on all the code and blog posts I've written in the past.
Fast deploys: An often maligned motto at Facebook used to be Move Fast & Break Things: that, too, is an intentional tradeoff to make. If you can't be sure of quality, make it very easy to deploy code as well - so changes can be shipped and reverted extremely rapidly.
Rethink what's possible: It's surprisingly easy to write and rewrite systems from scratch rapidly as you learn. Build to throw, evaluate, and repeat to get something knocked into great shape - I've built projects in multiple iterations, giving the agent the previous iteration and its limitations as initial context and constantly finding myself able to deeply increase the quality. This doesn't work as well when you're punishing your code reviewers though, so make sure to align with your team, or simply don't get the first several iterations reviewed.

knlb.dev | Resisting Cognitive Surrender
https://knlb.dev/essays/surrender

by ilyag on

Reply
#
…
Options