I have been doing this process by hand. Try to do something. Observe friction. Document how to avoid it. Rinse and repeat. It works and raises the quality going forward. Most importantly, the vast majority of my tokens are spent moving my project forward.
I would be extremely leery to use an automated hill-climbing loop, repeating the same inputs, and burning tokens on the same task over and over. That’s not going to move my project forward, and I don’t have unlimited tokens to spend.
Worse, how can I be sure that these agents will follow deterministic definitions of friction and priorities for removing it? I can imagine “write less tests” would remove a lot of friction. I see this just adding another layer, where I now would be documenting the system that writes my documentation. I learned how that is likely to turn out from playing the telephone game as a child: poorly.
To clarify, this does not require the hillclimbing to be unattended, especially if you are setting up the system for the first time!
This mainly automates the process that you are doing by hand. If you have your own definition of friction and priorities, you can tell this to your agent and it will fit into the CLI loop. You can review any change that is suggested, and spot-check transcripts if you feel the need.
It's like most AI systems in that you can have it earn more autonomy over time, as you add guidance and guardrails.
What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?
Active work should mostly be hillclimbing, or at least more exploratory. The search space is very large and you typically find a lot more low-hanging friction than you'd think by just running a broad battery of sessions. And I'd pair active hillclimbing work with feature launches/product work - essentially part of releasing new things is QA-ing them against coding agent workflows.
The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.
The search space being large feels like a fundamentally hard and costly problem to solve for. Where do you recommend folks start given such a high number of possible scenarios?
I have been doing this process by hand. Try to do something. Observe friction. Document how to avoid it. Rinse and repeat. It works and raises the quality going forward. Most importantly, the vast majority of my tokens are spent moving my project forward.
I would be extremely leery to use an automated hill-climbing loop, repeating the same inputs, and burning tokens on the same task over and over. That’s not going to move my project forward, and I don’t have unlimited tokens to spend.
Worse, how can I be sure that these agents will follow deterministic definitions of friction and priorities for removing it? I can imagine “write less tests” would remove a lot of friction. I see this just adding another layer, where I now would be documenting the system that writes my documentation. I learned how that is likely to turn out from playing the telephone game as a child: poorly.
To clarify, this does not require the hillclimbing to be unattended, especially if you are setting up the system for the first time!
This mainly automates the process that you are doing by hand. If you have your own definition of friction and priorities, you can tell this to your agent and it will fit into the CLI loop. You can review any change that is suggested, and spot-check transcripts if you feel the need.
It's like most AI systems in that you can have it earn more autonomy over time, as you add guidance and guardrails.
This is definitely something the companies most focused on agent experience are thinking about:
- https://supabase.com/evals - https://nextjs.org/evals
What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?
Active work should mostly be hillclimbing, or at least more exploratory. The search space is very large and you typically find a lot more low-hanging friction than you'd think by just running a broad battery of sessions. And I'd pair active hillclimbing work with feature launches/product work - essentially part of releasing new things is QA-ing them against coding agent workflows.
The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.
The search space being large feels like a fundamentally hard and costly problem to solve for. Where do you recommend folks start given such a high number of possible scenarios?
Sorry to be pedantic, but it's iteratively. Your README even calls it Iteration.
Thanks for sharing.
They won’t stop until the entire information capacity of the universe is consumed by autogenerated Swagger docs
Fair enough, maybe when we use it internally it would qualify?
But what if you use recursion for iteration?