In its current state, LLMs produce code that is on par with the average developer. This is not necessarily bad since most firms can only afford to hire average developers, difference in quality is negligible. And on the flip side, by producing the code it does, AI also raises the floor by eliminating code smells typical of below-average developers, so there's that.
We've been scaling quality alongside throuhgput for decades (at least since I joined the biz in 1999). AI is a large step forward in the automation game, but it's not new, nor are it's challenges.
Are other orgs not relying on the fundamentals of software engineering and adapting to this form of generated code?
PS: I don't allow one-shots. Same decomposition and quality assurance motions. The timeline has just shrunk, not the steps, nor the focus. The medium for documenting and governamce have shifted, same strategies and problems.
> But you don’t care about code quality anymore right.
Why did we ever? Because it made it more feasible to maintain and reduced bugs. Well structured, designed code bases are simply far easier to work with.
I'm sure we've all had greenfield AI projects where we just let it go nuts doing whatever it wants, then all of a sudden you need what should be a stupid easy change, and it just falls over completely or becomes borderline impossible. Each change needs 5+ other changes until you basically can't touch the thing. The difference is, AI will keep trying until you run out of money or you give up, but this way of programming is bad and we labeled it bad for a reason, and I don't know why as an industry we're thinking it's no longer important.
"We need formalized processes, and proven, repeatable control mechanisms to enforce the quality of whatever is coming out of these tools. These do not exist yet. They might emerge one day, but they have not yet3. Until then, I won’t just lay into it blindly. If you consider yourself a software engineer, you should probably give it a minute too."
The same argument cuts both ways. I have seen some horrid code (that I have had to fix) made by humans.
Yeah, but that bad code also had the chance to be caught and fixed in review. If you get rid of reviewing code (human-written, AI-written, squirrel-written, whatever) you get rid of that chance. Hence the not trusting it blindly part.
In my experience, the clankers can write great code when you give it a hole with the right shape in the bigger picture of code that looks how you like. I had the experience of setting a clanker loose on Wizard to go implement a Wasm proposal[1] that had reached stage 4, and it wrote the canonical V3 interpreter implementation, fast assembly interpreter implementation, and JIT support, as well as code validation and fitting the rest into the system, in just 40 minutes, complete with tests. The VM code was canonical. There wasn't a line I would have written otherwise. In the end it was only 144 lines. But it was the right 144 lines.
Clankers can do things right when they are given clear instructions and good context. They go bad when they are given agency and no oversight. What I have seen repeatedly is that even small amounts of correction to turn a refactoring this way or that way or make a better modularity decision, or to enforce a boundary that wasn't written down, helps a lot. Clankers don't know about your implicit information hiding going on. They don't know you've been working for years to get rid of a bad code smell that spread faster than you could stamp it out, or that you've been working on the same codebase for 10+ years and only now understand how you'd like to structure it.
Some people use the clanker to get something out of nothing. As engineers, we should be using them to engineer the heck out of things that we didn't have human effort available for. Things like refactoring, testing, cleaning up documentation, doing optimizations, simplifying, introducing proper interfaces, removing improper interfaces, etc.
[1] The wide-arithmetic proposal, which added 4 instructions with very well-specified behavior.
I'll miss the era when we could care about the craft of coding. I'll probably continue to do so in my spare time. Hobbyist woodworkers and hand-crafted furniture makers still exist, even if 99% of all chairs are made by machine now.
But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
Your job is no longer to produce code - it's to build systems that produce code. Tests and CI and Linters and custom lint rules and adversarial review and agent docs and automations and the next dozen tools we come up with as part of figuring out best practices around all of this. If you're getting woken up by agent-written code, the fix isn't to review every line of code - the fix is to improve your system so that class of error doesn't get written.
It's understandable if you don't want to do that work. It's almost entirely different from the work of reading and writing code. But if you don't, you're delivering less business value than someone who will. They don't employ hand woodworkers at Ikea.
> But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
That is false. Code craft is still very much an aid to business value. If you just turn an LLM loose and don't review its work, you get a buggy mess over time. If you use an LLM, you must review and understand each line of code or you are sabotaging the quality of your work.
My whole thesis is we've been doing this with humans since the beginning. I'm not finding anything unique about LLM generated code. You gotta define standards the "developer" can read and implement. You gotta review the code for compliance. You gotta test the code for quality. You gotta perfomrance test for scalability. Who gives a flying turkey whether a meat-puppet (human) or a clanker (llm) wrote the code? It's still just code.
In its current state, LLMs produce code that is on par with the average developer. This is not necessarily bad since most firms can only afford to hire average developers, difference in quality is negligible. And on the flip side, by producing the code it does, AI also raises the floor by eliminating code smells typical of below-average developers, so there's that.
We've been scaling quality alongside throuhgput for decades (at least since I joined the biz in 1999). AI is a large step forward in the automation game, but it's not new, nor are it's challenges.
Are other orgs not relying on the fundamentals of software engineering and adapting to this form of generated code?
PS: I don't allow one-shots. Same decomposition and quality assurance motions. The timeline has just shrunk, not the steps, nor the focus. The medium for documenting and governamce have shifted, same strategies and problems.
Am I really on an island with this?
> But you don’t care about code quality anymore right.
Why did we ever? Because it made it more feasible to maintain and reduced bugs. Well structured, designed code bases are simply far easier to work with.
I'm sure we've all had greenfield AI projects where we just let it go nuts doing whatever it wants, then all of a sudden you need what should be a stupid easy change, and it just falls over completely or becomes borderline impossible. Each change needs 5+ other changes until you basically can't touch the thing. The difference is, AI will keep trying until you run out of money or you give up, but this way of programming is bad and we labeled it bad for a reason, and I don't know why as an industry we're thinking it's no longer important.
"We need formalized processes, and proven, repeatable control mechanisms to enforce the quality of whatever is coming out of these tools. These do not exist yet. They might emerge one day, but they have not yet3. Until then, I won’t just lay into it blindly. If you consider yourself a software engineer, you should probably give it a minute too."
The same argument cuts both ways. I have seen some horrid code (that I have had to fix) made by humans.
Yeah, but that bad code also had the chance to be caught and fixed in review. If you get rid of reviewing code (human-written, AI-written, squirrel-written, whatever) you get rid of that chance. Hence the not trusting it blindly part.
In my experience, the clankers can write great code when you give it a hole with the right shape in the bigger picture of code that looks how you like. I had the experience of setting a clanker loose on Wizard to go implement a Wasm proposal[1] that had reached stage 4, and it wrote the canonical V3 interpreter implementation, fast assembly interpreter implementation, and JIT support, as well as code validation and fitting the rest into the system, in just 40 minutes, complete with tests. The VM code was canonical. There wasn't a line I would have written otherwise. In the end it was only 144 lines. But it was the right 144 lines.
Clankers can do things right when they are given clear instructions and good context. They go bad when they are given agency and no oversight. What I have seen repeatedly is that even small amounts of correction to turn a refactoring this way or that way or make a better modularity decision, or to enforce a boundary that wasn't written down, helps a lot. Clankers don't know about your implicit information hiding going on. They don't know you've been working for years to get rid of a bad code smell that spread faster than you could stamp it out, or that you've been working on the same codebase for 10+ years and only now understand how you'd like to structure it.
Some people use the clanker to get something out of nothing. As engineers, we should be using them to engineer the heck out of things that we didn't have human effort available for. Things like refactoring, testing, cleaning up documentation, doing optimizations, simplifying, introducing proper interfaces, removing improper interfaces, etc.
[1] The wide-arithmetic proposal, which added 4 instructions with very well-specified behavior.
I'll miss the era when we could care about the craft of coding. I'll probably continue to do so in my spare time. Hobbyist woodworkers and hand-crafted furniture makers still exist, even if 99% of all chairs are made by machine now.
But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
Your job is no longer to produce code - it's to build systems that produce code. Tests and CI and Linters and custom lint rules and adversarial review and agent docs and automations and the next dozen tools we come up with as part of figuring out best practices around all of this. If you're getting woken up by agent-written code, the fix isn't to review every line of code - the fix is to improve your system so that class of error doesn't get written.
It's understandable if you don't want to do that work. It's almost entirely different from the work of reading and writing code. But if you don't, you're delivering less business value than someone who will. They don't employ hand woodworkers at Ikea.
Tell me again how the code quality of the Linux kernel doesn't matter, only its business value.
> But code craft not the job. The job is to produce business value. Code craft used to aid that, but now it's an impediment.
That is false. Code craft is still very much an aid to business value. If you just turn an LLM loose and don't review its work, you get a buggy mess over time. If you use an LLM, you must review and understand each line of code or you are sabotaging the quality of your work.
My whole thesis is we've been doing this with humans since the beginning. I'm not finding anything unique about LLM generated code. You gotta define standards the "developer" can read and implement. You gotta review the code for compliance. You gotta test the code for quality. You gotta perfomrance test for scalability. Who gives a flying turkey whether a meat-puppet (human) or a clanker (llm) wrote the code? It's still just code.