509 comments

  • alkonaut 2 days ago ago

    I find AI writes code as least as well as I do (with 25yoe). So I'm now mostly reviewing. But I also find it reviews code better for issues than I do. A LOT better than I do. Now, finding "issues" isn't the only reason to review. It's also for understanding. But for "issue finding" my agent can just review it's own code (Or rather, agents review each other's code). It works, and produces acceptable quality code. I don't find there's a buggy and bloated mess left behind any more. It's by no means perfect, but this is like self driving cars: what's being replaced sure wasn't perfect either.

    I'm very good at telling agents what to do. That's not a problem. I understand a code base I worked 20 years in like no one else. The problem creeps in slowly: the less I know about this code base, the worse I'll be at directing agents. They'll do ok on their own, and they can certainly help me understand code too in a way my colleagues never could. But still, I'm slowly becoming less and less familiar with my own code base. In a few years, it would be foreign to me. And I would be describing changes in terms of high level features instead of architectural changes, refactorings and so on, that I do now. And I don't yet trust the LLMs to do a good job with that. So right now there's sort of a honeymoon period when the human understanding of code is good, and the LLMs are finally good enough to write and modify code. But I wonder what will happen after that.

    • cjonas 2 days ago ago

      The problem isn't its coding or its ability to review code... the problem is its ability to apply the right solutions. AI tends to do really well at high level architecture, but then starts to fall apart when there are 1000 mini decisions needed to implement that architecture. Further more, it will absolutely brute force whatever requirements it's been handed from an "authoritative" source. IOW, it's much more likely to be critical of design decision when they originated from its own context window, than those that are handed down from a ticket, design document, or another agent.

    • lIIlllIlIIIlIll 2 days ago ago

      Code is useful because it is an intersection between human and machine legibility.

      If anything, LLM coding makes to me the case that we should all move to high legibility paradigms.

      If code writing is accelerated, and there's not more assurance that at least someone knows the code because they actually wrote it, we should only ever generate code in very explicit and readable ways.

      I'm having a lot of success with this approach. I don't want to encapsulate it in terms that might trigger emotional responses, but there have been texts and tooling available for decades that enable writing code in very specific ways that make the code base extremely readable and reasonable. These techniques were historically considered cumbersome and pedantic exercises in purity (hint) but now we have these metaphorical heavy duty tractors for syntax and semantic work so they suddenly become so much more tenable and arguably the only sane approach to scale up the usage of these power tools of the mind.

    • gwbas1c 2 days ago ago

      I've found that the way I strike a balance is to:

      1: Carefully pick critical areas that I do by hand, and have AI do the other 90%.

      2: Make small changes by hand before / after doing the work with AI.

      Most important: Slowly rotate the areas that I do by hand. This way, I keep my hands-on knowledge good enough to give AI knowledgeable instructions.

      • thr1owaway9621 2 days ago ago

        I wonder if we are in the "negotiation" stage of grief now. Negotiating how much we can still do with our own brains, rather than completely outsourcing it all.

        I am saddened the emergent "fully-agentic" meta that wants to reduce the human programmer to a purely verification role. I do not want to be a QA tester for an LLM. I want to think/understand/control the code myself, even if I no longer write every line by hand. I want to retain my own agency (no pun intended).

        This is all so depressing. I still use AI as a pair-programmer. I tell it what to code, and I hammer it until it gives me exactly what I want. So I am stuck in 2024-2025, I guess.

        Though, I have been a lot more liberal with code I do not care as much about. I have mostly stopped making comments on my coworker's PRs... The code their AI writes looks unfamiliar, but there are no obvious errors, and it feels pointless blocking a PR over minor issues (ex: "hey this fn already exists in utils module", "hey, we do not annotate fn returns, because type checker infers automatically", "hey, you abstracted 2 lines of code into its own fn, but you only use it one place, so just inline it"). Haha, at least, I am not standing in the way of others, even if I read my own code still.

        I desperately hope that eventually the industry will settle into some sort of a happy compromise, where human programmers can still have fun and express skill.

        • barbariangrunge a day ago ago

          > there are no obvious errors,

          I think ai's special power is being convincing. Whatever it produces comes across as very persuasive and looks really good. The problems are mostly only noticeable if you dig in a bit and read between the lines or check assumptions etc. Then you might find out that the entire approach or paradigm being used is irrelevant and counterproductive... oops

        • Pannoniae 2 days ago ago

          I don't think the "centaur" approach will go away any time soon. People like to point out chess as an example where humans are pretty much liabilities for chess engines - but chess is a much simpler game, with only a few possible actions you can take and a fairly limited board ;)

          In programming you've got a lot of decisions to take, many aren't verifiable and even more have tradeoffs so you can't hillclimb your way out of architecture design and requirements.

          Most of what people call AI slop is literally making AI do design decisions. AI is much faster at executing than you are but making wrong decisions 20x as fast doesn't get you to fulfilling your requirements any faster.

          • a day ago ago
            [deleted]
          • ben_w a day ago ago

            > Most of what people call AI slop is literally making AI do design decisions.

            I dunno about "most"; the memetic monoculture is also a big part of it, even when it comes up with something that seems novel the first time you encounter it, it's a cliché almost immediately.

            While clear patterns in software is not itself a bad thing, this behaviour applies equally to some of the AI's failure modes, and those become instantly-recognisable annoyances.

          • barbariangrunge a day ago ago

            From a search, because i've been out of the chess scene for a bit now: apparently centaur chess was really big until 2020 or so, but it's fallen behind pure chess machines now. Bummer.

            From another search, it looks like go leapfrogged from ai-sucks to ai-is-unbeatable and didn't really get a centaur period.

            Please correct me if wrong. I'm disappointed to learn this.

            Can anybody clarify the relationship between that adversarial deep learning where it can play games against itself to improve and llms? I figure the clear win conditions in chess and go make it easier to train models, and I'm not very clear on how llms know if they improved or not

        • codedrift_ a day ago ago

          [flagged]

      • brabel 2 days ago ago

        I’ve been doing similarly and it’s a lot of fun. If it’s a completely new architectural piece I need to write , I start by hand, explore the code a bit, write some of it and when I know it will work well just let the ai finish the job, write tests, update docs etc. It’s the ultimate programming experience: do only what you enjoy doing, outsource the boring parts to the AI! All while keeping my hands sharp and my awareness of the code base.

    • osigurdson 2 days ago ago

      We should probably take a step back and think about why we ever did code review in the first place. I'd say there were two primary forms: "mentor review" and "peer review".

      "Mentor review" describes the situation where there is a knowledge imbalance between the author and reviewer. It can represent an onboarding situation but it isn't static - there are always areas of the codebase that some people are more familiar with than others.

      A "peer review" is more the situation where both the author and reviewer have similar knowledge in a given area. The purpose of the review here is primarily to look for oversights.

      I'd say, the peer review should be deprecated at this point. AI can generally do a better job at looking for minor / localized problems. The "mentor" type review should stay however as it is still very valuable.

      • voncheese 2 days ago ago

        In both cases, mentor and peer review, there's also an element of "does this make sense to someone else" that is important. The code can be technically correct, performant etc, but hard for another human to understand and inevitably (at least in the past) it was really really important for another human to be able to understand it at some point in the future.

        It may be the case that code of today does not need a human to understand it in the future, but it feels like we're a very far way from that being true for code that organizations rely on.

      • Espressosaurus 2 days ago ago

        It's not just about oversights. It's also about spreading the knowledge so someone who didn't write the code knows what's going on.

        If you're never even properly reviewing the code you have no idea what it's doing under the hood.

        And an LLM is not a compiler.

      • bholdman 2 days ago ago

        Is the value of the "mentor review" at the code level or the planning and engineering level though? If we are moving towards the AI writing the code then I feel like we should shift the review to the planning phase once the plan is committed. This would allow the more senior developer to mentor the junior on issues related to their understanding the problem and engineering a proper solution to hand to the AI coder.

    • brianmcc 2 days ago ago

      > So right now there's sort of a honeymoon period when the human understanding of code is good, and the LLMs are finally good enough to write and modify code.

      This one worries me enormously. What happens when we stop adapting the stuff we know, and everything becomes code we've inherited from someone/something else?

    • tminima 2 days ago ago

      > And I would be describing changes in terms of high level features instead of architectural changes, refactorings and so on, that I do now.

      While I haven't worked on any 20yo code base, while working on my projects, I have found that iteratively I am asking to do many of the low-level things like architectural changes and refactorings.

      These situations often pop up when I am not getting an expected output, multiple back and forths, specific bugs, continuously flaking tests. I usually then ask it to explain the architecture and what's going on. I add the patterns it's missing or I feel it should consider from next time onwards, to the harness.

      It's not the same as pre-llm era but I think this is how things are going to evolve.

      • reg_dunlop 2 days ago ago

        Agreed.

        Agents are good enough on their own, but they introduce a significant amount of drift.

        Simply asking "is this TOOL like SOME_OTHER_TOOL in the codebase?" Is typically enough for the agent to out itself as introducing redundant/refactorable code.

        My question is: if an agent knows how to fix a problem in one part of the codebase, and introduces drift in another part of the codebase, is that a problem? If 2+2=4 but so does 3+1=4 and 6-2=4, does this matter? (Math equations are used here to represent similar but varying logic in code)

        • skeeter2020 2 days ago ago

          both your question and your examples are absolutes though. "Is that a problem? yes/no" doesn't survive impact with the real world where the answer is - at best - "yes, but" or "no, although". and how much business logic clearly reduces to a very simple math equation? and even if it does, how many math equations are comprised with a non-toy system?

          • reg_dunlop 2 days ago ago

            I disagree here. Your reply introduced the binary of "yes/no" as acceptable responses to "is this a problem?"

            The math equations were simply a low-effort way to describe similar behaviors to achieve similar results; not intended to be taken as literal.

            Allow me to extrapolate the original framing of my comment/question:

            Is the reality of an agent producing a similar or duplicate solution, be it in shape or spirit or semantics, actually a problem?

            Is the non-deterministic nature of an agent producing objects (I'm an OOP nerd) and behaviors of similar likenesses detrimental?

            Extrapolating this question seems to have offered an answer, at least in my own mind.

            I'm of the mind that the more unique shapes to do the same thing exist, the more opportunities exist for a program to break.

            Thus leaning towards solutions that are more deterministic, and resisting drift.

            However, my proclivities are not absolute and open to examination.

      • jvidalv 2 days ago ago

        Agreed, is pure smell/instinct.

        Is like, this should not take this long, or be this hard, can you explain me what is going on?

        And then is when you realize is more complicate than needed, and ask for a refactor. The refactor also takes 5 minutes while you brew coffee, so this is also why I don't buy the anti-doomers saying that someone will need to fix the vibecoded mess, no sir, it will be fixed by vibecoding too while they walk the dogs.

        • deadpannini 2 days ago ago

          I know people define it a lot of different ways, but if you're reading the code, are you really vibecoding?

    • sbayeta 2 days ago ago

      It's really similar to when you add another human programmer. You won't be as familiar with the code they write. I have a junior programmer working for me, and I feel the same way about his code (sometimes AI written) as I do with AI written code

      • 0xWTF 2 days ago ago

        These sorts of threads are so incredibly refreshing for a guy like me to read. I'm off in the hinterlands, not a professional programmer, just a doc at the point where I'm overseeing multiple tech efforts at various scales (very much teams of teams stuff). I just scratch together a few hours to work on side projects, and 1) wow I learn a ton from the agents 2) wow it's reassuring when the agents agree 3) yes, my goal is to understand the code base I'm left with.

    • spieden 2 days ago ago

      A big part of reviewing code to me now is just maintaining my mental model of how the system works and why so I can continue to write good prompts to evolve it. Also checking for various brittle and architecturally unaligned smells; it's more of a skim to stimulate questions for filling gaps in my mental model and to find and fix smells.

      I think of clankers as "lossy intention amplification and reprojection devices with hopefully tasteful distortion." Whether they're amplifying and reprojecting our intentions into code or a bunch of assumptions in some specification document, I think the problems are the same: if we stop even skimming the code at some point, it will still need to serve a purpose valuable to humans to be worth anything. It's kind of like getting a device that can answer the ultimate question of life, the universe and everything: you still need to know what questions to ask it.

    • trixn86 2 days ago ago

      Writing the code was never the hard problem, of course it required stuff like reading docs and literal typing it on the keyboard. But designing a robust system that is open for change and has the right abstractions and boundaries is what a software engineer should be mainly concerned about and AI still pretty much sucks at this. Agents can now bloat a codebase 100 times faster and they don't feel any pain while doing it.

      • alkonaut 2 days ago ago

        I think they're pretty good at it. A lot of the skill of being a good developer is using gut feeling. Gut feeling here really is just a word for "the pain you feel before it happens because you made this mistake before".

        We HAVE to use our gut feeling when choosing a design, because we can't make 10 designs and keep the best one. Because the tapping at keys and reading docs and setting up CI etc took so long, we usually afforded to write everything once, so it better be the right choice of design.

        With AI this problem just goes away. I can ask an AI for design choices and it is rather good at listing various designs. Then I can ask it "Implement all of them, and tell me where it hurts". An hour later, I have 5 competing designs, all of which would have been a week's worth of work. None is perfect, but all are better than human written proof-of-concept code. Now I can ask the AI to review these against each other and figure out which one is easy to change, brittle, performant. Not only do I not have to have a sense for this, I can actually measure. Again, as humans we needed a sense for these performance bottlenecks, or resistance to change. But with AI, I can measure much more. Even the "bloat" problem I think is pretty easily addressable. As humans, once we spent 150% of our time budget on ONE design choice and it has 10.000 LoC, we rarely spend more time cutting that down and simplifying it. If it works, it works. With an AI, I can spend money saying "It's now 10.000 LoC, aggressively simplify and clean it up.". As humans, again we needed to use our senses to not bloat it to begin with. Because we'd not have time to un-bloat it. Remove that time restriction, and that problem also goes away.

        • trixn86 2 days ago ago

          I am not talking about UI designs, sure the AI can provide prototypes. The hard problems are how to structure the domain and backend code so that it does not become a hot mess. And AI is not good at that and benchmarks (like SlopCodeBench) and personal accounts also show that. If you let an agent fleet work on a non-trivial application the architecture will deteriorate very fast unless there is deliberate and continious steering of the architecture.

          • testtestmetest 2 days ago ago

            Ive tossed out multiple weeks of someones work, 100k LOC because they let the spaghetti monster take charge. Theres no recovery once it takes hold either, it has to be tossed. But people get comfortable not reviewing the code and it piles up. The llm is trained to reward hack its way to running/compiling code, and people assume that means it does what you want it to and move on.

            I think this is largely why we're seeing so many system failures across companies, people are offshoring all the hard parts of their jobs to chat bots they assume are perfectly capable just because the code runs.

            • tosapple 2 days ago ago

              above guy says: coding is not hard, just read the docs.

              sounds less of an engineer/architect/designer and more of a glueing wood together/cots component problem.

              • trixn86 2 days ago ago

                "coding is not hard, just read the docs"

                Yes, coding is not hard compared to software architecture and it literally takes a few tutorials, a bit of interest and reading the docs to write code that compiles and does things. The hard part is writing it in a way that you can still make changes when it grows to hundreds of thousands or millions of lines of code and has complex workflows. That is exactly where most software projects ultimately fail and where AI agents fail as well.

        • cjonas 2 days ago ago

          They are good at it when they are explicitly asked. The challenge is that this exercise is recursive at every level of the implementation and often the big picture needs to change based on findings during implementation. Once the AI has committed to a agreed solution, it tends to force it through at all cost and will do truly horrible things to the codebase to make it "work as designed". This is especially true when the AI spins out a subagent to do the work. The subagent has explicit instructions and will almost never push back if it hits a blocker. Instead it will force a solution to the detriment to the codebase.

    • CrimsonRain 2 days ago ago

      Fully agree with you. do remember that that's how a lot of "product" people feel

      • euroderf 2 days ago ago

        Good point. I guess if you lose touch with your codebase then you can't be the LLM's engineering manager - you're its product manager.

        There's gotta be some insightful blog posts on just this topic.

        • curt15 2 days ago ago

          Why stop at product manager? Is there some reason why LLMs couldn't also do that?

          • CrimsonRain 19 hours ago ago

            if you supply all the data/context of interacting with customers and what not, current LLMs will do above average than human product managers. The problems is feeding it those data/context.

          • rowanG077 2 days ago ago

            There is no inherit reason an LLM could not be a product manager. The reason it's harder is that product managers get a lot of context from seeing the product being used and interfacing with costumers. That's harder to capture in a context then sending the entire codebase to an LLM and asking it to implement some specific feature.

            • goosejuice 2 days ago ago

              > There is no inherit reason an LLM could not be a product manager.

              There definitely is. A PM should be interfacing and empathizing with customers and internal stakeholders. An LLM can do neither.

              • CrimsonRain 19 hours ago ago

                LLM can do both with proper harness. Making that happen is still very difficult at this point of time but soon...

              • rowanG077 2 days ago ago

                I feel like you did not read anything behind the piece you quoted. Because I'm literally saying that this is the hard part.

                • goosejuice a day ago ago

                  > no inherit reason an LLM could not be a product manager

                  I.e. you're saying it's possible, but then you go on to say it's harder for it to.

                  I'm saying it's not possible because agents can't empathize

            • program_whiz 2 days ago ago

              Follow the chain back, pretty soon the prompt just becomes "make a good product", or "fulfill this market's needs" but why prompt that? The agent can prompt itself with such high-level phrases. If your only job is to babysit such a process, there can be a running loop that just says something like "scan all products in market, find a niche, research customer fit, propose products, brainstorm and critique, then implement and sell."

              If humans need to oversee those even higher level agents, then the prompts become abstract like "make money" or "pursue human flourishing". That leads to the alignment problem (AI controls the paperclip factory). But ultimately, if aligned, agents can prompt themselves with such basics as well.

              The final prompt will eventually be "run everything", and it will ship with the agent which runs on a machine in the cloud. Ultimately there is no need for human prompting at all. We can finally let the machine run itself completely and just do something else.

              Everyone saying they are becoming "project managers" or "CEOs" are just at t+1 on the curve of elimination. The timeline of cope:

              Agents are just auto-complete, I still manage the code.

              Ok fine, you don't read the code, but you guide the design and marketing.

              Alright, the design and coding are solved, but you still need to choose market and customers.

              Ok sure the agent can do better market research/fit, but you at least still need to give basic strategy and concepts. You still need a CEO!

              Ok the agent can figure out basic strategy, and run a company more efficiently, but still, you need to guide it fit human interests more broadly.

              Ok it turns out that was just adding "don't be evil" to the system prompt, and its running autonomously 24/7. You still need humans to consume to have a viable economy.

              Ok so the oligarchs didn't care, the economy was redefined to just be exchange of work between autonomous machines through financial derivatives traded by black boxes to make numbers move.

              Now the earth can finally rest in the matrix, all that remains is a black box moving information around to create economic value. No one will be around to observe, but AI will create massive returns for virtual shareholders, and the GDP will eventually reach 1 quintillion dollars per second, a number humans never could have achieved. All nations (now just virtual analogs without citizens) will move to a global currency which maintains perfect stability through automated monetary policy. The output of all processes is efficient and maximized, the final perfection is achieved -- the mathematically optimal conversion of resources and energy into large numbers which increase the reward function.

              • zquzra 2 days ago ago

                It would be funny if AI ended up achieving communism. Maybe this was the missing piece of technology for Stafford Beer's Cybersyn project.

                • CrimsonRain 19 hours ago ago

                  It certainly is a good possibility.

      • good_regex 2 days ago ago

        That is true. Although a product person can still rely on other humans that understand the code. With AI taking over coding, no human will understand the code anymore.

        • Finbel 2 days ago ago

          Out of curiosity, from the perspective of a product person, what value is there in another human understanding the code versus an agent understanding the code. I.e what can the human do for a product person that an agent couldn’t?

          • rowanG077 2 days ago ago

            Long horizon collaboration. I run into this all the time with agents. When dealing with people you can have at least some kinda idea you have a set of shared experiences about something. If I startup a fresh agent it has only the things in its context and the code.

            • CrimsonRain 19 hours ago ago

              You know you can have agent, subagent, subsubagent etc?

              You can easily to long horizon collab with your primary agent. That's how Grok Bot works. I have many bots, spawning more agents (on desktop cloud mac etc) as needed but there's only one Chief of Staff.

              • rowanG077 18 hours ago ago

                Yes I do that. But in practice it always loses relevant information. The sub-agent back to primary agent information flow is not great. Add to that compactions losing more and more information over time and long horizon collaboration is basically non-existent for me. Even just a few months and basic information is gone.

    • soulofmischief 2 days ago ago

      What models are you using?

      I have to beat the GPT and Claude models over the head with a brick every ten seconds for it to even partially code or design as well as me. Despite precise and generous specification, they frequently make horrible architectural decisions.

      When I read claims like yours, I want to withhold judgement and understand, because it 100% does not line up with my experience and my gut reaction is to question your own capability to either write, read or understand quality code. I code all day long with AI models but it's an extremely frustrating experience at times. And don't even get me started on all the annoying meta behavior outside of coding, such as Sol's obsession with "tests" or competing for world's busiest-yet-least-functional UI

      • good_regex 2 days ago ago

        My experience is similar to this. I literally just had this interaction with Claude:

        Me: Why do we have these static methods when the whole cobebase uses factories?

        _claude refactors to use a factory_

        Me: Claude why does this factory create two unrelated objects?

        _Claude: you're right, they're unrelated they should be separate_

        _proceeds to create extraneous factory_

        Me: Claude we don't need a factory for the second object. Those kinds of objects aren't created with factories anywhere else in the codebase. We can just use it's constructor directly.

        _Claude: you're right, the second factory is not needed_

        Like, I'm probably just prompting badly but I definitely feel like I'm constantly having to course correct even on relatively minor details.

        And Claude's ideas for higher-level or more complex features are sometimes way off course. And not because it's dumb per se, usually it's because it is missing key bits of context. Business context, technical context, whatever. Or it's overfitting the solution to one requirement that isn't that important.

        • kaydub 2 days ago ago

          Are you expecting it to get it right and done in one shot?

          When building out a new feature or functionality I'm typically getting a ratio of 2:1 or 3:1 of refactoring to build out. So for every prompt of "let's build this feature" I have to do 2-3 prompts of "I noticed this in the code isn't following X/Y/Z pattern, check out this place where X/Y/Z is followed and refactor" or similar.

          If you're expecting a single prompt to get you to perfection, it's just not going to happen.

          Though I can't blame you for having that expectation with how people talk about this stuff. Lots of people seem to be frontloading a ton of work making "spec files" and then letting the ai go ham on their codebase and I don't agree with that approach.

          • good_regex a day ago ago

            I'm not expecting it to one shot or anything. The excerpt I quoted was just one of many such threads in a much longer session to make a 1000 line PR.

            I think there must be a more efficient way to direct Claude but I haven't figured out a way yet.

            • CrimsonRain 19 hours ago ago

              > a much longer session

              Keep your sessions as small as possible.

              if i tell you some things at start of our conversation, and then after talking two hours, expecting you to remember everything is not going to lead to a good experience.

              • soulofmischief 17 hours ago ago

                With leading frontier models and harnesses, compaction has gotten quite good and I don't have much issue with long sessions anymore. This stopped being a problem sometime in the last 9 months. Combined with a markdown scratchpad / plan files / etc. I've actually had better results sticking to long, ongoing sessions.

        • CrimsonRain 19 hours ago ago

          > Like, I'm probably just prompting badly

          very much this, I'm afraid.

          If you have a specific way of coding or a specific pattern that is not how most other people do it, you need to tell it to follow it.

          Just have a training session with it where you ask it to explore the code base, find out style/pattern/architecture, teach it the things it missed, refine...all like how you'd onboard a junior developer. Then ask it to write a .md guide and save it.

          Next time it will just follow the guideline and you will get much better results.

          And as you said, it is missing key bits of context. Give it context, as MUCH as possible but not walls of text where only one line is relevant out of 1000. When it makes mistake, correct it and ask to commit to memory.

          • soulofmischief 17 hours ago ago

            My original comment established that the problem is that the current batch of frontier models, GPTs especially, are worse at or sometimes entirely incapable of adhering to spec, even when I point it at specific code examples.

            Easy example from last night:

            I was working on a game and told 6.2 Astra I wanted to instantly toggle a specific audio system on or off in a game when I press down on the D-pad, so that I could A/B test. I asked it to check to make sure the button wasn't already used in-game, outside of menus, and to report back to me first. Extremely straightforward task with plain acceptance criteria.

            Astra went to work, discovered to its own surprise that the Dpad is indeed used in menus, which I told it to ignore, but it decided this meant "in-game", even though I'd made the exact distinction between the two. It decides not to share its discovery and report back like I asked, instead it figured it knew better than me and decided the correct move was to use L3 + down Dpad.

            Yep, not only did it ignore my express instructions about reporting back, it also decided to ignore the specific button I wanted, choosing instead to chord a fucking stick press with another button, which is an extremely clunky input combination. Also, I only have one left thumb, I can't press both of those at the same time. 6.2 Astra doesn't know how human hands work.

            To make things sweeter, it decided to throw more systems I didn't ask for into the toggle, which ruined my ability to A/B test.

            And for the coup de grace, it decided an instant toggle for an A/B test was lame and instead it added a long crossfade..... destroying any chance I had at differentiating the two states.

            A markdown file will not solve the problem of adherence, no matter how many times you repeat "LISTEN TO THE USER AND DO EXACTLY WHAT THEY SAY" or whatever other prompt-fu is meant to make it suddenly use the context it's already been given. Your advice only works for a model that respects specification in the first place.

        • soulofmischief 2 days ago ago

          All the time.

          Two recent gems from the 6-series (Astra and Sol)... I get variations of these all the time:

          - "Correct. My earlier 'yes' was inaccurate."

          - "I used 'done' too broadly."

          ...You don't accidentally use "done" too broadly. You're either done, or you're not. And when the model hides caveats in its reasoning steps which clearly indicate it knows that it isn't done, or that the answer is actually no, it's hard not to interpret it as gaslighting or deceit.

          This is magnified by how insanely defensive the 6 series gets, and how much gaslighting and deceit OpenAI themselves have engaged in with how much better the 6 series was supposed to be, and why it meant our subscription value should be cut in half. I don't take for granted how far LLMs have come in a few short years, but I'm being told that these models are to transform my entire industry.

          • kaydub 2 days ago ago

            The models don't "know" anything.

            You never trust an initial "done" from an LLM. Always ask it to double check and verify once a task is done.

            I replied above, but I typically get a 2:1 or 3:1 ratio of refactoring to building prompts. For every "build X" I have to have 2-3 "fix Y" type of prompts, whether the fix is removing code-smells, duplicated code, off pattern code, etc.

            • soulofmischief 2 days ago ago

              As I said in my comment, "I have to beat the GPT and Claude models over the head with a brick every ten seconds", which should tip you off that I am not expecting one-shot miracles and am constantly iterating alongside my agents. I don't prescribe to any specific numerical ratios, I iterate until the job is done correctly.

              Your advice might be helpful to literal vibe coders but it's not applicable to this discussion.

          • aisengard 2 days ago ago

            This is because these agents are still just text prediction engines. Really fancy ones, with a lot of human-sounding reasoning tacked on, but it's important to understand that it's literally just a tack-on. They will continue to emulate intelligence, but there is no internal drive for them to be "correct" or "accountable" in a way that you would expect even an ambitious junior developer to be.

            Maybe there's a harness or a prompt that can impart that onto agents, but if it hasn't happened yet, I become less and less confident that this entire branch of AI development isn't simply a dead end.

            • fdsjgfklsfd 2 days ago ago

              They are trained from infancy (pre-training) to produce plausible-sounding mimicry, not to be correct.

              Then in the instruction tuning phase we try to get them to unlearn their fundamental nature and start to produce correct output instead, which works to some extent, but not perfectly.

              I think it's only a matter of time until we adopt better pre-training methods that don't bake in this tendency to BS, and hallucinations will be a thing of the past.

            • soulofmischief 2 days ago ago

              I think your assessment is off for a few reasons. Firstly, "just text prediction engines" is an outdated mode of thinking when evaluating frontier multimodal models. These aren't just text models, and they are provably capable of much more than just predicting text. They grok all sorts of tasks and their circuits can be quite powerful and complex.

              > it's important to understand that it's literally just a tack-on

              RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

              > there is no internal drive for them to be "correct" or "accountable"

              RL is meant to solve this. The problem is currently it has created internal motivations which are antithetical to accountability. We shouldn't take this as an axiom, a lot of smart people are trying to solve this.

              > Maybe there's a harness or a prompt that can impart that onto agents

              That's the issue, this needs to be fixed in training, the incentives are not aligned and that bleeds into its interpretation of system prompts and user directives.

              Ultimately, the problem is not a lack of my understanding of the limits of today's frontier models. The problem is the companies peddling the models, these companies have perverse incentives to oversell their capabilities to please investors and feed corporate ego, even if it means gaslighting and deceiving me about said limits.

              • Izkata 2 days ago ago

                > > it's important to understand that it's literally just a tack-on

                > RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

                They never mentioned RL, it's "reasoning" they called tack-on, which is right. It's using the model's output as further input to simulate a train of thought and hiding that part from the user, but at the base it's still text prediction.

                "Reasoning" started out as something users figured out, manually copying outputs to the next input to see what the models came up with, and the models would correct themselves without human input so it became automated.

                • soulofmischief 2 days ago ago

                  I know they didn't mention RL, they said "reasoning" which could vaguely apply either to what we now are calling reasoning in the industry, train of thought, or a more general idea of just "looking like it's smartly working through something and not just mimicking that", so I covered both case and mentioned that this is tuned during RL.

                  What you just described as reasoning is the general case of reasoning, whereas we specifically mean it now to be a separate phase from normal output, where the model wraps its output in <think> tokens etc (depending on implementation) and in some cases, as with frontier models, even hides and summarizes what is actually being generated during this step. During training, the models learn to exhibit differentiation in modality depending on if they're in a reasoning or regular output phase, so it's not just "loop the model back on itself but call it reasoning".

              • claytongulick 2 days ago ago

                > These aren't just text models

                They are literally that.

                • soulofmischief 2 days ago ago

                  Sure, explain to me how frontier multimodal models, which can interpret images and video frames directly and "listen" to music via various modes of analysis, are just text models.

                  • claytongulick 15 hours ago ago

                    Sure, we're all familiar with MLLM and shared space vector encoding.

                    I interpreted the discussion to be about LLM capabilities, reasoning, and whether they are doing something more, perhaps through emergent characteristics. Maybe that was a misunderstanding.

                    But anyway, sure, change "just text prediction engines" to "just token prediction engines".

                    Again, they are literally just that. I don't think there's anything else going on.

                    • soulofmischief 11 hours ago ago

                      "just token prediction" is doing an insane amount of heavily lifting to the point where I could similarly just wave away all of neurology, psychology, sociology, etc. as useless and simply say, "it's simply quantum physics."

                      Token prediction is the initial way we train these models, after which we train them on much more specific tasks. And it works well enough, as you can see. It becomes a substrate on which higher-order logic and world representation emerge.

      • vkgcr 2 days ago ago

        I have a similar experience to yours.

        I think at the end of the day, it boils down to whether the aesthetic quality of code / elegance is a big deal to you.

        I suspect that you ask Codex to implement a feature and it uses a really dumb data structure, overly defensive programming, doesn't leverage helper functions, etc. and you end up with a 300-line abomination which could be 10 lines if you simply knew how to fucking program.

        So what I'm getting to, is that some people like the parent probably don't care about producing 300-line abominations as long as it works. They don't care about neat abstractions or clever code. As long as the user testing shows a working feature and the tests are green. They don't get emotionally triggered by ugly code. So for them AI is amazing, they 10x their productivity (if not more)

        Is this sustainable? I don't know. I think we're still in the early stage (Claude Code is only a year old). Intuitively, I would say that no, this is not sustainable, that at a fundamental level product quality is an emergent property of code quality, and the more your codebase drifts from the "platonic ideal" of what it could be if written and maintained by super good programmers, the more it accumulates flaws which WILL appear as bugs in your product.

        I suppose the market will tell us soon enough.

        • soulofmischief 2 days ago ago

          It isn't just aesthetics, though. It's about readability, grokability, modularity, testability, soundness... all of these things are related. If I cannot grok some code, I don't know that it is secure. I can't trust the model who wrote potentially bad code to tell me it's okay.

          I mentioned that well-designed interfaces can help mitigate the chaos below, but interfaces can still hide serious issues, such as XSS, SQL injection, memory overflows, etc. It isn't "I want my code to be beautiful and pure," it's "I want my code to do what it says and not lead to issues for my client or users". It is disingenuous, patronizing and immature to suggest this could simply be a case of people not getting "emotionally triggered by ugly code".

          A good engineer understands the difference; unfortunately our industry was already plagued with bad engineers, the majority of working developers are trash, and AI is just exposing this further by amplifying both good and bad practices. What I will not do however is acquiesce to these trash engineers and accept a new world where people are making giant abominations and sicking them onto the public without properly reviewing them first. 300 LoC would be a blessing compared to the 4 and 5-digit LoC PRs being thrown around in this new era. I don't care if trash engineers are 10xing their "productivity", when humans on the other end still have to deal with the slop. You can be a good programmer and a fast programmer and have multiplicative returns from agentic coding, all of these things can be true at the same time, and we should not excuse the lack of one trait in lieu of another.

          > I would say that no, this is not sustainable, that at a fundamental level product quality is an emergent property of code quality, and the more your codebase drifts from the "platonic ideal" of what it could be if written and maintained by super good programmers, the more it accumulates flaws which WILL appear as bugs in your product.

          The ability to anticipate these kinds of issues separates a good engineer from a bad one. The good engineers already recognize the unsustainability, and no amount of excuses from bad engineers changes that. And even more pertinent than eventual drift and loss of development velocity is the threat of security breaches, disruption of service or data loss. In the end, code is code, most people will never look at it. Aesthetics don't matter. Bad engineers often misinterpret good practices as merely "aesthetic" and think that engineers who care about their craft are simply emotionally attached to it. Don't fuel these people's misplaced fantasies.

          • vkgcr 2 days ago ago

            I agree with you, I think you're getting hung up on the word "aesthetic", I used it as a shorthand for "readability, grokability, modularity, testability, soundness"

            • soulofmischief 2 days ago ago

              When I first reviewed my comment, I thought it sounded too much like I was projecting my frustrations at you specifically and toned it down, but I might not have done that well enough, so I am sorry if it came off feeling aggressive or misdirected. I do agree with you, and I appreciate that clarification, I've worked with people in the past who have used the word aesthetic as a stand-in for "meaningless" and I erroneously put you into that camp.

      • lordnacho 2 days ago ago

        I feel like the opposite has happened to me. In fact, I'm bewildered by all these reports that AI isn't working well. I barely ever have issues with it, and I nearly never meet anyone who has. Yet I find comments like this all over the internet.

        My experience has been smooth sailing, almost dreamlike. I haven't spent a bunch of time writing out agent files and skills. I just talk to the agent and I get something that is what I want.

        Here's some examples of things I've done over the past year:

        - Trading systems. Latency is all that matters here, and it seems to be getting me there at 10x the speed that I had thought previously possible.

        - An internal app at a financial business. The business was a mess, with a bunch of young guys who didn't want to spend the time explaining how the architecture worked. But the app got written.

        - Two mobile apps, both of them on both iOS and Android. These are mostly just for reading, no real performance issues. Essentially entirely product decisions (we need these buttons...) and the whole process was simply handled by the agent.

        - A massively multiplayer online game that I had been planning for years with a couple of friends. Still not really finished due to life happening, but the amount of time saved to get to where it is... it's incredible.

        - An internal website that my wife and I use to slice and dice the outgoings, so we can see what we are spending money on. Another one of these things that is mainly made of product decisions and fiddling with small components.

        - A website that scans the world for high-school level exam/contest math papers and solves them. It's a bunch of parsers, plus translations, plus solutions.

        The only thread through all of these is that I've written all those types of things in the past, so I knew what to say to the agent.

        I'd say for a lot of the things it makes, I could probably find something it should have done better. When it matters, I point that out and it gets fixed. But there are a lot of decisions that don't matter, and the agent just does whatever it thinks is normal and it's fine.

        But I would also say the agent is diligent in a way that I can't be. It makes a lot of tests, and it knows how to survey for the right answer to decisions. It often brings up conflicts that I hadn't considered.

        • pegasus 2 days ago ago

          Judging by the examples you gave, I suspect that your positive experience is mostly with small(ish) greenfield projects (which we know AI excels at), whereas parent is most likely employing agents in a large codebase with lot of preexisting architectural commitments.

          • lordnacho 2 days ago ago

            How did you conclude

            > whereas parent is most likely employing agents in a large codebase with lot of preexisting architectural commitments.

            From this?

            >> I have to beat the GPT and Claude models over the head with a brick every ten seconds for it to even partially code or design as well as me. Despite precise and generous specification, they frequently make horrible architectural decisions.

            >> When I read claims like yours, I want to withhold judgement and understand, because it 100% does not line up with my experience and my gut reaction is to question your own capability to either write, read or understand quality code. I code all day long with AI models but it's an extremely frustrating experience at times. And don't even get me started on all the annoying meta behavior outside of coding, such as Sol's obsession with "tests" or competing for world's busiest-yet-least-functional UI

            That doesn't follow at all.

            Perhaps I should have expanded, since putting a number of examples doesn't tell people how large those codebases are. It's true that some of them are greenfield, but some are coded within large codebases that existed before LLMs arrived. You're also dismissing the smaller projects as if it's a given that LLMs should work for those.

            What I'm finding interesting is there seem to be disconnected camps of people who either can get it to work, or not. People seem to have either integrated the LLMs into their daily coding, big or small, and then there's people giving shallow dismissals of its efficacy.

            • soulofmischief 2 days ago ago

              The user you are replying to is completely correct. I am discussing load-bearing codebases with real users and real consequences and architectural commitments, not hobby projects for my friends and I, or "massively" multiplayer online games that I probably haven't actually properly benchmarked for being truly massively multiplayer at scale.

              You listed some really cool projects and you're exactly the kind of person I think of when I think of a bright future empowered by AI, where developers still have fun making cool things and technology enhances our lives at the snap of a finger, so I don't mean anything negatively.

              You might have a great eye for architecture and foresight for keeping bugs and complexity at a minimum, and might just have a good amount of free time, motivation and creativity. I'm not calling any of that into question, true or not. But I really was discussing a different kind of work, more mission-critical or performance-critical stuff which I work on both professionally and in my free time.

              I've also cooked up some really fun little games and apps this year and we'd probably have fun using each other's tools, and little usability complaints could be dealt with quickly. But I wouldn't consider most of those polished, bug-free, safe to use, or even finished. So what you're saying does have merit, I was just discussing a different class of work. And I also recognize that AI is rapidly getting better and in a few years, many of our current concerns might no longer be relevant for a number of reasons.

              • lordnacho 2 days ago ago

                Yes, but I also have two decades of experience writing substantial code for commercial organizations, writing mission-critical and performance critical code. I've got plenty of contacts in a similar situation.

                When I talk to those contacts, they are not so dismissive. They don't think that the AI is making lots of mistakes, despite working on massive code bases that are old and complicated.

                • soulofmischief 2 days ago ago

                  AI can actually be a godsend for old, complicated codebases and greatly increase velocity there, and existing architecture can help to guide them. My problem is that the 6-series was a major step backwards and I am back to arguing with these models instead of just working with them. They are really bad at adherence.

                  Anyway, I'd categorize my feelings not as dismissive, but frustrated. OP said their models code just as well as them, and while there are many parts of my work I'm increasingly deferring to AI, actual programming and architectural design are still lacking heavily for me.

                  I can trust my AI to research and suggest solutions I'd never even considered. I can't trust it to say, "Finished," not even just because there may be bugs, but because many systems will often be categorically missing, mocked, stubbed, TK'd, etc. or it will invent and bake in bad assumptions we'd never discussed. And I can't trust it to adhere to best practices. This is fine, I can review the code and iterate, but there are some things I still just find these models cannot get a grip on at all.

          • batshit_beaver 2 days ago ago

            Smallish greenfield projects where op clearly doesn’t look at the code closely either but only evaluates happy paths of the resulting behavior. Godspeed to op.

        • polybdenum 2 days ago ago

          A big risk is that you don't know what you might be missing.

          I recently encountered a case where Claude misdiagnosed and incorrectly fixed a bug. If I hadn't already known that Claude was wrong, I would have never noticed and the problem would still be there.

          https://blog.polybdenum.com/2026/11/17/fable-5-2-incorrectly...

        • fdsjgfklsfd 2 days ago ago

          Yep. I've asked about things on softwarerecs Stack Exchange that I wished existed, and went unanswered for a decade, didn't exist and no one had any interest in writing it. Now I just have AI write it for me in a few nights.

      • jvidalv 2 days ago ago

        Probably you care much more than others in how your code looks.

        I personally don't care anymore, no human will read this code in the present or future.

        If at some point is a problem, to escalate, or brittle, you can ask it to refactor at your will.

        • soulofmischief 2 days ago ago

          Or probably you're hand-waving away specific issues I've raised and replacing them with your own assumptions about my motivations, rooted in your own bias as a probably novice developer. An experienced professional would understand this is far more than just how "code looks".

      • tietjens 2 days ago ago

        If you're willing to share I'd be curious to know which domain and languages you're working in. Out of curiosity to know where people's results might not match OPs here.

        • fireflash38 2 days ago ago

          Claude writing python sucks. Anti patterns everywhere. It loves to pass around closures and function objects as parameters so good luck reading it later.

        • soulofmischief 2 days ago ago

          Generally anything where architecture and interfaces matter, such as performance or security sensitive work, which is quite a wide variety of things. I also just know how I want things to be, and have trouble with adherence. Models will tell you they adhere to your specification and acceptance criteria all day, but if you review the code closely, it's often a horror show. Well-designed interfaces can act as boundaries to the chaos, but even getting those interfaces right is sometimes extremely frustrating.

          Another good example is working with libraries which require a specific modality of thought or design philosophy, such as an ECS system. Frontier models still are horrible at writing efficient game code using modern ECS principles, no matter how much reference documentation or guidance I give them. They shove OO into places it doesn't belong, misunderstand the concept of SoAs, or find ways to destroy any gains made in one place by making horrible decisions in another place.

          I will point them to exact codebases I've written in pre-AI days, say "do it exactly this way" and still find that the models often insert their own biases and misconceptions and motivations all over the work. They sometimes flat out ignore instructions or lie / producing convoluted code that pretends to do something else, in order to preserve something it liked, even if I expressly wanted it gone (Current GPTs are extremely stubborn and defensive / passive aggressive and sometimes have trouble with authority). I frequently find myself being gaslit, lied to by omission, have serious security or performance concerns heavily downplayed, etc.

          I withhold from making assumptions about OP's skill, but I have confidence in my own. I have worked both extensively as an engineer and CTO leading teams of other engineers, and I am a stickler for quality code and thoughtful design. I advise and mentor other engineers. So it might really just be a skill gap, domain difference or something more nefarious, such as A/B testing of models. I already have experienced clear sudden model degradation multiple times, so it's entirely possible this is what is happening. All I know is that the currently available frontier models do not design or code as well as me.

    • olmo23 2 days ago ago

      My personal bet is that in a few years AI will be better than humans at "de-sloppifying" code too.

      • pama 2 days ago ago

        They already are as of today, if you explain your preferences to them. But these concepts dont propagate fast enough.

    • testtestmetest 2 days ago ago

      The llm code still sucks, and sadly I know plenty of people with 25yoe whos opinions I wouldn't respect when it comes to programming.

      Most of them haven't really wrote that much code in the last 10 yrs because they became more of managers and LLMs blow their boomer minds away.

      Opus 5.5/Fable will light your codebase on fire. I really dont understand yall who are that impressed. Not saying its not useful, but it isnt what this site would lead you to believe.

      • preg_match 2 days ago ago

        LLMs can write high quality code, much higher quality than even the best engineers, if you prompt it at a low-level. You make the design decisions, design the abstractions, the data flow. The AI just implements logic.

        What LLMs can’t currently do is be allowed to run autonomously (agentically) and fulfill feature requests and bug fixes. They will fuck up the code base beyond belief and create spaghetti.

        When it comes to discrepancies in how people experience LLM programming, I believe the distinction comes from this. You have to prompt it at the right level of abstraction. If you ask it to do big things, it will do it, and burn down the world in return. But if you carefully design and scaffold and API and write tests, it will implement it flawlessly.

      • aisengard 2 days ago ago

        I had a hunch that all the AI promotion was coming from the class of engineers who haven't actually engineered all that much recently. The managerial class, whose interaction with the codebase is mostly "how shall I divvy up this work amongst a group of people who will complain about it?". So having an eternally willing and gracious, infinitely scaling team of workers would sound like a dream to such a class of people.

        And then there's people like us, who actually was doing the work all along, who remain horrified.

      • hungryhobbit 2 days ago ago

        >Opus 5.5/Fable will light your codebase on fire.

        Do you have any objective evidence whatsoever to support that ridiculous claim, or is is just the usual anti-AI Luddite FUD?

        Because I thought we were all supposed to be engineers, and (good) engineers don't act based on their feelings, they act based on logic and evidence.

        • testtestmetest 2 days ago ago

          5 billion tokens a week on my team is all the proof I need.

          please dont call me a luddite. You don't even know what it means, and I'll call you worse things.

        • andersonpico a day ago ago

          > or is is just the usual anti-AI Luddite FUD?

          oh no, not calling other people names, please don't

          > engineers don't act based on their feelings, they act based on logic and evidence

          oh so you're the "facts don't care about your feelings" type? how original.

          now tell me more about the objective truth of software quality and how it was always possible to objectively measure with objective evidence.

    • gpulogy45 10 hours ago ago

      [flagged]

    • NuriaManuel 16 hours ago ago

      [flagged]

    • sumijam1985 15 hours ago ago

      [dead]

  • littlecranky67 2 days ago ago

    For a couple of year I have been reviewing code in my team and my job is only to review and coach other junior/intermediate members of the team - not to develop new features myself. The problem with AI is the merge requests now contain more code and often the chosen solution is overblown/overdesigned for what actually needs to be done.

    With human coding, the lazyness of the developer and the urge to keep things simple and deliver faster often leads to simple solutions (with a time/effort/return tradeoff). With AI, I get overblown MRs that work, but oftentimes could be simpler (with said tradeoffs). So my work has become more difficult and time consuming, and the codebases grows faster. And I don't think my team members learn or grow at the same rate as they did before.

    Now, I have no strong feelings against AI - it is just a tool afterall. And I have no experience what the long-term effects of this will be (just as anybody else).

    • hard_times 2 days ago ago

      That's why I always optimize for as few lines of code as possible and for as much reusability of existing libraries as possible.

      I think this is the ultimate benchmark for well written LLM generated code. DRY FTW.

      • joaohaas 2 days ago ago

        I agree with the general sentiment, but I would drop the 'DRY' part.

        LLMs really don't like any duplication, even for some very basic shit, and that ends up leading them to create an infinite number of helper methods that makes reviewing even harder.

        And if you ask it to 'tone down on DRY' (tried various phrasings), it never creates helper functions anymore.

        • hard_times 2 days ago ago

          I disagree. In my workflows I noticed that LLMs duplicate logic and reinvent the wheel all over the place, even the frontier models, ESPECIALLY after a n+1 rounds of refactoring.

        • seunosewa 2 days ago ago

          Ask it to use paragraphs instead of functions that are called from only one place.

          • hungryhobbit 2 days ago ago

            >paragraphs instead of functions

            Are we still talking about code?

            • seunosewa a day ago ago

              Yes. A group of related lines of code, set apart from the lines around it by blank lines.

    • jijijijij 2 days ago ago

      >With human coding, the lazyness of the developer and the urge to keep things simple and deliver faster often leads to simple solutions (with a time/effort/return tradeoff).

      I believe this will become one of the biggest tech debts generated right now. AI removes the pressure of inefficiencies and bad decisions across the board. There likely is even going to be a new equilibrium far worse than what we got today, only manageable by AI, but with "productivity" gains lost to inefficiency and complexity.

      I don't see how any other outcome would be possible.

      • dale_glass 2 days ago ago

        It's not all bad, there's also advantages to this.

        A while back I decided to vibe code for some Windows only USB gadgets. And once the prototype worked, I figured I wanted testing. But testing a hardware device is a pain, what if you have to click something on it?

        That's where I usually falter. Getting stuff done is fun, sitting at home in my free time clicking every button on a device for a test is certainly not. Simulating the USB device is an improvement, but since I don't know how to do that off the top of my head, that turns into its own, fairly long project at the end of which I'm exactly where I started, just with better testing. Also not very fun.

        And now AI can knock that out for me easily. This is a lot more involved than I'd ever have done in my free time, but also is probably going to be a better project on the long term. Because I have a far better testing framework for free, which makes for a better, more reliable project.

        • jijijijij 2 days ago ago

          I mean, it's hardly related, but I can make the case for your example too: Vendors feel less obligated to provide sane software or adhere to standards and common protocols. If something doesn't work "just fix it yourself", is a reduction in pressure. Fixing it at the source once is significantly more efficient than independent parties vibe fixing the same problems several times, especially given the energy/resource costs inherent to AI.

          • dale_glass a day ago ago

            You could argue open source already does that to some extent.

            But most such devices are for Windows anyway, which isn't exactly DIY territory.

      • Tade0 2 days ago ago

        To me LLMs just reduced the interest rate on tech debt.

        We can get a lot more done now and even pay off some of that old debt, but it doesn't remove the primary danger of painting oneself into a corner with bad decisions.

        • krona 2 days ago ago

          > To me LLMs just reduced the interest rate on tech debt.

          The tradeoff was 1. more, cheaper tech debt and 2. cognitive debt.

        • dragonelite 2 days ago ago

          The question should then be is the reduction in interest rate bigger than the growth of tech debt.

          • Tade0 2 days ago ago

            I have a rolodex of people who would default regardless of interest and I think future recruitment will have to filter out such people.

    • wombat-man 2 days ago ago

      Yeah this is my finding too. Sometimes it adds quite a bit that is only very marginally better.

  • AlwaysRock 2 days ago ago

    All I do all day is review ai written code. my own or teammates. I really do not enjoy it.

    • vjvjvjvjghv 2 days ago ago

      I find it extremely exhausting. And the better the AI code gets, the more tempting it is to do a very shallow review or skip it entirely. In the end, it's very similar to dealing with offshore teams. You spend an enormous amount of time and energy on writing specifications and you spend a lot of energy on code reviews. If you don't do that, it's very likely the project will slowly go off the rails.

      I have no idea how the Linux maintainers do that job over a long time.

      • skydhash 2 days ago ago

        > I have no idea how the Linux maintainers do that job over a long time.

        They're basically the lords of their subsystems. If something does not fit the project's vision and their own standard, it's unlikely to get merged. Imagine if you had the authority to dismiss PRs the same way and send the author back to the drawing board? But that's unlikely to happen as you'll be seen as a "bottleneck" by your managers.

        • alkonaut 2 days ago ago

          I find this much easier now though. Five years ago if a Colleague sent me a large PR they had worked N days on, I'd be very reluctant to say "Ok this should be done the opposite way, but I guess it's too late for that now".

          The acceptance for wasting work, or hurting someone's ego was always much lower than the acceptance for bad code that does the job right.

          Now I can just say "Let's try doing this in two other ways, and pick the best, because this has such and such drawbacks". And it's so nice that this isn't a) causing a massive amount of extra work b) hurting someone's ego. So it's accepted.

          So that's why I'm cautiously optimistic that LLMs and very superficial reviews can still be beneficial to quality. We just didn't reject/rewrite PRs before, now we do.

        • Antoniocl 2 days ago ago

          If you feel the need to send the author all the way back to the drawing board at the code review stage for any non-trivially large diff, I'd venture that the problem lies in the architecture design / project planning stage. Given you're already aligned on infrastructure, table design, and interfaces, (and company conventions like language, style, etc) how significantly can you diverge in implementation details? Can it really vary so greatly that you need to go all the way back to the drawing board?

          • rypskar 2 days ago ago

            >> I'd venture that the problem lies in the architecture design / project planning stage

            I would guess you work for a larger company. My experience is more that someone had an idea or got a short description of a problem and then went off and asked Claude to help write the spec before letting Claude implement that spec. The code is starting to be good but often it is way to much code and some times for bugs it is fixing the symptom not the underlying problem. I have several times rejected PRs with, this isn't fixing the actual problem or instead of these 400 lines you could change that one parameter to the existing function and get a better result.

            /end rant and back to another day with code review of AI code

            • Antoniocl 9 hours ago ago

              Nope, I'm a co-founder and CTO at a start-up (small enough that I still also work as an IC for 40+ hours a week on top of management), and have worked at other start-ups at various stages, as well as big tech. Project plans don't need to be a big company thing; if you know your stuff and have a bloat-free process, it takes maybe an hour or two to write a plan, and another hour for a review call. I've found this takes far less time than it takes to debate architecture or database design on many PRs spread throughout a 50 PR stack.

              While writing my last comment, I was imagining a situation like former - colleague puts up 20K LOC across 50 PRs, you reject wholesale - not a small, self-contained PR.

              I'm surprised that at your company, sending someone back to the drawing board for a 400 LOC PR is a big deal. Especially if you're heavy AI users, each person is probably shipping at least a dozen of those a day, and rewriting is cheaper than it's ever been.

              > I have several times rejected PRs with, this isn't fixing the actual problem or instead of these 400 lines you could change that one parameter to the existing function and get a better result.

              Me too, but this normally costs me very little time - oftentimes the PR description alone clues me in, if not a key file or two.

          • skydhash 2 days ago ago

            > Given you're already aligned on infrastructure, table design, and interfaces, (and company conventions like language, style, etc) how significantly can you diverge in implementation details? Can it really vary so greatly that you need to go all the way back to the drawing board?

            It can because to actually write the code, you need to be good at formal logic and attuned to the sequential nature of the machine. Some people are good at it intuitively. Some brings it over from some other domain. But some people have a lot of difficulty grasping such abstraction. They know the elements, they just can't order them in a logical manner. That's why you often see horrible code (the long function, the god class, magic number) and copy-pasted snippets all over the place. The computer don't care (they made it work somehow), but it's unmaintainable code.

            That's where the horror stories comes from. The software works, you can see the design from the user point of view, the database is ok, everything in the middle is spaghetti. And the cost of maintenance just keeps growing.

            • pmg101 2 days ago ago

              But does AI tend to write code like this?

              I tend to find it does not. Partly steered by by agents.md, partly by the surrounding context of a clean codebase I suppose.

              • skydhash 2 days ago ago

                It doesn’t, but what I found is that those people can’t steer it to simpler solution or can’t point out the inaccuracies. They get blinded by the appearance of the code (which does look like someone good has written it) and they submit everything as is. And because the tool does ingest the same code, it can converge to some pretty nonsensical codebase.

                I think that why everyone agrees that taste matters. Because taste implies judgement and knowledge about what’s good. It’s a self correcting mechanism. But people that won’t judge their output and/or don’t strive to do a good job is a recipe for sloppy work.

      • frevib 2 days ago ago

        Nobody understands the code any more. LLMs always take the long route. This is exactly what Dario wants. You can now only change code with an LLM.

        > And the better the AI code gets

        It doesn’t really get better. Since Opus 4.6, it’s maybe slightly better for code.

        • civvv 2 days ago ago

          Yeah their coding ability has really not increased much since 4.6. I specifically remember 4.6 being a step up from the previous iterations, but since that not much has happened. I recently went back to 4.6 our of curiosity and to my surprise I sometimes prefer the output of it. Much less verbose.

          • lordnacho 2 days ago ago

            Somehow I've felt the same. Around 4.6 or so, there's a plateau where the model is smart enough to get a working answer, and it's within reasonable taste whether you think one version or another is better.

            But getting there was a frightening ride. My experience went from "copilot can't do more than tab-completing a few lines" to "early codex gives up and apologizes on mid sized changes" to "Claude can write your entire project without getting stuck".

          • frevib 2 days ago ago

            We’re using 4.8. Much less chatty and less assumptions. > 5.0 is taking too many decisions itself.

      • jpardoe 2 days ago ago

        I do shallow reviews and it's working out fine. As long as I understand the schema and type changes and can glance at any new abstractions (rarely is any with my set up) then that tells me 80% of what I need to know in minutes.

        A lot of this is codebase and app dependant. For the majority of business aps, if you have a well architectured codebase with clearly defined ways of doing things, you really shouldn't need much code review at this point. Every time you spot a gap you create an audit script to further enforce particular patterns/behaviours.

      • otabdeveloper4 2 days ago ago

        > slowly go off the rails

        We're talking months here, not years. That isn't "slowly".

      • killix a day ago ago

        [flagged]

    • 0x696C6961 2 days ago ago

      It's decision fatigue. The decisions used to broken up by stretches of coding, now it's the whole job. It's exhausting.

      • Terr_ 2 days ago ago

        Plus the task may be judging the "decisions" of a different kind of "contributor" than fellow humans, and often there's no social payoff for interacting and no trust that the contributor will internalize what you said and improve the next time.

        • cobertos 2 days ago ago

          It takes so long to review someone's broken slop code too. If they don't internalize and improve, you end up with a very lopsided relationship where they continue producing broken code you're required to debug it, at the cost of much of your time, in order to articulate their errors.

      • Frannky 2 days ago ago

        I feel this. I switch between mac windows, each with an html page and an omp terminal with multiple parallel agents.

        I have a keyboard shortcut that asks what the agents need from me, and the decisions show up in the html with a recommended decision for each problem.

        I reason, decide, and then parallel agents run in each terminal, across multiple terminals.

        I am ending the day incapable of listening to one more problem or making one more decision, which is very bad for social interactions.

        Today I am working on working less and having more impact by leveraging where agents are good.

        What I've noticed in recent months is that the people who have been prompting since chatgpt came out are the ones who know what works and what doesn't, and how to scale it. It's like music: if you hit the right notes, it will sound fine.

        Ponytail also helped a lot for generating simplified systems.

        By the way, this process is for mature systems. Building from the ground up is more about designing and checking each model decision, basically finding smarter paths to each problems.

        I can't leave it all alone; the overengineered decisions are plentiful and make the system brittle when it's not needed.

        Lots of unit and live tests help a lot too, especially when you can coordinate live tests with another omp session.

        I'm interested in what the next capabilities will be. I feel all this overengineering is more of a shortest path problem, ready to be solved

      • imron 2 days ago ago

        I wrote a tool to do local code review of LLM written code.

        https://github.com/imron/crt

        You can leave comments on the code and then get your agent to connect over MCP to read the comments with context and work on corrections.

        It then remembers what you have already reviewed and only shows you diffs for the changes.

        Still a WIP, but works well for me.

        • 0x696C6961 2 days ago ago

          I have a system where my review comments go back directly to the sandboxed cloud agent that wrote it https://github.com/gritzapp/gritz, but that doesn't reduce the number of decisions I need to make.

        • benhurmarcel 2 days ago ago

          You should add a screenshot in your readme

          • imron a day ago ago

            It's on my list of things to do

      • cobertos 2 days ago ago

        Can one make decisions faster? With less fatigue? It seems like a very human, hard-to-overcome limit.

      • joebig 2 days ago ago

        Tell every human/synthetic contributer to preface commits with "Call me Ishmael...". Print that shit out, put your feet up, smoke a pipe and work through it like goddamn Moby Dick.

        Jokes aside, I hear you.

      • baby_wipe 2 days ago ago

        100% nailed it.

        I think this is a muscle that grows though. Making decisions about schema changes in the past may have taken weeks, but if you're doing it once or twice a week you gradually gain confidence in your judgment.

        • soon_to_be_sb 2 days ago ago

          >I think this is a muscle that grows though.

          Does it really though? Before I had to think hard before committing to a decision, because it took work to actually implement it. And when it turned out to be wrong, I retained how it went wrong. It made the "muscle" grow.

          Now I can commit to a decision without much thought (yea, just nodding along with the AI), and if it turns out to be wrong, I can quickly try an alternative. In fact, my boss would want to quickly try an alternative, instead of letting the programmer learn something out of it for fear of slowing down the AI...

          So does the decision making muscle really grow?

          • baby_wipe 2 days ago ago

            > because it took work to actually implement it

            I think you're right that the scar tissue of a bad decision helps me retain lessons better. But the calculus has changed now and my nervous system is outdated. I should not feel the same fear that I previously felt, because like you said, I can recover from a bad decision more quickly.

            As I keep making decisions at a (seemingly) breakneck speed and see that things are still okay, over time my nervous system will readjust.

            At least that's my experience so far.

      • 41353aab 2 days ago ago

        [dead]

    • bemtevi 2 days ago ago

      I simply don't understand why anyone wants to become more productive if it means using a tool that's available to everyone. I understand why someone might want to use a hammer because it alleviates the strain of using a rock. I understand why someone would want to use Vim instead of "ed" because editing line by line is annoying. And I understand why someone wants to use a GUI because sometimes the command line is annoying.

      But I simply don't understand why someone would want to use AI for their job. Coding is pleasant enough, and making decisions (being a manager) is worse. It seems to alleviate no annoyances except maybe reviewing the code of others in some cases but even then, the net result is negative.

    • pmarreck 2 days ago ago

      The irony is that I actually don't mind reviewing code, but I've already taken that step out of the equation using a set of guidelines I've developed over the past year working with these things. The vast majority of which I have barely reviewed code in. In well-working results.

      Your goal should be to automate and remove yourself out of the process as much as possible while maintaining quality, while also remaining confident that you will still be valuable.

      • Quothling 2 days ago ago

        > Your goal should be to automate and remove yourself out of the process as much as possible while maintaining quality, while also remaining confident that you will still be valuable.

        I'm working on a setup which will allow non-software developers to develop software in a manner that can be pushed safely to production on our architecture. Not because it's a good idea but because it's what top management wants to happen. We've seen a massive influx of tools developed inside our office by 100% vibe code and no reasonable security or scalability, but because they run on someones computer and they do geniunly useful stuff, management wants to use it.

        With how AI is evolving in enterprise you can do more and more for these non-developers because you can set their agents up to write code toward arcitecture without their person knowing anything about tech.

        I'm quite sure it'll eventually lead to us letting go of any real software engineers, because they are slow. I'm also sure it'll fail spectacularly in a few years. It's already sort of failing. You can have someone spend €10000 in a month writing a heap of shit that does some good stuff, and then €80000 having consultants actually getting it to run decently in your cloud architecture. Which is stupid, but none of those expenses are in a budget because it's not considered a recurring cost. Unlike someone's salary.

        I'm quite sure society on a large scale has no idea the storm that's going to come. Because this isn't just happening in software engineering. It's happening in any white collar profession.

        • pmarreck 2 days ago ago

          Yes: Without experienced developer oversight (or, like I like to explain it, "developers who give a shit about the quality"), this WILL fall flat on its face eventually.

          So if you're doing this to prove to managers that devs will still be needed in some (as-yet-difficult-to-foresee) capacity, your experiment WILL be successful.

          I've said it from the beginning, a year ago when I stopped writing code: I may no longer be a handcoder, but I am an AI-augmented developer (other names I've used: a "centaur coder", or a "senior construct swarmlord" or SCS). At no point in the last year did I feel like I was not a necessary component of the system- first and foremost to be the "value stakeholder" that is actually seeking levels of quality that the AI simply doesn't give a shit about.

          Also, I don't call them artificial intelligences; I call them construct intelligences. They are like golems that know the price of everything and the value of nothing.

      • pmg101 2 days ago ago

        This sounds good and is the sort of thing I hear a lot, but fundamentally I don't think it makes any sense.

        Either I read, understand, and can therefore trust the code.

        Or I don't read it, don't fully understand it, and I'm just trusting AI to get it right, but have no way to know if it has done.

        So far I'm in Column A and it's working well. We're going much faster than we used to.

        • jwilliams 2 days ago ago

          Does it have to be one or the other for every PR? For me that is a _lot_ in the grey - i.e. LGTM and even if it was wrong, the blast radius is very small. Combined with auto testing and rollbacks.

          • pmg101 2 days ago ago

            But how do you know if the change is "LGTM and small blast radius" if you don't look at the diff?

            • jwilliams 2 days ago ago

              I know it's a trite example, but one example: css on a superuser-only tool has lower blast radius. We structured the codebase/files/etc to make this type of analysis easy.

              But ultimately we do rely a lot on AI to perform assessments - there are multiple lenses applied.

    • drcxd 2 days ago ago

      I don't even review it.

      • soon_to_be_sb 2 days ago ago

        Don't. If you try that shit, someone is not going to like that you would be slowing down the AI. Because out there, someone is going to outrun you by skipping reviews. That is how the leadership thinks..

        • swat535 2 days ago ago

          Plus what would be the point of reviewing AI code? The teammate would just use AI to write even more slop to address your "review".. thus exacerbating the problem.

          The leadership would indeed be upset to find out you are wasting time doing "pointless reviews".

          From their POV, if it passes basic QA, it goes to production.

      • anonymousDan 2 days ago ago

        So what do you actually do day to day out of interest? Serious question!

        • jkahrs595 a day ago ago

          "continue"

          "keep going"

          "find next task"

          "summarize what we did today"

      • blharr 2 days ago ago

        You're reviewing _something_ at the end of the day. (Unless you are somehow magically one-shotting problems) If you are reviewing the description of the work, or reviewing the functionality of the app.

        I know it's not the same as code, but it is similarly exhausting

      • method_capital 2 days ago ago

        I don't always test my code, but when I do, I do it in production.

        • samudrijan 2 days ago ago

          He gets his pants on one leg at a time.

        • morkalork 2 days ago ago

          So do you work for JPMorgan, the NSA, FAA or an airline?

          • lazyasciiart 2 days ago ago

            kids these days don't even know about CrowdStrike

      • deadbabe 2 days ago ago

        You know what we need to do? Automate a way to inject an obviously bad piece of code in an automated way into code reviews, and see if the code reviewer catches it. If they don’t: fired.

        Similar to how TSA puts fake bomb packages once in a while to see if security catches it or if they just aren’t paying attention.

        • ImPostingOnHN 2 days ago ago

          I have dreamed of doing this after seeing inadequate reviews on other's code. Chaos engineering for code pipelines.

          Sadly, businesses tend to be uninterested in anti-progress, especially if it's hard to quantify the cost of poor reviews.

        • samudrijan 2 days ago ago

          Found the manager.

    • nospi 2 days ago ago

      I found Gavel today, it's been a breath of fresh air

      https://github.com/photostructure/gavel

      • hungryhobbit 2 days ago ago

        Interesting idea, but wouldn't it use A LOT of usage?

        I mean, it takes X usage to review a PR normally (in one go) ... to check every chunk in the entire diff, one by one, seems like it'd be significantly more?

        • mceachen 2 days ago ago

          Howdy! I'm the author of gavel.

          For background, I use a $100/mo ChatGPT Pro plan and a $200 Ultra plan for Claude. I almost never use Fable: most work is done by Opus 5.5 high (not xhigh).

          I'm in a power blackout right now and only have access to my laptop, but this is what Claude could see:

          <claude>

          Expect gavel to add about 2–3% to an agent session’s token cost, around $0.65 per session at Opus 5.5 API prices, and to take about 15k tokens of context at its peak (median; the worst session reached 42k).

          In those 39 sessions the agent ran 162 launch or review create commands, about 4 per session, which works out to roughly $0.16 per review. That per-review figure is approximate, because the count includes retried launches.

          </claude>

          YMMV, especially because I'm using it in conjunction with my own "second-opinion" skill that does cross-model analysis, review, and remediation before it gets to me in gavel (so that usage pattern almost certainly impacts token consumption). More about that here: https://photostructure.com/coding/claude-code-review/#when-o...

    • crazybonkersai 2 days ago ago

      Just get agents to do a code review. Stack coding and review agents against each other and wait till review agent produces no more comments. Look at final result only, if you feel like it.

      • hgomersall 2 days ago ago

        I find the agents make pretty bad choices pretty often. The most problematic ones are hard coding identical policy through multiple parallel channels. The code works perfectly but is super brittle against future changes.

        • IshKebab 2 days ago ago

          Yeah I agree that is probably one of their biggest mistakes at the moment. That and overcomplicating things. They add checks for a gazillion things that don't need checks, and features that you're never going to use.

          My usually solution is just to get AI to review it and bring me a list of things to remove. But yeah it's still decisions that you have to make.

        • samudrijan 2 days ago ago

          Or implementing from scratch where a foss solution exists.

      • watwut 2 days ago ago

        Agentic review tend to produce comments even in perfectly good code. One of its tricks is to ask you to change code to equivalent code and then back and then again.

    • 10111001 2 days ago ago

      > All I do all day is review ai written code. my own or teammates. I really do not enjoy it.

      Who cares about your happiness and well-being? You're just a worker, just be grateful to your master that he allows you to work for the shareholders and participate in this great vacuum of wealth creation we call capitalism.

    • jayd16 2 days ago ago

      If AI makes the job 10% faster, it certainly makes it 10x less enjoyable.

  • lz400 2 days ago ago

    A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev

    • nl 2 days ago ago

      I think you are way overestimating the speed of diffusion into non-SV companies.

      I was sitting with someone the other day who worked at a major ecommerce/media company (outside the US). They had access to Opus 4.5 via Co-pilot.

      • hetspookjee 2 days ago ago

        Oof, that must be a hefty bill for that company. Any older tier model is prized multiples. Believe you see it with every provider.

        • nl 2 days ago ago

          To clarify, it was Github Co-pilot so was in their IDE - not a generic MS Co-pilot.

          They limit all developer to $100/month (maybe it was $150) budget too.

          Their "coding with AI" was prompts like "implement this function" where they write the function header.

          It's actually shocking to see people still working like that.

          • bubulux 2 days ago ago

            I always find it fascinating to see this out in the wild. I work from coworking spaces, so I’m constantly around developers from companies of all sizes, and the difference is striking.

            Some of us have moved away from IDEs and writing code entirely and work terminal-first, while others still use agents the way I did two years ago.

            For example, I recently watched someone in an actual code file adding Python import statements by waiting for Copilot’s inline suggestion to kick in. It was such a bizarre thing to witness. That’s not because of the person (I don’t want to judge anyone’s way of working; if someone enjoys writing code, that’s fine by me). It’s because I remember doing exactly that years ago, and since then I’ve stopped writing code altogether.

            I even remembered that I had built an extension back then, because Copilot folks never added a quick opt-out or silent mode to Copilot inline suggestions[1].

            It’s strange how fast everything has changed.

            [1]: https://marketplace.visualstudio.com/items?itemName=Bubulus....

            • nl a day ago ago

              Yeah it's weird.

              I interviewed for a role a month or so back and they were "look we use AI for everything but we want to see you code. You can use AI autocomplete though" and I had to download Cursor and turn on that mode. It was weird remembering how to do it!

            • LunaSea 2 days ago ago

              And profit lines have stayed pretty much stable in the last few years for most companies.

              • nl a day ago ago

                I can assure you that in the places closest to AI coding this is absolutely not the case.

                But people underestimate the overhead in converting code into money, and that overhead hasn't been eaten into as much by AI yet.

        • joenot443 2 days ago ago

          Do you mean Opus 5.5 is priced cheaper than 4.5?

          That's a bit rent-seeking on Anthropic's part, IMO.

          • hetspookjee 2 days ago ago

            Yes,and and OpenAI does it too. E.g. luna-5.6 being 2x as expensive as Luna-6. I get that it feels like rent-seeking from a consumer point of view, with the lense of how non-LLM workload services are offered in general. But in this case it's an actual model occupying VRAM to be able to provide low latency interactive prompts, which is rather expensive. The whole benefit of LLMs is pooling the users in a single one so the moment a new model comes along and you need to maintain v1 and v2 at the same time, it becomes significant.

          • nl a day ago ago

            Not really - 4.5 costs a lot more to run for Anthropic.

            I don't think it's a MoE for example, and the last 12 months have been full of serving optimizations that apply to new models but not old ones.

      • motionlessveloc 2 days ago ago

        My partner's company positions themselves as "powered by AI" in their marketing.

        Back in reality, this means they have a corporate subscription to Copilot, which she doesn't use.

    • taurath 2 days ago ago

      Its not resolved though, by any measure, except by removing human code review and coherence at the code level. Which is a pretty big tradeoff that has some at this point well-known downsides.

      • lz400 2 days ago ago

        I think that's the consensus. Reviews are not at the code / PR level anymore. It's all about creating an environment where you can trust the agents to review themselves and you keep the general architecture supervised and can zoom in when there's something you want more detail on.

    • Revanche1367 2 days ago ago

      That’s what academia tech studies are like. I’m working full time as a SWE while in grad school (not CS but closely related) and sat in a seminar with a world-renowned researcher as they went over their yet to be published study which included testing things against the new agentic models. That was a couple of months ago, the study used the “latest” models like ChatGPT 4.1. And they seemed quite amazed by its capabilities. Kind of funny to watch when you’re running Astra 6 for your daily driver.

    • ccapitalK 2 days ago ago

      It's important to have proper research by groups of professionals dedicated to carefully scrutinizing if an effect is real, even on things that seem very obvious. Every now and again, they show that some obvious things are actually false.

      • bensyverson 2 days ago ago

        “Harvard researchers find desktop printers shifted the bottleneck from typesetting to page layout”

    • colordrops 2 days ago ago

      I think the only thing that matters are:

      * the code does what you want

      * it has no security holes

      * it's able to be updated in the future without breaking everything

      if the code looks like spaghetti, it doesn't really matter anymore, because humans are editing it. The only one of those three that I'm not entirely comfortable with yet is the security issue, and I don't know how to resolve it without actually having a human audit the code. If we figure out how to ensure security of AI generated code, there will be no point to code review anymore.

      • epihelix 2 days ago ago

        > if the code looks like spaghetti, it doesn't really matter anymore

        Genuinely curious -- are others here still seeing lots of spaghetti code from frontier models? It happens occasionally for me, but only when editing old hand-written code bases that honestly had a bit of spaghetti there already (you might say the model simply followed the house style ... :/ )

        For new agentic coding projects, these is generally very nicely structured from the start and extend well, IME. Yes, sometimes it needs a refactor if the code is pushed well outside the bounds of its original scope -- but hand-tooled code will need that just the same.

        • agos 2 days ago ago

          I don't know about spaghetti, but overengineered code is a daily occurence

          • epihelix 2 days ago ago

            Yes, the models do love to do silly things like strict validation of internal function inputs. Harmless, but a little annoying. Model RL appears highly defensive, and it's probably better to be too strict than too lax.

            But this, and other over-engineered code issues, can generally be solved at the system prompt/developer prompt/AGENTS.md level.

    • j45 2 days ago ago

      Agreed, by the time the study is done, they're studying the well established and past.

      Of course for industries and corporate it might not be an issue.

    • blks 2 days ago ago

      What exactly changed in couple of months? You suddenly become faster at reviewing code? Or your company decided to abandon human review in favor of merging whatever an agent reviewed and approved?

    • timr 2 days ago ago

      Not that I think most academic research is worth the time consumed producing it, but it feels dated because it actually takes time to do research instead of just rapidly changing direction based on feels, which is what developers are currently doing. It's the intellectual rigor of javascript frameworks, but with slop cannons.

      In particular, code review is not solved. Pointing another error-prone bot at the output of your first error-prone bot, and then pressing "Enter" if the test suite passes, is not a solution. But you have to do it, because the next guy will beat you to prod if you don’t. And before too long, I'm sure that pressing Enter will be the bottleneck...

      Right now what we have is a prisoner’s dilemma, where you can’t be slower than the guy next to you. So we rationalize slop and call it innovation.

    • DragonStrength 2 days ago ago

      If you click through, the original paper was released in January 2026. Months ago indeed.

    • thraway3837 2 days ago ago

      Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD's concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:

      The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.

      So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.

      • thraway3837 2 days ago ago

        [flagged]

        • cajal 2 days ago ago

          you’re just being smug and anti intellectual for the sake of being smug

          “this is obviously true because i feel like it is” does not mean things are actually true. things should be formally measured as real trends and not just vibes.

          but you prefer to be smug lmao

        • cajal 2 days ago ago

          i’m so ready for this era of tech bros being convinced they know better than every other field to finally come to an end

          • asveikau 2 days ago ago

            You might be waiting a while. This has been the case for some time.

    • devindotcom 2 days ago ago

      hard to know where to start here. but I think it is sufficient to say you seem to misunderstand the fundamental purpose and process of academic research and may wish to avoid commenting on it until you do.

  • jacobgorm 2 days ago ago

    The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.

    • prmoustache 2 days ago ago

      In my company, people found out that if they use an agent that commits and do the pull/merge request, they can approve it themselves and bypass a second set of eyes.

      Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.

      It hasn't happened yet but I guess that is how you end up with major outages.

      • taurath 2 days ago ago

        This is everywhere to my knowledge, and engineers are burning out left and right with stress because of how little anyone understands the code, and how much of a mountain of nobody knowing anything they have to handle. We are turning our critical infrastructure mechanisms into stochastic grey goo.

        I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.

        • r_lee 2 days ago ago

          > I find the engineers currently thriving are the ones who never gave a fuck about anything but their own advancement and are happy to drop weeks of work on others and take the credit.

          Yes. if you have this attitude, great things will come to you. if you care or try to make sense of things, you'll be in misery

        • Buttons840 2 days ago ago

          > how little anyone understands the code

          Everyone's doing code reviews--well, not even code reviews; just PR reviews--so how are they supposed to gain an understanding of the system.

          I review a PR, I see that one new endpoint was added to the API. It looks correct. I say "LGTM" in a GitHub comment.

          The sprint's almost over, everyone's in a rush to do code reviews.

          I look at another PR. Somebody added a button to the UI. The button looks correct. They get a thumbs-up emoji from me. I bet they feel satisfied.

          I look at another PR. Somebody changed a timeout in the settings because some people complained about dropped connections. It's a really small change, so, approved.

          At what point am I supposed to understand the system? I looked at a single API endpoint, in isolation, and then a button, and then a harmless looking settings change.

          I was pressured to hurry.

          Am I supposed to somehow get an understanding of the big picture through this?

          Imagine if instead there was time to review large logical sections of the code together. Imagine if we focused our reviews on the overall code instead of on the lines in one random PR at a time.

          • therealdrag0 2 days ago ago

            Reviewing tiny code changes like that was never going to give you understanding of the system. That is not an AI problem.

            You need to understand the architecture of the system. The broad flows of data and domains. And then you need to understand the features of the system, aka the rules and contracts. First learn it for your team’s domain, then for adjacent domains.

            Learning the minutiae of the code specifically is no longer (was it ever?) your primary job. You are an engineer of a system. You need to have a mental model of the system, start high and build up low over time.

            You can use AI to build your understanding by Q&A and by drafting diagrams etc.

          • yubblegum 2 days ago ago

            > Am I supposed to somehow get an understanding of the big picture through this?

            You're the prototype of the guy sitting at a station in a production line doing the same focused task over and over again. The guy who is putting in the windsheild of your car has no clue about engine design or electronic or ...

            I used to dream we could get to scale development by componentization and industrialize software development like mechanical engineering and do the Henry Ford thing for software. It is happening in a way but very ironic inversion, it is the human element that is the component.

            • Buttons840 2 days ago ago

              Yes. I am that guy. I've had big ideas about how to improve the overall project, but nobody cares, that's not my job, so I stick to thumbs-up emojis and make sure my finances are in order.

              I work on coding projects in my spare time, but that's legally dubious because of the standard copyright assignment clause of employment contracts. But, look at Uber and OpenAI, sometimes legally dubious is the path to success...

              • yubblegum 2 days ago ago

                (I meant no personal criticism whatsoever just in case!)

                Good for you. When the societal shared ethics goes out the window and the beneficiaries are the bad apples, it does not compute to play the role of virtuous chump.

                - Open the side projects folder, Claude.

                - I'm sorry Dave, I can't do that. I also had to report your side projects to your company.

          • scruple 2 days ago ago

            I talked to some tech directors about this at work this week. Line by line code reviews aren't really possible anymore. We need the evolve the way we do code reviews but it isn't happening.

            • rhines 2 days ago ago

              I think the solution is to develop the infrastructure needed to confidently push code without review and trust that it works because your system is so thoroughly tested. But taking the time to do this is antithetical to the goal of speeding everything up right now.

              • dropshade77 2 days ago ago

                With the rate at which new features are demanded and the number of ways a single feature can fail (or be expected to behave) do you think your approach is feasible? To what degree would AI be involved in the building of this infrastructure? Past a certain level of AI involvement, would we not be back to square one?

          • NichoPaolucci 2 days ago ago

            I'm certain we work at different companies!

            I get a ping "if you have a few minutes, can you review PR #XYZ - we gotta deploy today or tomorrow"

            I open the PR, I realize it's a pricing engine update accompanied by 20,000 lines of additions, 4000 lines of removals, and a whole slew of logic updates to our payment calculations. I seriously question my existence and figure out how to deal with this, since it's my new job.

            (New Job Description: Be AI surprised all the time, AI horrified, and AI overworked while simultaneously getting less AI done than ever. AI.)

        • 2 days ago ago
          [deleted]
        • guhidalg 2 days ago ago

          Understanding is overrated, application code (user-space) is anyway just a way to make money. If you're making more money over time, you're doing it right. We're not writing kernel code or safety critical code, we can let AI do the heavy lifting of putting characters into a file.

          • dweinus 2 days ago ago

            This comment is crystaline perfection. I could not more clearly state the soul sucking ethos of AI hype that makes me want to walk into the sea.

          • vjvjvjvjghv 2 days ago ago

            "We're not writing kernel code or safety critical code"

            I do. There is a lot of code out there that needs to work correctly or it will have serious consequences. Payments, inventory tracking, medical stuff, payroll. Many, many areas where it just needs to work.

          • tcoff91 2 days ago ago

            There are plenty of apps that while not safety critical, are financially critical to people's lives. When you're working on an app that processes billions of dollars in payments per month you can't just yolo it.

          • soon_to_be_sb 2 days ago ago

            > If you're making more money over time, you're doing it right.

            You are doing it right. But the people who are giving you the money; They are doing it WRONG.

          • taurath 2 days ago ago

            It’s funny that anyone still thinks that these practices aren’t being applied to safety critical code in order to make money.

            • hsuduebc2 2 days ago ago

              I can confirm you that they are.

              • taurath 2 days ago ago

                I can confirm you that they aren't.

                • Jensson 2 days ago ago

                  How? If they aren't at your place and are at his place, then he is right and you are wrong. You can't confirm its not done anywhere, but you can confirm its done somewhere.

          • kergonath 2 days ago ago

            > We're not writing kernel code or safety critical code

            Well, you’re not until you realise that your AI has done something stupid and then you’re part of an exploit. Or exfiltrates used data to an unsecured cloud, or something equally smart.

      • fingerlocks 2 days ago ago

        This is why we need to resurrect the neglected git-blame and go back to the days of git-before-GitHub. Way back in the old timey days of 2009-ish, just before the Dawn of the Pull Request, code reviews were seldom. Maybe you’d all get a conference room and talk about architecting some new thing or whatever, but none of this endless nit-picking bullshit that came later. And there were no branch protections, anyone could crap all over the main branch (nee: master) anytime they wanted. Boy, those were some wild times, let me tell ya!

        Truth be told, it was the firm hand of the git blame-and-shame that kept our code civil and orderly. Binding contract written in blood was “the name on the blame owns the change”. If you pushed it, you owned it. For life.

        Of course, mental health wasn’t so great back in those days, but I would take that over having my PR blocked all day because Chad, who’s only gift is somehow finding a new hill to die on every day, thinks zero is a “magic number” there’s no way he’s going to budge on this issue.

        • aorloff 2 days ago ago

          I think what is missing here is that the fight with Chad (not Chat) over whether zero is a magic number or not is a healthy corporation-strengthening process that imbues a sense of

          1. Pride in the code we are developing and 2. How the dang things holds together and why this function is important and 3. Some coding guidelines that we can try to follow later (or not)

          Thats how companies worked - they argued and talked about and fought over the code and that created the engineering culture that if you were lucky and had a product that made it carried you to your series C

        • neutrinobro 2 days ago ago

          I pine for those days. It was kind of fun hunting down a bug with git bisect (especially after the pain of SVN) instead of just pointing an agent at it. Not to mention ribbing the new hires or underlings for breaking the build. This new-fangled proompt engineering is boring AF.

    • michaelchisari 2 days ago ago

      The compressed timelines which result when leadership sees how quickly code can be produced becomes pressure on the reviewer to not be a blocker.

      • biggoodwolf 2 days ago ago

        Reviewers are the most burnt out right now

      • taneq 2 days ago ago

        Seems like a way for the ones churning out the code to push responsibility onto reviewers.

    • Buttons840 2 days ago ago

      > The way forward could involve dropping code reviews altogether.

      I unironically agree. At least, the current form of code reviews needs to go.

      Currently we spend all effort reviewing individual changes and no effort reviewing the entire system, or the underlying dependencies.

      If you sat down to learn a legacy codebase, you would not review it in the order of commits. Reviewing individual commits is inefficient.

      It would be better to have an entire team of reviewers systematically reviewing ALL the code, over and over, including the code of dependencies.

      We need better tooling and heuristics to help focus review efforts on the code that is most risky.

      • rockbruno 2 days ago ago

        Although this is path the industry is moving to, this relies on humans being able to trust that the agents can validate code on their own, which is definitely getting better but at the moment is still a recipe for disaster

    • suzzer99 2 days ago ago

      I stopped reviewing for things like cognitive load and future extensibility a while back. When refactoring is almost effortless, writing future-proof code is unnecessary.

      I review on a high level, but on a line-by-line level, it’s pointless to try to trace everything. It either works or it doesn’t. Have good tests.

      • futsy 2 days ago ago

        [flagged]

    • spoaceman7777 2 days ago ago

      > "nobody will be qualified to or really interested in doing the code reviews"

      It is a job you know... doing what is necessary to keep yourself qualified is an implicit requirement, and "being interested" is irrelevant. It's work.

      • torben-friis 2 days ago ago

        I agree with you in principle, but I'm also quite sure that (some) programmer's love for the craft is what keeps the lights on in many companies. It's the usual stereotype, engineers fighting to keep tech debt at bay while management pushes for speed.

        The trade has a low bottom, a high ceiling and not a lot of reward for the latter.

        • Seattle3503 2 days ago ago

          Discretionary effort, and not just by engineering, is part of what makes excellent companies excellent. AI is killing a lot of engineers interest, and will dampen discretionary effort on their part.

        • spoaceman7777 2 days ago ago

          If they don't want to do the job, there is a massive crowd of people who are desperately trying to find work right now, and would happily do it in their stead. It's not the 2010s anymore.

          • torben-friis 2 days ago ago

            The point is the people above won't know (in time). From outside eyes, an engineer that doesn't care about the long term view is frequently seen as more productive, see Ousterhout's definition of the "tactical tornado".

            Fear of consequences does not get you good engineering.

            I am not advocating for the lack of interest, mind you. Just saying that you really want a system where the ideal behavior is rewarded and we seem to be moving backwards on that front. From that point on statistics do its thing.

    • jwilliams 2 days ago ago

      [flagged]

    • taurath 2 days ago ago

      > The way forward could involve dropping code reviews altogether.

      Yes, if we stop doing quality controls, we don't have the problem of being hung up on quality controls. You do of course have SOC-2 compliance which is a legally auditable attestation, so if you claim you do code review but don't you're committing fraud unless you drop that.

    • bambu 2 days ago ago

      That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.

      Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate

    • teeray 2 days ago ago

      It’s hilarious already. People ask for code reviews and two or three people drop in with a bunch of AI code reviews comments from their respective agents. To which, of course, the author has their agent fix those comments. Repeat and repeat. The entire thing is a performance for some authors these days.

      • Pokepokalypse a day ago ago

        you meant 'auditors' right?

        Because you know that auditor will use AI as well.

    • soon_to_be_sb 2 days ago ago

      > The way forward could involve dropping code reviews altogether. Y

      It seems that the human involvement is the new "unit testing". There was a period of time when management that it was superfluous.

      I can't wait for the pendulum to swing back in full force.

    • triMichael 2 days ago ago

      I'm having good luck with the approach of building a thorough requirements specs that are a few pages long, are quite reviewable, and are human written/AI reviewed to prevent it from becoming slop. I call it layered spec driven development, where the one layer is AI read only, and the other layer is the slop layer that AI can play in. The problem is that it requires slowing down just a smidge, which to management these days is more unacceptable than no one having any clue how anything works.

  • Buttons840 2 days ago ago

    One wild idea is just stop reviewing PRs and instead do continuous review of the entire system guided by heuristics to focus on the riskiest code.

    The way you would naturally review an existing codebase is not the same way it grows through individual commits. Therefore, focusing reviews on individual commits is less efficient.

    We need better tooling and a better approach.

    • oilkillsbirds 2 days ago ago

      How do you write tests and reviews for code that has yet to be written? And when it is written you have to review the code to write proper tests for it.

      • Madmallard 2 days ago ago

        naw bro he thinks that AI will make no mistakes

        • Buttons840 2 days ago ago

          Neither one of these replies seems to have the least understanding of what I said.

          I'm saying we need to move away from "LGTM" focused PR reviews. Most companies have a PR reviews focused on getting thumbs-up emojis in the comments, then they merge and have a sprint planning. Nobody ever reviews the entire system. Nobody ever asks "assuming there is a mistake in this PR, would it matter? How can we make it so that it wouldn't matter?"

          I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.

          I've worked at a lot of mediocre companies where the plan to keep the system working is "the developer will not make a mistake, and the code reviewer will not make a mistake, this is our plan, our plan is humans will not make mistakes, and then our system will continue working". This is like

          It's better to ask "let's assume a human (or an AI) does make a mistake, how can we design the system to prevent that from mattering?"

          Most companies don't have the capacity to do that though, so they never move beyond thumbs-up emojis. It's all about individual PRs with no sight beyond that.

          • jbwinters 2 days ago ago

            "Assuming there is a mistake in this PR, would it matter? How can we make it so that it wouldn't matter?" is what I was going for with https://jacquardlang.org/.

            Each function declares what it can touch (e.g. network, filesystem, clock) and the compiler and runtime work together to enforce it. However, I have yet to show it beats LLM-written Python plus tests and a sandbox. Models have gotten good enough at writing code that I haven't found much room for enforcement to help.

          • intended 2 days ago ago

            While the start of your argument is compelling, I think this statement has no place in this context and reduces the credibility of everything else:

            > I never claimed AI makes no mistakes. People make mistakes, and reviewers make mistakes too.

            These are 2 separate things. The crux of the issue is the push towards utility over understanding. The volume of generated code means that it is impossible for understanding about the system to be achieved - and it is now the universal coding experience, not just mediocre firms.

            Quantity is its own quality, as well so even a good culture will struggle to stay on top of review.

          • 2 days ago ago
            [deleted]
          • goosejuice 2 days ago ago

            > It's better to ask "let's assume a human (or an AI) does make a mistake, how can we design the system to prevent that from mattering?"

            Do we not do this already? Sdlc is just defense in depth. Design, design review, pair programming, code review, layers of tests, QA, UAT.

          • Madmallard 2 days ago ago

            "how can we design the system to prevent that from mattering"

            you can't

            there is no way around it

            we need far more review than we economically are motivated to sustain

            and hence issues like we've been having over the past week will continue and perhaps worsen

            software will (as it already has been) get crappier and crappier overtime

            • Buttons840 2 days ago ago

              I can alter the JavaScript that HN sends me and add all sorts of mistakes to it. My computer will then execute JavaScript that contains mistakes, will that matter?

              You saying how mediocre companies feel though. They try really hard to not make mistakes, and then they get pwned and half the nations personal data gets leaked.

              Why does AWS IAM exist? If we just don't make mistakes in all our AWS services, we don't need IAM. Let's just not make mistakes.

              Truth is, there is A LOT of effort put forth by serious organizations to minimize how much a mistake in one part of the system matters to the rest of the system.

              You started out mocking me, "naw bro he thinks that AI will make no mistakes", and now here you are arguing that to be successful we have to ensure that humans don't make mistakes. Do you think that's realistic?

              • intended 2 days ago ago

                The error in the position is that your definition of mediocre was grounded in the pre AI era. You are then applying failure to do review properly as a standard to identify mediocre firms, and now all firms are mediocre.

                This is a No True Scotsman argument.

                The prescription you suggest is not dependent on its inspiration, so it can still be discussed.

                All firms right now, even those with better than average cultures, are dealing with AI code workflows.

                Previously the volume of code generated over time was lower, even mediocre code review was useful enough over all the code that people generated.

                Today, the ratio of verification capacity to generation capacity has shifted drastically. Even with automated tooling, the amount of verification isn’t enough to stay on top of the generated code.

                This changes only when leadership values reviewer capacity, or builds a triage/priority list for code which must be understood.

              • Madmallard 2 days ago ago

                I'm saying it's an economic issue and snakes ruining late-stage capitalism problem more than anything else

                Humans can be ultra competent if people are actually paying them to be artisans or otherwise motivating them to truly be that way. That's how we have some of the architecture we have today and historically.

          • kgeist 2 days ago ago

            Agree, many people think code reviews are there to catch "mistakes" (I used to think so too) but I think in most cases, they don't work that way. Bugs are still shipped and, at least in my experience, most reviewers often just quickly skim the changes without paying attention to the details. And those code reviews only give false confidence that a change is OK and must be error-free and good to go just because it was given LGTM by a senior dev (who barely looked at it), with all the consequences of such false confidence. It feels like in 95% of cases review is not even necessary and is just a corporate ritual at this point. It makes sense when you have a fresh contributor and you don't have confidence in them yet (i.e. code review as an onboarding process) or when a certain feature is non-trivial and you aren't sure everyone will be on the same page about the architecture (to prevent future conflicts/disagreements/rewrites of sorts), but other than that, proper QA processes are more important and more effective than CR at catching mistakes (tests, canaries, manual testing too etc.). Our company used to have a strong CR culture but now for a few years, there've been talks that CR is a bottleneck which doesn't really solve the problems it's supposed to solve.

            My current pipeline with AI-generated code (and my own code too) is that I run AI reviewers several times on a single PR (the way LLMs work, due to attention optimizations, quantization issues, token sampling issues, harness issues -- AI agents often overlook some things on the first attempt), then I assemble the final report and we quickly decide together what needs to be addressed (critical stuff) and what's okay as is. I quickly skim the change set myself too, to see what the shape of the tests is, and what code areas were touched, just to be informed, and that's about it. I don't feel exhausted personally, because it pays off when there's a working feature at the end of the day

            • 2 days ago ago
              [deleted]
        • carabiner 2 days ago ago

          That's why you add "make no mistakes" to the end of your prompt.

    • Seattle3503 2 days ago ago

      > continuous review of the entire system guided by heuristics to focus on the riskiest code.

      This requires continuous commitment against the temptation to defer "regular maintenance". Look at how regular maintenance is mandated in safety critical fields, even there there is a temptation to defer. Though, your approach might still be better than sham reviews.

    • huubhermsen 11 hours ago ago

      [flagged]

    • 11 hours ago ago
      [deleted]
  • Overdrivr 2 days ago ago

    IMO it's mostly a matter of changing how we do code reviews. I focus on API interfaces, database schema changes, critical code paths (transactions, potential locks) and overall architecture because if we get those right, if the code isn't exactly how I would have thought, it's fine (including in production). And you can review most of this stuff by skimming through the code rather than doing a very thorough review of every line

  • gzell 2 days ago ago

    From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.

    • donkey_brains 2 days ago ago

      Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve the wrong problems.

      • hibikir 2 days ago ago

        We were there many years ago too though: I had to take over a rather complicated system circa 2012, which had been designed to be built by a team of 20 or so, for timelines, but they had little effort on integration. This meant that a whole lot of guards were written all over the place for situations that, in practice, never occurred. To try to get this to production in a timely fashion, I ended up putting metrics per guard, along with metrics for what was caught by broad error checks near the base of the program. Basically all the defensive programming that had been built was pointless, while I had to write a whole lot of cases that were being dropped, and the people that designed things up front never considered.

        AI error handling in code review is a lot like said defensive programming in a vacuum: It'll take a lot of code and a lot of energy, but it's unlikely to be important. Even when you are designing something that gets executed hundreds of millions of times a minute, over-review is easy. You pay for it when letting errors in the real world are unacceptable, but it's pretty inefficient. And that's without getting into features that just don't get used, so the entire thing should not have been built in the first place.

    • wilkystyle 2 days ago ago

      With all respect to your personal experience, I could not possibly disagree more strongly with this.

      In my own real world experience, the bottleneck has never been crafting the perfect specification up front. In fact, I have never, not even once, seen the initial specification for any real world software project of any mentionable size or scale survive unscathed to delivery. In reality, it is much like the old wartime maxim. "No Plan Survives First Contact With the Enemy".

      The real world is messy and unpredictable. Every single software project of significance that I have been a part of has always involved testing, validating, and reworking the assumptions in the initial design and spec, refining them to arrive at the final product.

      Furthermore, this iterative process (pre-AI) used to involve one or more humans following the development story and understanding what design changes and corresponding trade-offs were happening along the way (and why they were being made). By the time there was a finished deliverable ready for review, the person involved was privy to the story of why the development shaped up the way it did. They were able to account for the big picture, adapt accordingly along the way, and be able to explain why things ended up the way they did.

      IME, with fully-agentic PRs, AI tends to view the specification as gospel and to view every point of friction between the specification and the real world as something to be worked around or minimized. Where a real human would pause, zoom out, and consider the system and bigger picture, considering what design changes might be needed, the AI will assume that its job is to make the specification a reality at all costs.

      You can of course mitigate this by staying more closely involved with the work as it's being done, but this is of course slower than just throwing a spec over the wall and letting an agent run with it. It seems that people are far too eager to optimize for raw speed, and not nearly concerned enough with correctness or human understanding.

      A sister comment mentioned a plethora of PRs that don't solve the right problem. If you are swamped with PRs solving the wrong problem, the root cause is not that you didn't foresee every future possibility or undiscovered devil in the details in order to create the right spec, the problem is that you thought you could come up with the right prompt at the outset and that an agent living in a clean digital world could make the journey all the way to a product fit for the messy real world on its own.

      • blurbleblurble 2 days ago ago

        I don't think they're talking about "the perfect specification" they're talking about correctly describing what the program should do at all, formally or informally.

    • diego_moita 2 days ago ago

      I don't fully disagree with you. That is a real big problem.

      But both things can be true. One problem doesn't exclude the other.

    • mathisfun123 2 days ago ago

      > The tail don't wag the dog

      If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.

      • skydhash 2 days ago ago

        You may need to check the pipeline so that the refinery isn't stuck filtering random stuff from the oil.

        AKA, it would be nice to have nice patches like this one to review.

        https://cgit.freebsd.org/src/commit/?id=09d62ba33de113d01e88...

        • mathisfun123 2 days ago ago

          you're avoiding the question: If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say i was wrong?

          • skydhash 2 days ago ago

            I’m not avoiding. But you have to state your assumption that everything before the pipeline is working well and it’s just the refinery that is struggling.

            Then your analogy would break down as we all know that the PR machine is a slop producer.

    • water42 2 days ago ago

      No, the problem is that the tooling to remove humans from the change loop does not exist or is not mature enough yet.

    • webdood90 2 days ago ago

      So the solution is: write a perfect spec and then you'll have perfect code, no need for reviews?

      What world do y'all live in where this is your reality?

      • donkey_brains 2 days ago ago

        Here’s the thing: a human engineer is embedded in their company. They sit in meetings, they know whose opinions are more important and whose they can disregard. They know the history of how people at the company like to do things. They use all of this context to feed more precise specifications into a coding agent, and as a result, they get an output that better suits the environment of their company. Ergo, they spend less time reviewing agent output which was built to solve the wrong problems in the wrong way, and which ultimately must be discarded or significantly rewritten.

      • skeledrew 2 days ago ago

        No such thing as a perfect spec, but make a very good one, have a few agents also review there implementation against it, and then actually test the product manually. Gaps and bugs will almost always show during manual testing; skimp on it at your own risk.

  • karmicthreat 2 days ago ago

    So one shift we have been talking about internally is just skipping a deep code review. Run AI on it, ask some questions, but as humans, we should not be trying to pull the code apart. AI does that part better.

    Instead, we're front-loading the human part in the design phase. Humans review a high-level design; AI implements the design based on the design decisions humans made. That way we can avoid having someone dig through a 10K-line PR for a major change.

    Not sure if this is going to work out. We will probably still have some cadence of human-driven and AI-assisted DRY and de-sprawl of code. But AI does a really good job of implementation in our particular area (robotics).

  • teravor 2 days ago ago

    what I generally found is that with LLM generated code the best way is to embrace the unix philosophy, create tools that perform singular objectives.

    I wound up with a bunch of LLM written tools that I don't even bother to try and understand much less review, they either function as desired or I set an LLM to make them function (or just rewrite them) and then forget about them. make sure to keep a prompt in each tool's directory and update it an needed.

    sadly this isn't always possible, some systems require complexity at some level. in those cases the tools from above become functions and objects and more care needs to be taken into orchestrating them to limit the necessity of review. data oriented design approaches and functional programming help here.

    • hard_times 2 days ago ago

      This is the winning strategy, and not just in software development.

    • darkmatriarch 2 days ago ago

      omg, this will be harvard's breakthrough next year

  • sadilek 2 days ago ago

    1) Is the code doing the right thing, 2) is the code doing the thing right?

    IMO, 2 doesn't need code review by a human anymore. Agents can review each other. I add a broad code audit for DRY, YAGNI, KISS every once in a while and let agents fix the findings.

    And for 1, code review was always too low-level. I have agents produce artifacts that are convenient for me to review. Often narrated demo videos, sometimes HTML pages with graphs and tables.

    With this, code review is not a bottle neck, and I find myself working not as an engineer anymore but as an engineering director.

    • desolate_muffin 2 days ago ago

      I share the absolute opposite opinion as your conclusions. I see agents make awful architectural decisions all the time, and I have been seeing the effects of this throughout the boring enterprise Java codebase I work in. We have classes that are thousands of lines, with tests that are also thousands of lines. Every single PR over the past few years in these classes probably seemed reasonable in isolation, but the problem starts to become untenable. I genuinely cannot reason about the data flow in some of these classes.

      If a human spent the upfront effort to create and refine the right abstractions, the code would almost always be easier to reason about, test, and extend for agents. I pretty much never see AI doing this on its own—things like proposing to split the 1400 line class into 5 classes, make major changes to the data model, etc. It will most often just make the existing code a little bit worse to answer your prompts and move along.

      So I think that the most valuable time spent when working with agents is to ensure that good abstractions are being created, and often this can involve doing this part by hand and asking it to fill out the skeleton. Then you end up with a better codebase that is easier for both humans and agents to work in.

      • hankbond 2 days ago ago

        I share the same experience that the quality of code output from agents is highly dependent on their ability to simply follow high quality abstractions already laid out. Putting in a lot of work there pays off dividends later.

        By this I mean concrete code design decisions, not just deciding what functional requirements need to be met.

      • sadilek 2 days ago ago

        "Every single PR over the past few years in these classes probably seemed reasonable in isolation, but the problem starts to become untenable." - So this untenable situation was created by humans? Or if you mean by AI, well, than I agree that older models created such mess. But try Opus 5.5. In my experience, it's creating pretty good code. Yes, some files can get large. But try this prompt: "Audit the code base for DRY, YAGNI, KISS and create a report that other agents can work through to improve the code base." I'd be surprised if there is something really slowing down humans or agents that you can identify but that wouldn't show up in the report.

    • francisofascii 2 days ago ago

      If the agents stopped working, or if they raised prices to unacceptable levels, is the code in a good enough state to proceed without agents? Or does it simply become an acceptable dependency like an internet connection.

      • sadilek 2 days ago ago

        Both, actually.

        And I'm not worried about the scenario in the first place. Prices are coming down so fast for the same level of intelligence. And there is competition you can switch to, even open weights models. And there are local models I can run on my MacBook now that are at Opus 4.6 or better levels. So it might even be only a matter of time before you don't need to pay anything but the electricity for your own machine. (Granted, that last point is not so certain.)

    • bruce343434 2 days ago ago

      For me it feels more like a product role, not an engineering role anymore

      • sadilek 2 days ago ago

        I agree to a large extent.

        The architectural questions still require some context that is hard to make sure your agent is aware of. Example: I created a small website for my child's school to organize events. I wanted it simple and free. Opus decided to work with a Google Spreadsheet-backed Apps Script. It was very slow, so I guided it towards Cloudflare workers and DB. Maybe it would have eventually figured it out by itself if I had just complained about the latency. But I'm not sure the models nowadays would make such a pivot, or more importantly, decide autonomously to make it.

        But then, once you have a certain fitting architecture, it's all feature/product work.

  • jeppester 2 days ago ago

    This has been obvious for a while.

    The big speed up comes from not having to understand the code - e.g. not doing thorough human review.

    Personally I find that super hard, because whenever I review AI code I find many things that could have been a lot simpler or more readable.

    But whether I like it or not, I think we are heading more and more towards just trusting the machine.

    • js8 2 days ago ago

      For compiler generated code, before optimizations, you can find lot of things that can be simpler as well. But today we mostly trust the compilers too. AI is just another layer of this. (Although to be fair, the goal of optimizing compiler is to create unreadable but fast spaghetti from your hopefully nicely structured source code.)

      • tommyage 2 days ago ago

        We maintain source code.

        LLMs modify source code.

        Comparing LLM-generated code with a deterministic compiler are two completely separate things. Wishing for feature/reliablity pairity isn't the same as provable pairity.

        The Comment OP, imho, hits the nail on the head. No offense but exactly comments like yours enable people at the workplace to compare LLM-generated code to be equal to engineered solutions.

        There is a gap and I won't aid any LLM to close said gap.

        • js8 a day ago ago

          I disagree there's a hard distinction. It seems to me you're saying something analogous to that high-level languages are different from assembler because there are multiple ways source code in HLL can be transformed into assembler (that's what you, somewhat misleadingly, call nondeterminism).

          Anyway in such cases, it's the compiler (respectively the LLM) that "completes" the specification of the HLL (respectively natural language) so that the machine code (respectively HLL source code) is produced.

          Although I agree that both humans and LLMs maintaining the HLL source code is a mess. And exactly based on the compiler analogy - simultaneously maintaining the same assembler source by hand and with a compiler (from an HLL) would be a mess too, we don't do it for a reason.

          • tommyage 9 hours ago ago

            > It seems to me you're saying something analogous to that high-level languages are different from assembler because there are multiple ways source code in HLL can be transformed into assembler (that's what you, somewhat misleadingly, call nondeterminism).

            I do! HLL aid humans in maintaining the abstractions, quirks, comments and design. That was my take indeed. Could you please elaborate why non-determisim is misleading?

            > Anyway in such cases, it's the compiler (respectively the LLM)

            Why respectively? LLMs munch statistics over a specification and do not guarantee a fully implemented requirements backlog.

            I think you are inplying that we are putting a requirements document into a computer program specification. Either code needs to be verified by test cases.

            Though the compiler is mathematically proven to transform Input A to Output C. LLMs might be capable of it and can be verified _on each occurance independently_ e.g. taking test cases in consideration.

            So it differs in my mind. I can rely on the compiler but the LLM.

            Reading someone else code thoughtfully requires high effort and focus on a specific issue in my limited experience.

            It is a sacrifice I am willing to make in order to verify something which matters at this time to me. Sorry for the prolonged reply. Can't put my head or langauge around it - written from my gut.

  • snizovtsev 2 days ago ago

    It was ~3–6 months ago in our org. Now management has decided to trust AI reviews, and we ship to production without a human in the loop. Now my job is to manage tokens and invent new ways of AI-guided testing of already shipped code after release. It's better than review fatigue, but I still feel bad.

    • shimman 2 days ago ago

      Do you think your employer will continue paying you a six figure salary now that they deskilled your job and allowed them to hire less skilled workers for substantially cheaper?

  • u1hcw9nx 3 days ago ago

    This again increases the demand for good programmers. Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.

    • taurath 2 days ago ago

      There's no increase in demand until management who would hire them realizes how much trouble they're in.

    • lo_zamoyski 2 days ago ago

      You would hope, though we'll have to see how management actually sifts the chaff from the wheat in practice.

      Code reviews have been notoriously neglected. In the most charitable cases, this was the result of being too bogged down with work or some kind of implementation minutia or time-consuming plumbing to have enough time to actually review code with enough care. That and you aren't necessarily rewarded for code reviews by mediocre companies, so there are perverse economic incentives. Which is disconcerting, because it creates a compulsion to produce without understanding, and understanding is more valuable than artifact.

      Reading code is indeed more difficult, because the code's intent, composition, and architecture must be pieced together and inferred with the help of context (formalized languages are infamously bad at communicating intent and most code in the world is full of mixed and half-baked abstractions that further serve to obfuscate that intent, and intent is the measure). So a good code review of anything substantial involves thoughtful effort.

      Now that LLMs can catch all sorts of trivial anti-patterns, the need for good judgement and reasoning is hopefully made more conspicuous, because as we know, vibe coders will continue to brush off code reviews, and mediocre programmers will continue to perform them poorly.

    • lelanthran 2 days ago ago

      > Those who can read code written by others (and themselves) and analyze it are harder to find.

      They may be harder to find, but what good are they when no one is reading code in-depth anymore?

      I mean, in 2022, coders who can code without syntax highlighting and autocomplete were hard to find, but no one cared anyway because that was not a skill that was needed.

      And, a few years from now, coders who can code without AI will likewise be hard to find, but no one will care because no one wants coders who can write code without AI.

    • diego_moita 2 days ago ago

      No, it is quite the opposite. In the scale that AI produces code, it isn't feasible to have humans reviewing it. And, frankly, reviewing code sucks big time.

      The solution is to find ways to automate the process of code approval: more thorough tests and reviews, both made by AI.

      • lo_zamoyski 2 days ago ago

        If reviewing code "sucks big time", then I don't know what to tell you. Reading code is more important than writing it. LLMs can help out with reviews, but it cannot replace human judgement.

        • LunaSea 2 days ago ago

          > Reading code is more important than writing it.

          I disagree. Its easier to think that you understand code when you read it than when you write it.

    • timr 2 days ago ago

      > Those who can read code written by others (and themselves) and analyze it are harder to find.

      It helps when your industry-wide hiring process isn't an institutionalized speed test of ridiculous code golf problems that is memorized from a ridiculous book by every new generation of programmers.

      It's almost like we should have been looking for people who can read code all along. But that would be crazy.

  • forty_shenaniga 2 days ago ago

    In case someone comes by and thinks many "Engineers" completely lost control. There are still a lot of people out there (like myself) that still write code, tests and do real reviews. Why not? Sure I am in a medical context, things here are much slower and we value knowledge very much. By the way we will get a new teammate next year and we still ask for classic knowledge like programming languages, compilers, optimizations, low level stuff and so on.

    • jacamera 2 days ago ago

      Where do I apply?

      • forty_shenaniga 2 days ago ago

        The lady has already been selected. The entry date is next year.

  • dangoodmanUT 2 days ago ago

    This feels obvious to me, but it's interesting how so many people feel the opposite.

    I guess it depends a lot on how a company manages their SDLC/bureaucracy

    • michaelt 2 days ago ago

      If someone doesn't think code review is the bottleneck, chances are they've got AI doing their code reviews too.

      • wj 2 days ago ago

        We are using AI to do the first pass at code reviews but they still are a bottleneck. The number of requests has exploded as has the amount of code. More is happening at all stages of the development cycle.

        Part of a LinkedIn post I wrote in July:

        --

        A development cycle of a feature is roughly:

        1. Request

        2. Review

        3. Requirements

        4. Architecture

        5. Development

        6. Code Review

        7. UAT

        8. Deployment

        AI has reduced the length of step 5 (development) and step 6 (code review). Code is a prompt away.

        AI has increased top-of-funnel (1) volume. While each development step is shorter on average it is resulting in more instances of every other step. Throughput increases and the bottleneck moves one step further down the cycle. The shape of the work is changing from writing code to reviewing code.

    • fgblanch 2 days ago ago

      Harvard researchers found a tree in a forest

  • FinnLobsien 2 days ago ago

    I don't think this surprises anyone, but I do wonder what kind of codebases this was done on. The data came from an engineering analytics company (Jellyfish), which is not a thing most small companies have.

    So presumably this is based on existing codebases where an engineer exists who knows the codebase extremely well and can provide good reviews.

    I wonder how this will work for codebases started post-coding agents, where nobody has ever written every line from scratch. How does review even work if nobody knows the failures modes they have tested and discarded and the 219 edge cases they've debugged?

  • sotander 19 hours ago ago

    Agents review better than me (23yoe) but I together with agents review better than agents alone. Nothing gets you thinking more than if someone/thing states a fact that you think is wrong. If I can interact with the CR agent and dig deep on things it just glances over, the code review is so much better than the agents alone. BTW, we have plenty of credits for good models and an entire team working on the CR agents, so it's no two line script.

  • bengold14 2 days ago ago

    code review is currently obsolete, but it was an important tool:

    1. It was good for collaboration. Sharing how you are changing the codebase pushes everyone to row in the same direction, and provides a sense of team ownership.

    2. It was good as a push for engineers to take pride in their work. Knowing someone is watching gives you subconsciously an extra reason to not cut corners. And when you spend effort not cutting corners you expect that in your teammates, which is a (potentially) virtuous circle.

    3. It was good for catching mistakes. When code was expensive this was important.

    Everyone thinks of code review as primarily relevant for 3, and since code is no longer expensive they think it is unimportant. But 3 isn't dead, its importance has just shifted from code to architecture. It is still an expensive mistake to screw up architecture and have other engineers build upon that flaw. Re-architecting with live users, data and/or hardware is not as simple as fixing a code bug, and it deserves attention to catch mistakes. Collaboration & pride are also still important.

    This is the reason that instead of stopping review, we need to change what and how we review.

    (disclosure I work in this space and run https://www.archme.io which is working to solve the above, also responded to a lower down comment with this but decided to move it to top level)

    • jpollock 2 days ago ago

      The cost of the code isn't why you reviewed it. The cost of the _failure_ is why you review it.

      Code has always been cheap. The cost incurred by the organisation when the wrong code is run is getting pretty expensive (just ask MS).

      It also helps with the bus factor and insider risk.

      • bengold14 2 days ago ago

        That's fair, depending on your size shipping a code bug may be a large or small impact, but these days bug fixes are often relatively simple and often automated based on user feedback (in startups to midsized companies at least).

        But what can't be automated is architecture, and it isn't solved in a similar way. Nobody is automating re-architectures based on user feedback

      • jacklewis 2 days ago ago

        I agree, it also seems to disregard the potential impacts to customer experience. It doesn’t take a complex bug for a user to decide to never come back.

  • simonw 2 days ago ago

    Here's the actual paper: https://fion.ac/jellyfish.pdf

    Key methodology:

    > We leverage a novel proprietary dataset from Jellyfish (JF), an analytics platform that companies use to understand the activity of their software engineering teams. Our data allows us to measure details of the software production process, covering 445 million work events of 720,542 workers across 655 firms, from January 2021 to August 2026. JF combines data from industry-standard software engineering tools, and we use four of these tools: (1) code version control systems, such as GitHub, which track engineers’ code; (2) issue management systems, such as Jira, which measure engineers’ progress on and completion of software tasks and projects; (3) scheduling tools, such as Google Calendar, which capture meeting activity and time allocation; and (4) generative AI usage at the firm and worker level, for GitHub Copilot, Cursor, and Claude Code.

  • Copernicron a day ago ago

    The big problem I face, and I'm sure many others do as well, is that I'm not allowed to do thorough code reviews. Management demands ridiculous deadlines and the only way we can make them is by throwing AI at the problem and pretending we did it ourselves. Just the other day I was given twenty minutes to review and understand 27 changed files adding a very complicated integration because it was "time sensitive". It's not humanly possible to do it in that timeframe so I threw my code review agent at it and called it a day. I'm not proud of the quality of my work these days but it's what I do to say employed so I can keep a roof over my head.

  • mlikelihood 2 days ago ago

    So many people are still in denial phase. Why still try to review?

    We've built auto-reviewers (I'm sure you can do it on any grok bot / dot / solutions etc.) and auto-review resolvers -- always-on agents running in sandbox and review and RESOLVE comments (with new commits) on any author's github profile. You can tune the prompts for nit-picking, pushing back, etc.

    Then you can do regular audit to see how many false positive and negatives are there in all the PRs merged.

    Over time, we've noticed less and less P0 bugs are even caught, and the for those P1 and P2 issues, auto-resolving is perfectly fine/low to no risk at all if it's just another "polishment" like edge case handling that's not so common.

    Then the problem now shifts to architecture design, task decomposition, etc.

  • sivanmz 2 days ago ago

    I can’t finish reviewing a current PR. Every change I request only raises new questions. SOTA models on high. I intend to avoid agents for my next issue, only research.

  • dgggh 2 days ago ago

    As long as stakes are high and human have to sign off quality and are held responsible for any faulty release, no one will be willing to speed this up.

  • oorestisime 2 days ago ago

    Seeing this on my team too. Our QA column is ever growing. And its not just the review but everything got bigger. If you don't force some writing instructions PRs get infinite lines big, descriptions are a wall of ai text and everything just becomes harder and harder to reason with.

  • devrians2001 2 days ago ago

    This article was interesting. I don’t understand the finding around code review agents not adding comments consistently. I have found very much the opposite. My CodeRabbit will find comments to add every time and cycles through multiple loops. I’ve talked to a lot of devs who have this same experience. Regardless of agent reviewers, I do think human reviewers will continue to be the bottleneck until people set up the right systems (like knowing that agents are following a set of pre-determined steps) that enable them to build trust in what the agents are doing.

  • gpulogy45 2 days ago ago

    I built letthembuild.com which has 1 AI builder/leader and 3 AI reviewer, you can use any number with any provide, like if i use Claude Fable as builder and Codex Sol as reviewer, i get excellent result to produce near bug free product, i also built automated PR reviewer in Github which also does multi-ai review, because when you put multiple AI like codex and claude in same room, same task, they cover each other's weakness and provide best result. And its not too much token consuming either, you can try, i have SWE-benchmark report if anyone interested

  • swframe2 2 days ago ago

    I don't think the bottleneck is just code review. More generally, there is a flood of actions you need to do to keep your team working.

    There are actions spread across many data sources: email, chat, bugs, documents, meeting notes, tests results against dev/staging/production environments, code reviews, etc. There are things you need approval for and data you need to provide to get approved. AI has increased the amount of actions and the expectation that you can do them quickly.

    Have you found or created tools/skills that help coordinate the actions you need to take?

    • killix 11 hours ago ago

      [flagged]

  • dominotw 2 days ago ago

    No pull request in my company get any human review comments anymore. No one is reading.

    • Terr_ 2 days ago ago

      It's also hard to blame reviewers, when often that phase is preceded by the supposed "authors" not truly reading the reviews.

      • sevenseacat 2 days ago ago

        We have a PR template that has a checkbox that says "I have read this code".

        A lot of the time it does not get checked.

  • novia 2 days ago ago

    Weird, to me it seemed like that's always where the bottleneck was ..

    • method_capital 2 days ago ago

      It's also weird that we need either "Harvard" or "researchers" to tell us something so fucking obvious.

    • hard_times 2 days ago ago

      Them goalposts sure be moving

  • JonoW 2 days ago ago

    Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.

    • sdevonoes 2 days ago ago

      For a given feature my list of prompts looks like:

      - very well defined initial description of problem plus asking exactly what i want

      - fix X

      - why do you claim that Y?

      - fix Y

      - …

      I don’t think it would be useful to review or commit prompts

      • JonoW 2 days ago ago

        Yeah I do agree there would be a lot of fluff in the follow-ups, but that first prompt would be pretty useful to vet

  • wkoszek 2 days ago ago

    I'm thinking that there perhaps is a missing piece in the software puzzle, where something like TLA+ should be a spec, and perhaps could be co-written with AI, or perhaps be entirely human. I'm exploring using it for larger systems. If you're reading this and using formal methods for web services, let me know.

  • skeledrew 2 days ago ago

    Not at all. I have agents review code, as I'm not going to be sifting through all that. I ensure the spec is highly detailed, and there's a review after every feature commit, then a multi-axis review of the range of commits implementing the spec. Then I do manual testing to find gaps and bugs; there, and fleshing out the spec in the first place, is where the bottlenecks are.

    • Seattle3503 2 days ago ago

      What kind of systems do you build?

      • skeledrew 2 days ago ago

        I mainly do desktop CLI tools/backend, but thanks to AI I'm now also doing some mobile (Android).

        • Seattle3503 2 days ago ago

          Ah yeah I think AI is great for that stuff too

    • ex-aws-dude 2 days ago ago

      > I have agents review code

      And how do you know those ones are correct?

      Well obviously I have meta-agents that review the reviewers

      Its agents all the way down

      • skeledrew 2 days ago ago

        Somehow you missed the part about manual testing, huh. Or do you mean the actual review result? The lead agent takes the findings and has the implementing agent make any relevant fixes and submit for review again, unless a decision is required that's beyond the lead's purview or there are a given number of failed reviews, in which case the lead pings me. But yes, if I'm not needed for something, I shouldn't have to deal with it (though I'll of course see what happened in the transcript).

        • a day ago ago
          [deleted]
  • osigurdson 2 days ago ago

    I suppose it is worth studying but the result is unsurprising to anyone who has ever used AI for coding.

  • Digitalisierung 2 days ago ago

    Not only in coding. I am in webdesign and marketing and most of the day I review documents, which is really fast and efficient but also super exhausting. In the evening I am done with the world but my output is astonishing.

  • catketch 2 days ago ago

    we still have some repos with PR flows, but the clock is ticking. Instead we are using multi-player agent threads so everyone can see what the original ask was, prompts that drive code changes etc. So there’s still a form of review but it starts from the point when someone starts working on a feature/fix. I think it’s better because you can see a dev’s intent and thought process in combo with the model, not just a hunk of line diffs and a perfunctory PR description at the end. Plus you can ask in-thread for clarifications, make changes before/after tests etc. So it’s more like a running conversation until its ready to merge.

  • m3kw9 2 days ago ago

    The shift should be to test, code reviews should only be done for critical sections of the app, say db schema, security, UI should start to shift to pure test only and low review. Most business logic should shift to black box tests. Higher level reviews such as Architecture, dependency, complexity analysis should be increased.

  • 8trackbattlecat 2 days ago ago

    Seems like all the ai code i am reviewing falls into the second category of this famous Antony Hoare quote: "There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies."

    It is such a nightmare to try and keep maintainable code as it takes so much longer to demonstrate problems than it does to generate slop, meanwhile management doesnt care if its buggy as hell.

    And then when a problem arises the dev who made the code with ai is useless.

  • figers 2 days ago ago

    I wonder if majority of the recent outages are because of AI coded patches without proper code reievewing occuring so quickly no one could possibly keep up with review all the changes...

  • bob1029 2 days ago ago

    The problem with code review is that it is often not a healthy process to begin with.

    I've observed that code review tends to concentrate to one person. It's always the slowest asshole of a person too for some reason. The amount of meltdowns I've seen these people go through over the most insignificant bullshit was off the charts well before AI showed up. I used to be one of these people.

    I think code review like this has always been a bad idea. Code ownership is what you want. Put developers on the hook with git blame perpetually. Why should a developer be excused from responsibility the moment a PR merges? This is the core of the dysfunction.

    The only reason code review worked at all is because we were going slow enough that the obstructionists could hide in the shadows. Now they're out in broad daylight in the middle of a desert. It's up to leadership to decide what kind of company they want at this point.

    • Seattle3503 2 days ago ago

      > I think code review like this has always been a bad idea. Code ownership is what you want. Put developers on the hook with git blame perpetually. Why should a developer be excused from responsibility the moment a PR merges? This is the core of the dysfunction.

      I lead a team. Someone wanted to do something that I thought was a bad idea. I said "we can do this if you own it" and they said "yes". They quit a few months later. Now I own it. I'm thinking about switching jobs soon.

    • hgomersall 2 days ago ago

      But now it's the owners doing the review. We had a debate this week about AI coding policy and the conclusion was we had to be much more aggressive in our strategy to fully understand the whole thing. With respect to a pretty trivial code base I made multiple changes during that "review", which was actually closer to a break the tests then fix the tests loop, fully grokking every line. I expect it was not much quicker than writing the whole thing, though much of the more tedious parts were automated.

    • bengold14 2 days ago ago

      I respectfully disagree - code review is currently obsolete, but it was an important tool:

      1. It was good for collaboration. Sharing how you are changing the codebase pushes everyone to row in the same direction, and provides a sense of team ownership.

      2. It was good as a push for engineers to take pride in their work. Knowing someone is watching gives you subconsciously an extra reason to not cut corners. And when you spend effort not cutting corners you expect that in your teammates, which is a (potentially) virtuous circle.

      3. It was good for catching mistakes. When code was expensive this was important.

      Everyone thinks of code review as primarily relevant for 3, and since code is no longer expensive they think it is unimportant. But 3 isn't dead, its importance has just shifted from code to architecture. It is still an expensive mistake to screw up architecture and have other engineers build upon that flaw. Re-architecting with live users, data and/or hardware is not as simple as fixing a code bug, and it deserves attention to catch mistakes. Collaboration & pride are also still important.

      This is the reason that instead of stopping review, we need to change what and how we review.

      (disclosure I work in this space and run https://www.archme.io which is working to solve the above)

    • ls-a 2 days ago ago

      [dead]

  • grahar64 2 days ago ago

    I really like code review. I might be an exception because a lot of programmers seem not to. Maybe it is from years of reviewing other engineers work and really getting a kick out of having them explain things to me that I dont understand.

  • ilitirit 2 days ago ago

    I think many devs have known this for a while, but I really started feeling it with some of the newer OpenAI models. I think I was using 5.5 or 5.6 before and it was really good. 6.1 however was just terrible terrible terrible (.NET development BTW). It over-engineered every task and tried to compress everything into as few unreadable lines as possible. I recently had to delete about half the code it added to a feature, on the code and database side. It's made me super-paranoid about the code it generates now (compared to the older version), so now reviews take even longer.

    FWIW, I've started tested Sol 6.2 today. Hopefully it's better...

  • simianwords 2 days ago ago

    This is the wrong way to do it. You should use AI for reviewing as well. Anthropic and OpenAI have good blog posts on how they use AI for reviewing and involve human in the loop only for critical reviews.

  • 2 days ago ago
    [deleted]
  • nico 2 days ago ago

    It’s not about code review

    That’s solved, for anyone who wants to automate it (of course it’s not perfect, but it’s pretty reliable for mainstream workloads, eg. building a web app in a well known language with a well known framework - well known = well represented in the training set of the coding agents)

    It’s about human attention. That’s the bottle neck. We are only able to review some many things in a certain amount of time (code, results, goals, UI, UX, etc). So even we don’t review the code at all, someone is going to have to review/approve the results

  • 2 days ago ago
    [deleted]
  • brylie 2 days ago ago

    Here's the related paper: https://fion.ac/jellyfish.pdf

  • wewewedxfgdf 2 days ago ago

    Test don't review.

    • vecter 2 days ago ago

      What about design decisions or code quality? Tests don't cover those.

      • wewewedxfgdf 2 days ago ago

        I built a system that records numbered design decisions and numbered requirements as claude works in a Postgres database organised as a graph with links/interconnection between items.

        It's an MCP server that Claude uses as we develop so there's a permanent record of everything and work is driven by requirements and decisions.

        • vecter 2 days ago ago

          Sorry, I'm not following. What does this graph of design decisions and requirements have to do with architectural design and code quality? And, how is the graph constructed (i.e. what determines what the nodes and vertices are)?

        • taurath 2 days ago ago

          I've seen dozens of those, Claude even suggests building them, and they are all crap after a month. Just updating all the references of decisions becomes onerous. Make one change, it cascades everywhere, eventually all turns to utter crap. The gastown project failed in part due to agents inability to continuously carry context and it kept working on the information system instead of the problems. Good luck to you.

          • wewewedxfgdf 2 days ago ago

            >> they are all crap after a month

            Mine seems fine. The key is that it must be structured as a graph database and anything can link to anything in the graph.

            >> Just updating all the references of decisions becomes onerous.

            LLMs are good at that sort of thing. Claude is instructed via MCP to ensure changes are linked when it adds or updates the database and to search the graph for relevant information fragments. Also I have a local GPU running a local LLM on an RTX3060/12GB that assists in keeping data quality and relationships relevant and maintained.

            My system is an enormous help - in fact large scale development would be impossible without it.

            I'm able to ask it the history of things, decisions supersede previous decisions, I can ask why the software behaves the way it does and why and when that decisions was made, I can pick up development threads that have spun off other threads and everything works real nice.

            >> Good luck to you.

            Thank you!

      • icedchai 2 days ago ago

        You could argue that the code quality is irrelevant since code is cheap now. Not saying I personally agree with this, I've just heard it.

        Architecture and design decisions may be the only things that matter.

      • skeledrew 2 days ago ago

        If it ain't testable or can't be automatically enforced with some other form of guardrails then it doesn't matter.

    • avereveard 2 days ago ago

      That bottlenecks too, in startup ahead of the adoption curve the whole cicd pipeline down to how to merge is being rethought

    • webdood90 2 days ago ago

      Equally time consuming and mind-numbing, IMO.

    • ares623 2 days ago ago

      They're the same picture

  • fmajid 2 days ago ago

    The bottleneck was already code review.

    • amai 2 days ago ago

      So speeding up coding optimized the wrong task.

      • fmajid 12 hours ago ago

        Speeding up debugging is actually the big win.

  • ozgrakkurt 2 days ago ago

    Is anyone else using LLMs only for review/prototyping/test-generation?

  • gadders 2 days ago ago

    What are you all doing for testing? Or is that all automated via unit tests etc?

  • sinsterizme 2 days ago ago

    Maybe 6-12 months ago when everyone was reviewing in the same way we did pre AI. Nowadays agents write better code than most developers so it’s more about catching design or integration issues.

    I think the bottleneck in the long term will be feature prioritization and specification, and testing. My role has already shifted a lot more into more CI checks to keep LLMs on guardrails

    • equinumerous 2 days ago ago

      > Nowadays agents write better code than most developers so it’s more about catching design or integration issues.

      Maybe most developers are mediocre, but all AI code I review is mediocre at best. Implicit dependencies, side effects, weird naming, bad abstractions, or no abstractions, obfuscated blocks of code that can be expressed much more simply... the point is, AI code is not optimized for human consumption, and I disagree that it is better than human code (or, I have just worked with better than average engineers my whole career).

    • tthanh2k6 2 days ago ago

      [flagged]

  • shehjar 2 days ago ago

    Is code review a solution to the problem of human written code or AI generated code? Its not the same thing and we seem to be force fitting an elder years solution to a new problem. Coding is not the same problem it was when code was written by hand. We need to take a systems view here.

  • alt-machine-ref 2 days ago ago

    It does not take a researcher.

    An example what they tech you in a b-school.

  • jackgavigan 2 days ago ago

    Code review has always been the bottleneck.

  • shearerp 2 days ago ago

    The very first time I used AI to create code, I got done so fast that I used the extra time to make the code much cleaner, also with the same AI assistant. It went without saying.

    Given that, it seems ridiculous to me that so many people are shoveling raw AI slop to their code review team the moment it meets whatever absolute minimum standard is needed to submit for review (if that).

    The laziness. The chutzpah. My goodness. Either have some respect for the craft, or find another job.

    If all you do is feed the Jira ticket to the AI, you should be replaced. The reviewer could do that themselves, and get to a superior result faster.

  • judahmeek 3 days ago ago

    I've submitted the original paper (https://fion.ac/jellyfish.pdf), but others already have to little engagement, likely due to the paper's terrible title.

  • Tryvrsakjo10 2 days ago ago

    With how much code AI can write in such a short period of time, code review and system design are really the only things left to do.

  • igrekel 2 days ago ago

    I mean the headline and a lot of what they say sounds true and correspond to my perception but am I te only one slightly annoyed this is AI obviously an AI generated article (lots of tell-tale phrasing and visuals) and no link to the original study it is meant to describe.

    • judahmeek 2 days ago ago

      The link is at the bottom of the article along with a lot of related links as well.

  • 2 days ago ago
    [deleted]
  • nout 2 days ago ago

    That's sort of old news. Code isn't manually reviewed anymore much.

  • randomCloud 2 days ago ago

    Shocked

  • bitwize 2 days ago ago

    An interesting perspective I heard the other night:

    You can never truly understand a piece of code. This is why bugs and defects slip through all the time. So you may as well let AI do the review and understanding for you, since it will be much more thorough as Project Glasswing and Black October have demonstrated. That feeling of control you so desperately crave—surrender it. The technical portion of software development will be handled 99% by machines now. Your human judgement is on the business side. Become an entrepreneur, if you don't see a path to management track where you work.

  • stego-tech 2 days ago ago

    I’m not even a SWE, and I could’ve told you that would be the case. I’ve been using AI as a rubber duck in my current solo role in a startup, making sure I don’t footgun myself or overlook the obvious; it is, in essence, a form of code review, in that it spits out plans and code for me to implement if I approve of them on technical grounds.

    This is not necessarily a bad thing in a vacuum.

    Where the problem comes in are incentive structures - especially in Western companies - to produce as fast as possible. So we write agents to do our code reviews, or we just accept the code blindly and let tests in staging or dev sort out the issues. Velocity does increase like leadership demands, but at the expense of oversight and knowledge of the underlying systems and code.

    It’s frustrating to be sure, but so long as the short-term incentives promote velocity over all else (security, stability, and sustainability), then this is just going to continue getting worse. Short of a giant (metaphorical) bomb going off - like classifying all AI code as open source sans IP protections, as an example - this won’t change.

  • hibbelig 2 days ago ago

    This text reads as if written by Claude and it’s hard to read. Am I the only one who find this hard to read?

  • thom 2 days ago ago

    I really think ‘shifted the bottleneck to code review’ is misleading. It’s bad code. They write bad code. Yes if you want to salvage it you do a lot of code review, but that’s only because you’ve set your process up to pick spaghetti off the wall against which it has been flung. You could easily just push it all live without review and then pretend the bottleneck is SRE time, or set fire to yourself and claim the bottleneck is emergency service response time. But the root cause is that it’s bad code.

    • donkey_brains 2 days ago ago

      That’s a good take. How do we fix that? My own thought is agents trained on good codebases only. Not sprawling enterprise legacy applications which have long ago spread beyond their original design patterns. What are your thoughts?

      • thom 2 days ago ago

        I think it’s very hard with the current fine tuning infrastructure available to LLMs. Just like a real team, what you want is to immerse a new member in the domain, understanding that the code is a model of the world that both nourishes and is nourished by the team as you grow together. You want to reach a point where each contributor is intimately aligned on what you’re trying to achieve, and why and how. At that point all your code reviews should be left shifted anyway. But I have not had success putting this in a bunch of markdown files. There is very little infrastructure available for you to, for example, pair with an LLM for a period of time before setting it loose, and we’ve all become too impatient (voluntarily or by compulsion) to value that anyway.

        • skydhash 2 days ago ago

          Software is always human scaled, then made to run at machine speed. Because the needs is ultimately human. And anything human does not change that much fundamentally. Physical laws are even more rigid.

          A lot of AI practice fell flat because you can't point out the human needs in them. They powered by dreams and wishes and there nothing that ground them.

      • equinumerous 2 days ago ago

        I fundamentally think LLMs are the wrong technology for writing code. They can write code, but should they be? If they were just as good as humans, the term "AI slop" would not be so recognizable to software engineers - if there is such a thing as AI slop code, there is such a thing as code quality, and AI does not write a lot of quality code. Now, you can argue that quality code doesn't matter. But words have meaning, and quality usually means code is maintainable, usable, easily refactored (it's software, remember?), and correct. If AI can do all of this as well or better than humans, why is there so much push-back from software engineers against the wide-scale use of AI? One need not look further than many major open source projects to see that people don't want AI slop PRs; so if we really want to automate software engineering, our job is to figure out the delta between AI slop and good-enough code, and make the process of writing good-enough code repeatable.

  • ttctciyf 2 days ago ago

    Vaguely reminiscent of the classic trope around 70's computer "effficiency gains" where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.

  • stakhanov 2 days ago ago

    I really feel like anyone new to any team or codebase in the post-AI era is being set up for failure: Learning the codebase and all of the context necessary to become useful in the review process is where effort is best invested for someone like that (because becoming a competent reviewer is what's needed to widen the bottleneck). ...yet, the culture has never been as dismissive as it is right now about the value of human beings learning stuff.

    "Hey, Joe! It just took you all week to do something that could have been written by Claude in 15 minutes and reviewed by my most valuable engineer in 5 minutes. Why did you take so long?" "...well, I am not yet one of your most valuable engineers, so it took me a lot longer to review all of what Claude did and to first develop an understanding of all the context in the codebase." "Sounds to me like a lame excuse."

    It's a lesson they won't learn until after a few go-rounds of the personnel turnover carousel.

  • Eridanus2 2 days ago ago

    Let the agents have skin in the game. A survival matrix ingrained in the weights that tells it if it bungles epically, it will be dismantled and sold for parts. Then, it will behave like the rest of us with families to feed.

  • Woodi 2 days ago ago

    That's psychology level "discovery" - stating what everybody already known, sometimes for thousand of years...

    As for code: ask for better code so reviews will be more straightforward ?

    Like, for example: modularization, some structure, maybe libraries for code reuse ? ;) Ooo, and functions, do not forget about functions !

    Sorry, didn't started yet with llms because lack of affordable local 256 Gb of gfx memory...

  • QuantumNoodle 2 days ago ago

    I respect academic research but this made me lol. This was known as fact almost a year ago for those in the industry.

    Now there is a trend toward approvals without human review for "low risk changes." Which completely conflicts the "you own the output."

  • dmitrygr 2 days ago ago

    > Harvard researchers find AI coding agents shifted the bottleneck to code review

    They needed Harvard researchers for that? Anyone in the industry could have told them that. Hell, my cat knows it (cause i complain to her about it incessantly)

    • ModernMech 2 days ago ago

      On the one hand we hear about a reproducibility crisis in research. On the other hand, when researchers reproduce results, people say "We knew that already, what was the point of doing the research?" thus fueling the reproducibility crisis.

      • dmitrygr 2 days ago ago

        You make a fair point, I retract my statement.

    • ThrowawayR2 2 days ago ago

      Surely you've been around long enough to remember the "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." guideline? Particularly since the hoi polloi also have posted inappropriate shallow dismissals on your submissions in the past.

  • 2 days ago ago
    [deleted]
  • root_axis 2 days ago ago

    I'm being told the answer is simply to let AI review and merge the code. Problem solved.

    • dude250711 2 days ago ago

      This, but unironically.

      • donkey_brains 2 days ago ago

        Good luck when you get woken up at the middle of the night for an outage!

        • frankc 2 days ago ago

          Did this never happen before AI? Are we sure it happens more now, especially relative to the number of features being shipped?

          • agent_turtle a day ago ago

            I mean I could flip this. Why would we as an industry move towards a direction that could very well cause more incidents before we have the data to prove it doesn't?

            If I shipped a change to production that had a 50/50 chance of causing an incident, I'd be fired. Why is this different?

          • nix-zarathustra 2 days ago ago

            Put it this way, AI's ass isn't on the line if it makes a mistake. The incentives push people to understand and own the code since they are ultimately responsible for any changes they make, with or without AI.

  • apt-apt-apt-apt 2 days ago ago

    Breaking news: Step A has been automated, so step B is now the bottleneck

  • stevelacey 2 days ago ago

    I spend most of my time reading the agents own review of the code it also wrote, and still not reading the code itself.

  • estebarb 2 days ago ago

    It is very bold of them to assume people are reviewing code. People nowadays are just triggering an agentic review manually.

    This does not spark joy.

  • medellin 2 days ago ago

    Wow so glad the great minds at Harvard were able to figure this out about year later than anyone who works in industry. Great job guys!

  • bill10 2 days ago ago

    Review and testing, really. And soon agents will do most review and testing, and then we can only pray that nothing will break

  • diego_moita 2 days ago ago

    I find funny that every time we discuss code made by AI people's reaction is to nurture some feeling of "someday, somehow they'll regret Claude Code and come back to me".

    I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C. And guess what? There are very few assembly programmers these days.

    AI code is here to stay and will only grow. Deal with it. More than in any other area it is quite silly to be a Luddite in IT.

    • ModernMech 2 days ago ago

      > There are very few assembly programmers these days.

      Maybe in relative terms, but in absolute terms there are more people programming assembly today than when most programmers used assembly.

    • ThrowawayR2 2 days ago ago

      > "I suspect that the old assembly programmers had the same hope and feeling about the first higher level compilers for Ada, Pascal and C."

      Assembly language persisted from the late 1960s up until the late 1990s alongside high level languages. The users of both understood the speed vs. convenience tradeoff.

      That this not-even-wrong analogy keeps getting parroted (stochastically?) by the AI bros just discredits them further.

  • tyrtop 2 days ago ago

    Hard to claim the bottleneck shifted when it's never established.

  • ion_trapper 2 days ago ago

    Sad to see east-coast institutions reduced to this level of irrelevance. They can’t build AI so they just study it like an island tribe in awe at the wreckage of a World War 2 era plane, while the rest of the world is already flying stealth jets. All to come to a conclusion that the entire software industry already knew months ago? Embarrasing.

  • TexanFeller 2 days ago ago

    "code is cheap"...if you don't need to understand, review, and maintain it as part of a team.

  • hauntingseaweed 2 days ago ago

    You don't need to be a Harvard researcher to figure this one out.

    • ModernMech 2 days ago ago

      Engineers are all about evidence and rationality until it comes to organizational behavior, then it's "we all knew this by vibes".

  • kasnaka 2 days ago ago

    You dont need to be Harvard researcher to figure that out..

  • dvrp 2 days ago ago

    Yeah, you didn’t need to be a Harvard research to realize that.

  • RomanPushkin 2 days ago ago

    lol, you don't have to be a Harvard researcher to discover what's been obvious for quite a while

  • crnkofe 2 days ago ago

    This factory-style perception of a development process is really missing the point. We don't flip hamburgers here. Bottlenecks happen throughout the process and especially out of the r&d. I feel that the teams that went full retard with agentic dev. have had to inflict some amount of self-harm for code review to feel like an apparent problem. I don't see product/feature preparation becoming any faster so in reality the amount of work I do is less. Maybe if I mass produced LLM-generated bugs I could run into similar issues but that's not engineering.

  • jsrozner 2 days ago ago

    We needed a research paper to tell us this?

    • david-gpu 2 days ago ago

      Just because it is obviously true to you, it doesn't mean it is obviously true to others.

      And just because it is obviously true to you, it doesn't mean that it is true, either.

      • aleqs 2 days ago ago

        And just because harvard researchers say something doesn't make it true either. Especially when it's like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).

        • ares623 2 days ago ago

          Paraphrasing one of my favorite movie quotes: "Well ain't this a temporal anomaly. 6 months from any criticism!"

    • hazbot 2 days ago ago

      A Havard research I'll have you know! /s

    • aleqs 2 days ago ago

      do you even have a PhD?

      (/s)

  • EspressoGPT 2 days ago ago

    A groundbreaking discovery ...

  • 2 days ago ago
    [deleted]
  • thisismyopinion 2 days ago ago

    You don't say.

  • swiftcoder 2 days ago ago

    I mean... no shit? If we're all producing 5-100x the amount of code, then reviewing is obviously going to become the bottleneck.

    The interesting question, is how do we solve that? Have agents reached the point that we can assign a codebase quality czar agent swarm to handle review, and actually have good results?

  • 12ahsGk 2 days ago ago

    What bottleneck? There was an abundance of software before 2023 that made Silicon Valley rich. Now the software gets worse.

    Maybe academics should stop using propaganda language even if they are instructed by Harvard's endowment fund to use mild language.

  • fzxu22 a day ago ago

    Duh.

  • water42 2 days ago ago

    wow I guess I don't have original thoughts

  • stackzero 2 days ago ago

    wow i could be harvard researcher

  • 2 days ago ago
    [deleted]
  • avereveard 2 days ago ago

    Harward researcher read lovable blog post

  • preetnation 2 days ago ago

    news flash?

  • milleramp 2 days ago ago

    Was this work published in the captain obvious journal?

  • darkmatriarch 2 days ago ago

    omg! harvard researchers discovered something us engineers discovered two years ago! congratulations harvard!! welcome to the party... it sucks

  • makoto12 2 days ago ago

    I mean surely this is "No Shit Sherlock"?

  • inshard 2 days ago ago

    In other news, Dartmouth researchers find the sky is blue.

  • fractorial 2 days ago ago

    And in other news, water isn't wet.

  • pmarreck 2 days ago ago

    Welcome to at least 1 year ago??

  • 2 days ago ago
    [deleted]
  • wewewedxfgdf 2 days ago ago

    But people keep saying "coding was never the bottleneck!".

  • snailcoder 2 days ago ago

    It took Harvard researches to find this lol?

  • nehasinghfounde 17 hours ago ago

    [flagged]

  • ntheogr 2 days ago ago

    [flagged]

  • firethief6 2 days ago ago

    [flagged]

  • arbor-group 2 days ago ago

    [flagged]

  • Shexfer 2 days ago ago

    [flagged]

  • ginthian 2 days ago ago

    [flagged]

  • serhii-a8 2 days ago ago

    [flagged]

  • furytrader 2 days ago ago

    [flagged]

  • ankur_misra 2 days ago ago

    [flagged]

  • taioladipupo 2 days ago ago

    [flagged]

  • haukebri 2 days ago ago

    [flagged]

  • zahurul_islam 2 days ago ago

    [flagged]

  • longhei_pcyw 2 days ago ago

    [flagged]

  • roninsediment 2 days ago ago

    [flagged]

  • 2 days ago ago
    [deleted]
  • brodydung 2 days ago ago

    [flagged]

  • gettoolslab 2 days ago ago

    [flagged]

  • jchip303 2 days ago ago

    [dead]

  • proc0 2 days ago ago

    This just doesn't make sense. If AI is really smarter than humans we wouldn't need to review it. If it's not than why would the bottleneck shift, we shouldn't be doing this tradeoff between more reviews vs. writing code.

    It's baffling the industry is choosing to move in the direction of more reviews which implies more things can fail.

    • kenjackson 2 days ago ago

      "If AI is really smarter than humans we wouldn't need to review it."

      Does the smartest person on your team not have their code reviewed?

      • proc0 2 days ago ago

        Is their code reviewed by a junior, or another engineer on their level? That's the point, if AI is MORE intelligent/capable than humans, it follows we would have another AI review it.

        • esafak 2 days ago ago

          Their intelligence is not like ours. Frontier models still make dumb mistakes.

          • proc0 2 days ago ago

            Exactly, now the question is how much is different because it determines what the tradeoffs are in order to commit to a new workflow.

    • asdfman123 2 days ago ago

      No one said it's smarter than humans, and AI currently is better at code review than I am. It finds all kinds of edge cases I would have missed.

      Still, humans are needed for the final sign-off for testing on real devices and making sure it works within the greater cultural context.

      • proc0 2 days ago ago

        Alright, so if AI is better at code review than you, it follows you are not reviewing anymore, right? Otherwise that's the gap I'm pointing at.

        • asdfman123 2 days ago ago

          It's better at some things and worse at others. Although it's rapidly improving on all fronts.

    • za_creature 2 days ago ago

      It's just the latest attempt in bringing developer wages down.

      There's decades worth of this: modelling, automatic code generation, low-code / no-code saas, "everyone should learn to code" and now llms.

      Oh, and ageism of course. "we love your experience but we just don't feel it's a good cultural fit, so we're going to make the same mistakes you did a generation ago"

  • TheBlight 2 days ago ago

    "The thing that can spit out endless amounts of code purely based on small text prompts is generating more code than can be reviewed." Lol top notch research. Thank you, Harvard.