Why is Gemini 4 Carbon taking so long?

(kernel.dev)

282 points | by deenadz 2 days ago ago

153 comments

  • tristanj 2 days ago ago

    It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

    1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

    2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

    3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

    • kyrra 2 days ago ago

      1) "Gemini 4 Hydrogen is adapted from Gemini 4 Carbon." From https://deepmind.google/models/model-cards/gemini-4-hydrogen... . Also, the Agy team is pushing is that Carbon is a great planner, and then use hydrogen as the worker. They want people to use it.

      2) lots of people inside Google use carbon. If they didn't care about it, it wouldn't have been pushed on all of Google.

      3) Deepmind is competing with cloud and others. they are the only Frontier Lab that actually has to compete with other business units within an org for compute. GCP and others want that compute as well.

      (Googler, opinions are my own)

      • zamadatix 2 days ago ago

        1. Small, efficient models are often derived from "the best" models. That doesn't mean "the best models" are efficient too.

        2. GP said they don't care about cost, not they don't care about the model

        3. Competing for existing compute + being able to assign the projects own capex investment for new compute provides more compute than needing to assign all compute via new capex investment. But it's hard to say if that's actually the type of case or not unless Google wants to tell everyone publicly - so I'd take that point with a grain of salt in either way.

      • the13 2 days ago ago

        Thank you. Does Carbon take longer for you than comparable Anthropic and OAI models?

    • aberoham 2 days ago ago

      Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.

      It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.

      One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.

      Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.

      • p_l 2 days ago ago

        This has been acknowledged by recent "agy" release notes, it was timing out in client not in backend

        • elij 2 days ago ago

          Yup I use carbon hours daily with 10+ minutes between requests at the extreme without timing out. That timeout has to be the harness. I also haven't seen API errors since release day.

      • yard2010 2 days ago ago

        I never got any google model to work using their infra - 429, billing errors and timeouts. As someone wrote here a few months ago - it's an exercise in managing frustration. So I had to resort to openrouter.

      • embedding-shape 2 days ago ago

        Lots of Google tooling is the same for me, simply unusable whenever I try even simple stuff.

        I think no one tests this as a "non-Google employee" or something, they all have specific testing backends or something, and once deployed to the public, there is no one responsible for the experience of non-Googlers and no one testing it, that's why most of the Google stuff has such poor UX because of hundreds of small cuts all over the experience.

        • antihipocrat 2 days ago ago

          Google never had the mandate to align design standards across internal and external tooling. BigQuery and some of the other analytics tools are great though, I'd be surprised if staff didn't use the same internally - maybe someone here can confirm?

          • kccqzy 2 days ago ago

            Staff used Dremel and then F1. At least when I was there. I only heard non-Googlers praise BigQuery.

        • kyrra 2 days ago ago

          What specific things are like this? If you have a complaint and it's something I can reproduce, I can attempt to file a bug for it.

          • htrp 2 days ago ago

            don't bother... you'll just end up getting frustrated when you realize no one in the org cares.

      • kingleopold 2 days ago ago

        this happened even before agy btw, their CLI was subpar level for months even before 3.1 pro release and now too

      • thanh81646 2 days ago ago

        [flagged]

    • npn 2 days ago ago

      4. they are so behind that they can not create effective datasets like other frontier labs do. So instead they do all the tricks the Chinese labs do: using RL with the single purpose of benchmaxxing the AA score.

      Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks

      • Doohickey-d 2 days ago ago

        Which is weird, because Google is in the best position for getting training data: they have YouTube (which I guess is why Google's models are good at multimedia), as well as the Google crawler. Which effectively websites cannot opt out of if they want to be Google searchable. So Google's bot probably is the _only_ bot (scraper) that hasn't been blocked from a large portion of the internet.

        • htrp 2 days ago ago

          Google legal refuses to allow them to do so.

        • efromvt 2 days ago ago

          I think raw base text training data may be decreasingly important vs good RL environments/gyms, and maybe less clear that Google wins by default there?

        • giancarlostoro 2 days ago ago

          Not only that, they pioneered some of the key technologies... Who isn't still on transformer models? Also funny when you consider Meta is in a weird similar boat, introduced key things, so far behind the frontier.

          • aswegs8 2 days ago ago

            The innovator's dilemma. Large companies are structurally worse at innovation.

            • adventured 2 days ago ago

              To go along with that they're extremely slow (massive orgs) and don't feel their existence is at stake.

              If GPT slips meaningfully behind, OpenAI is dead. The same goes for Anthropic. Both would die extremely quickly.

              If you're within Google that is not the sense you have about the current context, nor within Meta or ByteDance or Amazon or Microsoft. They certainly view all of this as very important, they don't think they'll die within a year or two if they don't stay on top of the frontier. Meta has run from behind endlessly, as has Google, and their businesses are doing just fine (for now). If OpenAI were in that position, there would be no reason for them to exist and they would be dead by now.

              Microsoft likely doubts their existence is at risk just because they can't field a frontier model. For now they don't see Office / Windows being killed off as a short-term risk (plus they have Azure and other products).

              AWS is extraordinarily difficult to replicate, it's more akin to trying to compete with dozens of steel mills (industrial infrastructure) than a mere piece of software. And the rest of Amazon's cash spigot is advertising on its retail platform, which is facing close to zero threat from the frontier race, so Amazon can take its time and endlessly screw-up on AI (and so it has).

              A frontier LLM can't manufacture a smartphone or laptop for you, and certainly can't manufacture 100 million of them with a zillion dollar manufacturing contract with TSMC et al., Apple isn't particularly worried. They want intelligence on their platform, they're not overly concerned about needing to own a frontier model. They can afford to be patient and get it right.

              Capital is more powerful than AI. The capital owns the GPUs the AI requires, the capital owns the datacenters, the capital owns the grid, the capital owns the generation, the capital owns the factories, the capital owns the machines that produce semiconductors, the capital owns the patents. Apple, Microsoft, Google, Amazon, Nvidia, Meta, etc control the capital, financially they're far more powerful than most mega banks these days. And if Apple needed $100 billion for a project, it's one human phone call to Berkshire Hathaway. For now AI is feeble standing next to capital.

              AI is the little bitch of capital, to put it bluntly. AI is on its knees begging capital for more compute: please sir, may I expand?

          • pfannkuchen 2 days ago ago

            Well “they” didn’t, people who work for them did. Are any of those people still at Google?

      • owkman 2 days ago ago

        Seeing how every new benchmark that has come out has Carbon doing well, I'd have expected the benchmaxxing allegations to die out. But they've been ok incredibly persistent. It's jagged like all frontier models. You could argue about it's warts, but benchmaxxing is unfounded. Did it benchmaxx every public and private dataset? Is that what RL is perhaps?

      • jckrichabdkejdb 2 days ago ago

        [dead]

      • unbrice 2 days ago ago

        [Citation Needed]

    • fsh 2 days ago ago

      4. Google did a worse job than Anthropic and OpenAI.

      • dpe82 2 days ago ago

        Well sure, but parent is trying to reason about why.

        • andy_ppp 2 days ago ago

          It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.

          • exe34 2 days ago ago

            Why would data and training environment lead to frontier intelligence but slow?

            • andy_ppp 2 days ago ago

              Because the environment did not reward being as efficient as other models?

              • exe34 2 days ago ago

                Oh so it's using more tokens to do the same work. Make sense, thanks!

          • LoganDark 2 days ago ago

            Why would a search engine that's had nearly the entire internet for decades not have the data of a lab that's only been scraping for a few years?

            • PaulHoule 2 days ago ago

              The web is mostly trash. Go look at commoncrawl!

            • andy_ppp 2 days ago ago

              Because Google are probably more hamstrung by bureaucracy and little things like the law.

              • LoganDark 2 days ago ago

                Since when? Google recently disobeyed some legal requirements because "haha, couldn't catch me / too late now bitch". (This is when they cut down a forest for a datacenter)

                • tibbar 2 days ago ago

                  Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.

                  Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.

                  • andy_ppp 2 days ago ago

                    Yes I'm very much talking about data used in the RL section, I'm sure Google are indeed very good at pre-training.

                  • htrp 2 days ago ago

                    Too bad pretraining hasn't been relevant for model performance since 2025

    • zarzavat 2 days ago ago

      It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.

      • kyrra 2 days ago ago

        Googlers had been testing carbon for some time prior to the announcement.

    • Danywalery 2 days ago ago

      Google has a huge flow of user data; they were the first to figure out about the Transformer, yet they’re lagging behind in the race. This is very strange, although their flash models are quite good, they’re expensive compared to the Chinese ones.

    • teknologist 2 days ago ago

      Also, Google uses their own TPU architecture rather than GPU/Nvidia.

    • vintermann 2 days ago ago

      So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.

    • serioussecurity 2 days ago ago

      (2) isn't right. Cloud is on track to become the largest source of revenue and a lot of that comes from selling inference.

    • moffkalast 2 days ago ago

      Maybe they should be more efficient and sell fewer ads.

    • cyanydeez 2 days ago ago

      Wouldn't it make more sense they simply don't dedicate compute resources to each user session?

  • horacemorace 2 days ago ago

    I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.

    • artdigital 2 days ago ago

      My thinking as well! I really like Carbon

      It’s hella slow for sure and agy harness makes this worse by telling it to read a ton of files before making any changes!! But once it starts, its solutions are very good and it usually continues until the task is fully completed! The output was so good at times, I was honestly surprised by how thought-through and properly implemented everything was. It implemented edge cases I would have missed and made decisions that I would call sensible. For example adding a —-flag that absolutely improved the product.

      Code review still catches medium priority issues but that’s the same with all of the models.

      I’m now putting Carbon on bigger tasks that I don’t care how long they take. As I am writing this I’m having dinner while Carbon slowly chops away at a task queue on my server.

      All in all, Carbon is great, Hydrogen is great, Google is killing it!

      (agy is still bad though, but you can tweak the “read a million files first” behavior with your system prompt)

      • joshuanapoli 2 days ago ago

        My experience is similar: Carbon found better ways to solve open-ended problems than Fable. In these cases, where I don’t have a solid design ahead of time, it’s OK to wait.

    • dhruvmittal 2 days ago ago

      Definitely agree with this, I regularly find that carbon writes less overengineered code and considers more of my existing architecture without additional documentation or prompting. Maybe that's what it's doing with all of its "wasted" turns?

      That said, it's incredibly slow. I haven't played much with the orchestrator that agy added but if it's Carbon occasionally delegating to Hydrogen, that might be a little backwards. I feel like what I actually want is Hydrogen delegating more difficult tasks to Carbon instead.

    • weatherlite 2 days ago ago

      So it's slow but good? Reasonable trade off I think

      • antupis 2 days ago ago

        it depends but IMO Carbon is best at the moment with code reviewing and documentation writing. Opus is still my go-to model for code generation but at moment Carbon does better job for those auxiliary jobs.

        • eknkc 2 days ago ago

          Why not Fable? I've mostly been using GPT side models and have recently been dipping my toes into Claude models. I went big with Fable 5.2 as it was "the best" the offered. Apart from token usage, is there any particular reason to use Opus?

          • antupis 2 days ago ago

            Our organization still blocks Fable because reasons probably something related to storing prompts.

      • glimshe 2 days ago ago

        It depends on how much you're willing to wait. I used Argon a few times and while it did some interesting things, it was so slow that I couldn't fit it in my workflow. GPT Sol was great and 5-10x faster.

        I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.

      • bagacrap 2 days ago ago

        In case folks missed it, Google essentially recommends using carbon and hydrogen together: https://antigravity.google/blog/gemini-4-hydrogen-in-google-...

        • thisisauserid 2 days ago ago

          Using them together is faster and cheaper and not quite as good as just Carbon (if I read that right). So that's a pretty good sweet spot for most use cases.

          Gemini gives you a lot of options.

      • eknkc 2 days ago ago

        [dead]

    • nautilus12 2 days ago ago

      I agree, the best way i can describe Carbon is that it is "cautious". Very very refreshing attitude for a model to have.

  • hetspookjee 2 days ago ago

    Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers. I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models

    • wronglebowski 2 days ago ago

      I do like Argon, but usage continues to be an issue for me. The economics of everything pales in comparison to the value you get with DeepSeek. I'll end up using my limited opaque usage based limit with Antigravity to have Argon review something or maybe write a plan, but the bulk of the work has to be done with cheaper models.

      • hetspookjee 2 days ago ago

        I use it similarly but have the subscriptions to do other worker models. But yea I tell it only to delegate except when the change is something tiny. But most of it is done n via delegation.

      • Apocryphon 2 days ago ago

        Any opinions on DeepSeek vs. GLM and the other open-weights?

      • soared 2 days ago ago

        Similarly $20 on opus in Claude code seemed to last me 10x longer than argon in antigravity. And for some reason there doesn’t seem to be an opus equivalent in antigravity - but in Gemini I can select “pro” which doesn’t exist in antigravity (flash or argon only). Antigravity $20 plan also gives a comical amount of Anthropic credits… it can literally only answer one question for me before hitting limits.

    • eitally 2 days ago ago

      I've also become a fan. For most of my current projects I'm using Agy with the Carbon orchestrator, but switch to Hydrogen when I'm bantering back & forth. It's convenient, with things like Swarm, to have an orchestrator that can automate the selection of models for sub-agents. I've used quite a few harnesses -- both for local and cloud dev -- and although Claude Code has some clear feature benefits over Antigravity (for desktop/sandbox control & interaction), I'm generally very happy.

    • chubot 2 days ago ago

      What harnesses do you use, to switch back and forth between models, and to orchestrate?

      • hetspookjee 2 days ago ago

        I tell t to send it to Claude -p opus 5.5 and or Astra. I made the model mappings explicit in GEMINI.md. I don’t ever leave agy anymore and anytime I need to I dislike how Astra and Claude always beat around the bush

      • WarmWash 2 days ago ago

        I haven't used Carbon or Argon outside antigravity, but historically Gemini models take a bad hit outside of it.

    • htrp 2 days ago ago

      So basically the Argon is your scheduler and then the other frontier models actually accomplish the work?

    • xnx 2 days ago ago

      There's dozens of us!

    • moffkalast 2 days ago ago

      That's right, you are probably the only one :)

  • fwee 2 days ago ago

    Carbon is slow, but I've also realized that it works really well. I don't need to get involved or keep checking the results over and over again. I think it's pretty close to Opus 5.5 in terms of work quality.

    I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).

    The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)

    Anyway, great article!

  • bagacrap 2 days ago ago

    This part actually seems like a feature:

    > Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.

    I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.

  • timtam 2 days ago ago

    Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.

    I only got around the issue by strictly setting an effort budget or a stopping rule.

  • jedbrooke 2 days ago ago

    I noticed a similar issue going from Opus 4.6 to 5.5. I also however upgraded my harness during that time, and I found out that it was injecting something like “Be helpful, read in between the lines and proactively go beyond the user’s request” at each turn. I immediately patched that out and saw decent improvement in that area. TFA mentions they tried different harnesses though, so yeah it’s prob baked in to whatever RL setup they’re using to train the original model.

    This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good

    • ghm2199 2 days ago ago

      This. Yeah I think if one knows what they are doing, e.g. a seasoned developer. They can get a lot of mileage out of an LLM that is good at instruction following. which is why i like codex models. Astra may not be as good as Opus but the workhorse really is 6.2 sol. The best workhorse when it comes to the pareto frontier of `cost vs accuracy` or `inference time vs accuracy`.

  • Otterly99 2 days ago ago

    > Closing loopholes the ticket had already ruled out. Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

    This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.

    It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.

  • chriddyp 2 days ago ago

    I've tested 75 models so far, and Gemini 4 Carbon is 68th/75 in terms of slowness [1]

    Looking into it, there are 4 areas that contribute to speed:

    1. Time-to-first-token is slower than others: 5s (median) to start on a one-line prompt and 11s on a ~130K-token cached prompt, against 2-2.5s for Gemini 4 Hydrogen and 1-5s for GPT, Claude, Grok, Kimi and Qwen. Long tail though: one call took 50 s and one timed out.

    2. Tokens/s is slower, but not that much: 3-9s for Carbon vs ~5s for Hydrogen.

    3. It’s not wordy: thinking included, Carbon wrote ~500 output tokens per request vs ~780 for Hydrogen, and less over the whole exam (220K vs 259K). Slow-but-wordy models like Claude Fable 5.2 and Kimi K3.1 Pro write 1,600 to 2,200 per request.

    4. More thorough, more tool calls: 10.3 model calls per question vs 7.7 for Hydrogen. On data-quality audits it made 17 vs 9, and on our "find the hidden pattern" questions it ran 41 queries on average vs a board median of ~25.

    The 4th piece is pretty interesting as it’s more about the behavior than the infra. If the task is “find all of the issues”, and the model is very exhaustive in its search, then “time per task” calls it slow but in practice we’d call it thorough.

    [1] https://plotly.com/blog/gemini-4-carbon-plotly-data-analytic...

  • Lerc 2 days ago ago

    A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.

    It really looks like something is broken more than just being inefficient.

    • xnx 2 days ago ago

      How recent is that video? Google just reduced time to first token (https://x.com/_mohansolo/status/2122821676249625014): "Time to first token (TTFT) for Carbon is now 1.5x faster on Antigravity. Will follow soon with Argon too.

      Lots of work went into this so you all could have a faster experience. More to come "

    • 96khz 2 days ago ago

      are we talking about token prefill here?

      • Lerc 2 days ago ago

        11 seconds worth?

    • teknologist 2 days ago ago

      Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals

      • LiamPowell 2 days ago ago

        Every time I see a Theo tweet he's judging something on the most absurd metrics. Today for example, he was saying Astra 6.2 is bad because when he gave it a single line prompt telling it to create a new level for a game it used a name for the level that he'd seen before. Judging a model on how often it reuses a name is one of the most ridiculous benchmarks I've ever seen.

        As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.

      • 2 days ago ago
        [deleted]
      • ywvcbk 2 days ago ago

        They have excluded all OpenAI models, though? Of course I doubt they'd have the highest scores but likely would be most price efficient.

      • spprashant 2 days ago ago

        I do not like Theo, but he is transparent about his OpenAI bias. He criticizes Anthropic frequently - mostly for their hostile closed-garden policies.

      • knollimar 2 days ago ago

        I’m not sure Claude favoritism is fair; he loved OpenAI for a while. He just has really arbitrary and frankly bad takes imo

        • teknologist a day ago ago

          Depends on which is paying at the time I would think. YouTube influencers tend to have price lists.

  • p_l 14 hours ago ago

    Interestingly, agy 1.5.13 (but seen that with 1.5.12 already) now works hard to coalesce multiple steps in tool calls, just like Opus does - i.e. building small bash one liners with separators

  • totalhack 2 days ago ago

    Carbon has been slow, but excellent at deep dive analysis of some of my projects, finding optimizations other models missed AND explaining them in a clear and concise manner, even as the conversation develops and really becomes four or five subthreads on the opportunities that it brought up. Maybe that massive context window helps.

  • logicchains 2 days ago ago

    Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.

  • bugzbenny 2 days ago ago

    Using any of the latest models within Antigravity is incredibly slow. I use Claude Desktop for sending off coding agents into the cloud and I get about 10x more done with it than I can do with Antigravity + Google models. The only advantage to the Google models over Claude so far that I've noticed is they are way better at writing copy.

    Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.

    I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.

  • lmeyerov 2 days ago ago

    It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models

    My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy

    While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models

    My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem

  • 2 days ago ago
    [deleted]
  • SubiculumCode 2 days ago ago

    That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.

    • ahsillyme 2 days ago ago

      Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)

      • horacemorace 2 days ago ago

        Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up. Junior SWEs usually try to do too much.

        • bot403 2 days ago ago

          This. There should be 2 PRs. The core fix, the the unrelated fixes. Because I guarantee sometimes those unrelated fixes will cause regressions so they have to be scrutinized too.

      • shwetanshu21 2 days ago ago

        Sometimes that problem might have downstream other effects which causes bigger problems which is why it is left as it is.

        Most of the times it has to do with refactoring downstream services or dependencies or just unexpected behaviour that shows up.

  • seizethecheese 2 days ago ago

    We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.

    Biggest differences:

    - Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.

    - Output tokens 16.4k for Carbon vs 6.8k for opponents.

    - Tool calls: 21 for Carbon vs 11 for opponents.

    All three of these are statistically significant.

    The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).

    Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.

    - Both finished 83 carbon wins, 52 losses.

    - Early votes 2 carbon wins, 40 losses!

    Basically, Carbon is good when you're willing to wait.

    [1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel

  • lp92 2 days ago ago

    >Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

    That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.

    With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.

    • gazebo2 2 days ago ago

      I find this behavior to be common across most of the big models -- Sol 6.1 does it a ton, adding lots of super defensive programming cruft for things that A) by design can never happen and B) should fail loud anyway, despite prompting. It gets a little better after steering tweaks but still common.

  • 2 days ago ago
    [deleted]
  • mengqijin 2 days ago ago

    I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.

    But with simple tasks, Carbon is way too slow, and much higher cost.

  • niltecedu 2 days ago ago

    I am pretty sure caching is busted, atleast on github copilot, I tried doing it via pi harness however cannot seem to get any cached inputs post 5 minutes.

  • 2 days ago ago
    [deleted]
  • 5555watch 2 days ago ago

    Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.

    In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.

    So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!

    Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.

    Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.

    • joshuanapoli 2 days ago ago

      Yeah, Carbon is good when the problem needs some imagination, when a better solution is worth waiting for. Fable 5.2 moved towards faster or more efficient responses. So it’s more practical for most tasks, but tends to be less creative.

  • oytis 2 days ago ago

    They claimed a precious domain I should say, shame that they are not using it for anything kernel-related

  • jrflo 2 days ago ago

    Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?

    • murderfs 2 days ago ago

      I'd guess that it's because Google eats its own dogfood, and if they didn't train the models to be overly meticulous, the existing codebase would implode under a tsunami of slop.

  • dada216 2 days ago ago

    ITT it's that huge max output token of 1mln tokens.

    It's pretty far beyond other models (Opus 5.5 is at 128k).

  • ahmadyan 2 days ago ago

    i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.

  • cybice 2 days ago ago

    Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls

  • timcobb 2 days ago ago

    really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".

  • user_7832 2 days ago ago

    I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

    I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

    Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).

    • vintermann 2 days ago ago

      I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.

      Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.

      Still, those are much better failure modes than hallucinating, which powerful models are still prone to.

    • incognition 2 days ago ago

      Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme

      • p_l 2 days ago ago

        Heard a rumour that Carbon is ~10x the size of Opus 5.5

        • netrap 2 days ago ago

          I liked flash as well but it is gone now?

          • p_l 2 days ago ago

            Still have 3.8 Flash available here

    • SyneRyder 2 days ago ago

      Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).

      • knollimar 2 days ago ago

        Opus 5.5 trashes everything not fable and astra for my knowledge work. It’s an insufferable pedant though.

        I haven’t used a lot of Carbon though.

    • wellcolormesurp 2 days ago ago

      They are cracking software and math, not finance, law or medicine or whatever it is you do.

      Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.

      • oblio 2 days ago ago

        > The other dominos will fall automatically once you got the engineering automated.

        Based on...? Wishful thinking?

    • auyez 2 days ago ago

      At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.

      • londons_explore 2 days ago ago

        The Human mind so far has shown remarkably little ability to redesign other human minds to be smarter.

        The existence of AI able to design a smarter AI I would not say is a given.

  • nandanadileep29 2 days ago ago

    yeah i tried using it its not efficient at all. just burns up the token and not get the work done.

  • resters 2 days ago ago

    i wanted to test carbon but ran out of quota before i could get a definitive result.

  • WeZZard 2 days ago ago

    I didn't try but some people said Gemini 4 Carbon served by VertexAI is faster.

    • 2 days ago ago
      [deleted]
  • kristjansson 2 days ago ago

    Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?

    • p_l 2 days ago ago

      [dead]

  • jaren 2 days ago ago

    any hypothesis why slower inference?

  • gdevenyi 2 days ago ago

    I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.

    It might do good work but I'm having a hard time figuring out if it is.

  • 0xbadcafebee 2 days ago ago

    Gemini is consistently the worst performing AI product, agent, model, etc I have used. I don't really understand who keeps using it.

    • dannyw 2 days ago ago

      Companies with a GCP commit.

  • dario_amodei 2 days ago ago

    how much?

  • coldbreeze485 2 days ago ago

    [dead]

  • sodowork 2 days ago ago

    [flagged]

  • georgestones 2 days ago ago

    [dead]

  • ath3nd 2 days ago ago

    [dead]

  • NoahZuniga 2 days ago ago

    Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.

    • lukax 2 days ago ago

      You probably meant a version of Argon?

    • kzrdude 2 days ago ago

      Then that's "test-time scaling" again and Google then showed that they can reach frontier level benchmark results using it. It's pretty cool as a technology actually (the demonstrated capability), even if the model is not so useful when it's slow.

    • joedwin 2 days ago ago

      Argon right?

  • erji77 2 days ago ago

    You idiot, stop pushing AI news to me from now on.