Gemini 4 Neon is the new #1 on the Models Engineering Index

(git-aarvin.github.io)

27 points | by hn-aarvin an hour ago ago

12 comments

  • YuechenLi 9 minutes ago ago

    Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.

    But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.

  • quantisan 20 minutes ago ago

    When and why should I trust this index over others?

    • ummonk 17 minutes ago ago

      I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.

  • js4ever 16 minutes ago ago

    Is it just me or all Google models are terribly bad? They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)

    • hadlock 9 minutes ago ago

      After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.

    • SwellJoe 12 minutes ago ago

      I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.

    • Yahivin 6 minutes ago ago

      With Neon Gemini is good now.

    • mjburgess 13 minutes ago ago

      And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.

      Very high hallucination rates too.

    • copperx 13 minutes ago ago

      Are you saying Neon is just as bad?

    • azan_ 11 minutes ago ago

      Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.

      • andunieee a minute ago ago

        You mean they are not providing you with their best models for free?

  • andunieee 2 minutes ago ago

    How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?