GPT-6.2 Astra scores 100% on ARC-AGI-3

(twitter.com)

37 points | by rzk 15 hours ago ago

15 comments

  • lcprk 12 hours ago ago

    It's fascinating to see these benchmarks being saturated where, a priori, solving it seems to require human-level general intelligence, yet when you try the models they are still clearly not broadly human-level. I am very curious to see if a particularly clever set of benchmarks is all you need to elicit genuine human-level intelligence.

    • Aperocky 12 hours ago ago

      Maybe the problem is the concept of "benchmark" itself.

      Maybe we do better at things that cannot be a benchmark, i.e. uncertainty and randomness.

      You cannot put things that cannot be graded into a benchmark.

      • imimayj1337 12 hours ago ago

        I think also there's an element of these benchmarks being static (and therefore gameable) which is holding us back? I know we do multi turn assessment yadda yadda but I wonder if we had an adaptive framework of sorts which could actually adapt to model behaviour whilst dynamically leveraging some kind of internal scoring criteria knowledge graph it might at least address the issue slightly? Reality is just not matching these benchmarks (or any of those which I've built privately for assessing models even!)

      • interstice 12 hours ago ago

        And yet we make children take exams to prove their value

  • PoignardAzur 13 hours ago ago

    Looking forward to see what they come up with for ARC-AGI-4.

  • datadrivenangel 12 hours ago ago

    100% 'with adaptor' and 70.3% in standard mode...

    • rzk 5 hours ago ago

      [dead]

  • coffeebeqn 15 hours ago ago

    So is this AGI?