The Two MMLU Scores: What a Benchmark Name Does Not Fix

(zatona.dev)

8 points | by gmays 3 days ago ago

3 comments

  • bbstats 15 hours ago ago

    "What a ____ does not ____" is painfully Claude-Coded (no pun intended)

    • entrope 15 hours ago ago

      So is "does [not] fix [attribute]". The opening dot list has several others, such as "X, Y, Z stay open", "the claim carries its ___", "___ is a statement about ___ and says nothing about ___", and repeating a pair of numbers followed quickly by the difference between them. (It's +0.009!)

      And I have to think hard to guess what it probably meant by "Comparability is a property of the reference the results are traceable to, not of the number". I don't think "[X and Y] are different scopes by canonical bytes" even makes sense.

  • mpalmer 13 hours ago ago

    I genuinely have no idea what this content is supposed to convey. Even the TL;DR lead-in is inscrutable. If there is one nice thing to say about this, I honestly can't find it.