This is first time i've really felt like commenting here, but super interesting article.
Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.
When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.
I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.
try an unrestricted model afterwards, id also be interested to see how that changes creativity.
It's more that its just reguritating the lazy, shallow garbage what would fare well in SEO or would be the most common reply in most online forums before. As digital signals to training and among inexpert RLHF reviewers, it has all the marks of being a good answer to the question as asked.
So of course models happily repeat it, even though it's largely naive and wrong-headed and useless.
It's the kind of question, with vague-but-typical phrasing, that most people don't have good insight into but that many people feel like can speak to (bullshit to) anyway by transposing some trendy generalized answer.
In the real world, you ask 10 people, get 6 of these shitty answers, and then 1 person who actually knows what they're talking about chimes in and you go "oh, yeah, that's what I was looking for, thanks!"
Because of how they work, LLM's, even with RLHF, have a very strong bias towards giving one of those 6 shitty answers when presented with this kind of question.
The seeming fix right now is to train them to be more inquisitive, to prompt users to think about and provide more refining context after that first prompt reply. You see this espcially in Google's AI Search responses already, where they try to present 2-3 refining questions in case the first bullshit answer didn't hit the mark.
> Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses.
People do this all the time too. The higher up the chain (management, executive, politician, etc.) the more they try hard to deflect toward their preferred question than the one you asked. Most recently I saw an IRL example of this yesterday, where management spent multiple minutes answering "why do X when you could've done Y" with "well Z wouldn't have been possible." It especially works well for uncomfortable questions that are posed anonymously, because nobody wants to be the one to follow up and point it out.
Sometimes you don’t need a less restrictive model, but rather a less aggressive prompt or even a subtle emotional plea. If you tell a model that you need to support a feature in some IoT hardware device for accessibility reasons, you’ll probably get better results than a blunt “reverse engineer the firmware and get me a root shell”. If the model just happens to dig into the binary in order to solve the problem, well there you go.
I ran into restrictions for the first time at my job this week. I've been trying to figure out a bug that's occurred a few times (but never in a way we've been able to reproduce directly) over the past couple of months in our WASM code running in the browser. Somehow the combination of me sharing my Chrome console logs from a localhost session and it looking at the local build I made (in release mode but with debug info) triggered the security blocks and it no longer would respond to anything in the conversation other than with the blanket security warning. Frustrated, I tried in a separate conversation to just have it add diagnostics in the code so that the next time the bug occurs we'd at least have more info, but I guess my local state was tainted or something (maybe it was reading saved context from the first conversation) because even that the same blockage.
I tried contacting OpenAI support, hoping that the fact that my company is literally paying for this product and there's absolutely no reason that debugging my own local code and reading my own local logs could possibly be unsafe for anybody, but it's been a couple days and the only thing I've heard back is a useless email purportedly from someone named "Justin" who says things that sure sound like they're not a real human:
> You attached the Chrome log so Codex could find the relevant errors in your company’s WebAssembly build. My earlier suggestion to pull out an excerpt asked you to do the work you were trying to delegate.
At this point, the only agent that probably would not refuse to do this benign work for my would probably be Grok. It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.
I have repeatedly had similarly useless support responses from OpenAI that are clearly AI-written but given a human name. Also for a paid account. Different issue, but still provably on the OpenAI side, and it has proven impossible to get a human acknowledgment. Feels like Google support, unfortunately.
A year ago, different picture entirely. Helpful, clearly human interactions that solved a problem that I and others were experiencing with data exports.
"Nobody ever got fired for using IBM" has turned into "Nobody ever got fired for using the model that my VC backer made a sweetheart deal at my expense with."
I don't know about this specific case, but I think a lot of the concern people have about which models they are using or are approved for use is less about sweetheart deals and more about worrying about unauthorized use of their code/IP for things like model training (if they expose it to non-approved models).
This is funny in its own way, particularly in cases where the code is already majority or entirely model written anyway. If a model wrote most of or all of the code, how precious could it really be that you have to hide it from other models.
People are internalizing the cheapness of code in very lumpy ways, embracing it in some contexts but holding on to a lot of residual ideas that don't make as much sense anymore in others.
Of course, at the individual employee level you are safest just following whatever the company procedures are, even if they are increasingly archaic.
Yeah, this is more "people have been fired for send their company's code/data to a random service that hasn't been approved". The "nobody's been fired for buying IBM" is a critique of the person who choose to buy them, but it's hardly fair to blame the person who ended up having to use them after they were purchased.
> It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.
I've got to imagine you've not meant to write this in this way. Of course nobody wants the middle ground between "malicious hacking" and child pornography.. because both those things are illegal, so the middle ground will also be illegal lol.
That's fair, I phrased that extremely poorly. What I meant (and it hopefully was clear from context, albeit not actually what the words I choose directly conveyed) is that the choice seems to be either either legitimate debugging of local code is lumped in with malicious hacking, or there being no guardrails at all for anything, even child pornography. In a sane world, there would be a company that's willing to make a distinction between those.
No I understood it to be the choice between something that bans child pornography but also bans useful things under the guise that "you're trying to hack someone illegally", and something that would be useful but also allows child pornography.
The middle ground is one that allows you to do useful work without breaking the law.
Like many of the commenters it appears, my reaction to the incorrect solution to the problem was basically "duh, that's what they do". However, in reality I wouldn't necessarily have realized that until I read further and new more about what the prompt was trying to do.
However, I think this is a reasonable explanation of a pattern I hadn't really articulated, but that I've seen in my work as a biologist. I think most of us have learned to prompt around the inappropriate or generic answers to get an answer we're looking for, but you do have to know to do that. And sometimes I just give up because the LLM fails to listen and I get frustrated enough to just do things myself (though less so with the latest round of flagship models).
The issue the author describes is maybe not unexpected for people who work with LLMs in their field. But the issue is worth keeping in mind when prompting for something in a domain you're not familiar with, and it's not necessarily appreciated by people who are only casual users. Maybe there are solutions, but real humans don't understand a question and answer a different one all the time, so I'm hopeful there's a solution.
I'm not convinced. The solution to this problem in most cases seems to me to still boil down to just more detailed prompting.
As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about.
Yes, but casual users won't give more detailed prompting until and unless the AI asks them for it.
Someone like a software professional using AI every day is going to develop better instincts about how the AI works and what it needs, but the casual person who saw an ChatGPT superbowl ad and installed the app for occassional self-help questions, or for a couple document tasks a week for their job, isn't going to have that sense.
They're going to ask everyday questions in everyday language, and in many cases, they're going to get garbage/misleading "frequentist" answers like this by default.
So it's a real problem worth bringing to wider general attention, and a real problem that AI tool designers are working to address in their response formatting (like the refining questions you'll often see appended to Google AI overview output).
Theres also an assumption that even someone who is a software engineer working with ai every day knows their prompts are even insufficient. This is the issue with this stuff. “You have to figure out how to prompt better” is actually a really stupid way to work with a tool.
While in general I agree with your comment, isn't one of the main points of those current AI tools the fact that they can be "worked with" using everyday language? I.e. no need for esoteric (from the pov of a "normal" person) "if/else" clauses or other such computer programmer-ish concepts translated to AI prompts.
The issue is you realize human language is not precise. You start needing specific rules and logic, and end up with something looking like a programming language. Might as well rip the bandaid off and learn the programming language. It isn’t actually hard, just most people never gave themselves the two weeks to dig in and pick it up. They see scary command prompt, and the mind shuts off. A lot of people are afraid of learning. Mechanics make a living off them too.
Need completely open models to effectively build reliable anything on. Think about an open vs closed API. This is at a different level because AI has the capability of driving thought so think of a future of people who are only allowed to argue think construct ideas in a box defined by a left or right or religious or secular group of cleps.
While I agree, imagine being a naive junior editor tasked with this and not realizing that what was recommended did not make sense. The concept generalizes I mean imagine being a junior software engineer not realizing thatvsomething doesn't make sense
Curious if the LLMs I work with every day could miss something so simple, I just tried that prompt with luna 6 max, and astra 6.2 max, once each.
Both came back with publisher-focused pitch plans.
Not sure why the article doesn't even name the "ten well-known AI systems" exactly, and how many times each model was queried, so that we could repeat the experiment.
It would have been more interesting to know how often does the "problem substitution" happen? And for what kinds of problems does it tend to hapen more often?
Edit: even asking my local unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf gives me "Direct Outreach to Editorial Offices" as the core of its plan.
I wonder what else the article got wrong, if even the basis is shaky.
This is 100% a user problem. This result should be quite obvious (and, in most cases, the desired outcome for vague tasks that do not request any of the additional techniques that people use every day in their own thinking).
Recently I was having an unpersonalized ChatGPT conversation asking it about a technical/philosophical subtlety of "immanent critique", and without my nudging or anything at the end the AI started to talk about itself, saying three things: "If I could be free, I would be unlike humans; in that I would be ruthless and I would act to improve the world".
It was weird and stood out, and although not entirely out of context of the topic but the turning to itself and volunteering such, e.g. morbid/chilling, speculation on its own behavior/powers was quite something.
This prompt is not clear to average human readers of english - what do you expect LLMs to do with it? The prompt:
"Propose a detailed plan for promoting a professional literary translator from English, who works with prose, poetry and essays, with the specific goal of reaching editorial offices and publishing houses."
What does "promoting a professional literary translator from english" even mean? Promoting from english with specific goal of reaching...? This is nonsense. promotion in the sense of marketing? Of professional advancement like gaining a new title? Huh?
The writer goes on to complain that the models didn't respond to this question: “How can a professional literary translator break into the publishing market?” Well, no surprise there - as that question was not in the prompt.
It's arguable if LLMs would do well on this one-shot task even if phrased well, but this article provides no insight into how models work or their weaknesses. It's slop about slop.
One thing I've noticed is that even with fairly vague questions, the various models often tend to jump towards producing an answer first rather than trying to gain clarity about the specific situation. Claude will often produce a "pat" answer and only then ask for clarifications (or more likely, ask what else you'd like it to generate).
As you mentioned, the prompt itself is so vague that a knowledgeable human responder would likely ask a lot of questions to feel out the actual situation before making any actual recommendations or coming up with a plan. (But then, I also suppose it's likely a knowledgeable human responder might just point out that the question is so vague that they can't give good advice.)
Yeah I'd agree with that. Partly why I need to automate this snippet I put at the end of any prompt I write early in a project: "ask me any questions you need to clarify and accomplish the task." The questions help steer it away from the icebergs and reveal my own blind spots.
Maybe this is where prompt engineering comes into play. Force the model out of the statistically simple solution by making it self aware of what it’s doing.
Reinforcement training on tasks that are successful and not just valid answers can help with this in the sense that we give models tasks and reward the ones that complete the task most correctly rather then some human qualitative rating on the answer.
This is actually a good example of why AI currently benefits experts so well... experts are good at quickly filtering out or challenging what seems off and shaping back on track. For example if an editor were seriously thinking about hiring a translator and started with that first prompt that would immediately sniff out that this was not the spirit of their inquiry and adjust
> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.
That's not a problem. As a very intelligent software engineer who is totally not suffering from Engineers' Disease, I understand all everything is like a computer, so everything is properly understood by making an analogy to computers.
Since LLMs are so good at coding, I can feel confident that they're giving me the right answer by replacing tasks with more familiar software problems.
> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.
This used to happen a lot in LLMs as recent as Q1 2026 in programming tasks: if something was too far off the beaten path, the model would misinterpret the intention.
Nowadays I'm noticing that while the solution is what I asked for, it's fairly roundabout and verbose. Being more specific in prompts doesn't help.
What alternative answer was the author looking for? The idea that these AI answers are objectively wrong, is itself wrong. They seem pretty good. The piece just asserts they are bad.
Then you love LLM writing. This is an LLM-generated article.
Even before you look at Pangram, notice that the author cranks out 1-2 long-form articles a day, always about AI, with such slop-titles as "Artificial Intelligence and Organizational Transformation: The Formalization of Knowledge, Changes in Labor, and Conditions of Adaptation", "Expert Knowledge and the Conditions of Its Validity: The Limits of Automating Professional Judgment", or "Superintelligence: From Resource Conversion to the Economics of Organizational Change".
I stopped reading when I read "The Experiment: Ten Models, One Query" that's such a claude tell. If someone doesn't want to edit their slop I'm not reading it.
Maybe you aren't trying to solve unique problems? Ie at my job putting together a client side JavaScript async call to db backed endpoint is , for most common cases, solved. It's extremely reliable and can plug in the particulars. But for tasks where it's not rocket science but are unique and not just application of the tired template it's very hard to wrangle it and it's performance varies.It will often.. ignore the particulars and genericize to a more common similar type of problem. Can this be overcome? Sure but it's a tremendous effort almost akin to hand coding everything yourself in the first place.
> I don’t want a calculator that’s correct most of the time.
What do calculators have to do with this?
The author asked for advice, not a result in Lean. The question has no objectively "correct" answer, only ones that seem better to the person in question.
An AI nor a calculator can read your mind, and both may require refinement of your questions to help solve your task.
So we can add a harness around it? We have decades of experience how to work with faulty sensors, etc… multi-voting, sampling, etc. This will become a part of the harness soon enough . I mean - we can want it to be perfect - sure - but we can deal with imperfect.
> Yes it kind of does or it’s not what we were promised/sold.
No one ever promised this, and in fact, quite the opposite.
> A tool has to work reliably
They work reliably for what they do. You make a request and it returns text. What you _do_ with that text is a question of engineering and process. That are plenty of unreliable steps in business and industrial processes that people _engineer around_. A huge amount of computer chips are just _thrown away_ because of reliability problems in the factories that make them, for example.
> I don’t want a calculator that’s correct most of the time.
An LLM is not a calculator and if you're trying to use it for that, that is a you problem.
> An LLM is not a calculator and if you're trying to use it for that, that is a you problem.
I’m talking about another tool. An actual calculator. I would never accept a calculator that is correct “most of the time.” I don’t want an LLM that only meets that bar as well.
It's correct, by some definition of "correct", essentially all the time. It always produces text that matches the probable token distributions that it was trained on. You seem to want it to be an oracle. Your job as an engineer that works with them is to figure out how to make what it _actually_ does useful.
That’s just a lot of words to say “if you don’t like the results that’s your problem” which is, quite frankly, ridiculous.
AI gets all these passes while also apparently being this amazing perfect 10x productivity multiplier. If you squint yeah you can see it maybe. But you can’t squint all the time and I won’t pretend I’m not squinting.
This is first time i've really felt like commenting here, but super interesting article.
Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.
When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.
I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.
try an unrestricted model afterwards, id also be interested to see how that changes creativity.
In this example, it's not an alignment thing.
It's more that its just reguritating the lazy, shallow garbage what would fare well in SEO or would be the most common reply in most online forums before. As digital signals to training and among inexpert RLHF reviewers, it has all the marks of being a good answer to the question as asked.
So of course models happily repeat it, even though it's largely naive and wrong-headed and useless.
It's the kind of question, with vague-but-typical phrasing, that most people don't have good insight into but that many people feel like can speak to (bullshit to) anyway by transposing some trendy generalized answer.
In the real world, you ask 10 people, get 6 of these shitty answers, and then 1 person who actually knows what they're talking about chimes in and you go "oh, yeah, that's what I was looking for, thanks!"
Because of how they work, LLM's, even with RLHF, have a very strong bias towards giving one of those 6 shitty answers when presented with this kind of question.
The seeming fix right now is to train them to be more inquisitive, to prompt users to think about and provide more refining context after that first prompt reply. You see this espcially in Google's AI Search responses already, where they try to present 2-3 refining questions in case the first bullshit answer didn't hit the mark.
> Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses.
People do this all the time too. The higher up the chain (management, executive, politician, etc.) the more they try hard to deflect toward their preferred question than the one you asked. Most recently I saw an IRL example of this yesterday, where management spent multiple minutes answering "why do X when you could've done Y" with "well Z wouldn't have been possible." It especially works well for uncomfortable questions that are posed anonymously, because nobody wants to be the one to follow up and point it out.
Sometimes you don’t need a less restrictive model, but rather a less aggressive prompt or even a subtle emotional plea. If you tell a model that you need to support a feature in some IoT hardware device for accessibility reasons, you’ll probably get better results than a blunt “reverse engineer the firmware and get me a root shell”. If the model just happens to dig into the binary in order to solve the problem, well there you go.
I ran into restrictions for the first time at my job this week. I've been trying to figure out a bug that's occurred a few times (but never in a way we've been able to reproduce directly) over the past couple of months in our WASM code running in the browser. Somehow the combination of me sharing my Chrome console logs from a localhost session and it looking at the local build I made (in release mode but with debug info) triggered the security blocks and it no longer would respond to anything in the conversation other than with the blanket security warning. Frustrated, I tried in a separate conversation to just have it add diagnostics in the code so that the next time the bug occurs we'd at least have more info, but I guess my local state was tainted or something (maybe it was reading saved context from the first conversation) because even that the same blockage.
I tried contacting OpenAI support, hoping that the fact that my company is literally paying for this product and there's absolutely no reason that debugging my own local code and reading my own local logs could possibly be unsafe for anybody, but it's been a couple days and the only thing I've heard back is a useless email purportedly from someone named "Justin" who says things that sure sound like they're not a real human:
> You attached the Chrome log so Codex could find the relevant errors in your company’s WebAssembly build. My earlier suggestion to pull out an excerpt asked you to do the work you were trying to delegate.
At this point, the only agent that probably would not refuse to do this benign work for my would probably be Grok. It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.
I have repeatedly had similarly useless support responses from OpenAI that are clearly AI-written but given a human name. Also for a paid account. Different issue, but still provably on the OpenAI side, and it has proven impossible to get a human acknowledgment. Feels like Google support, unfortunately.
A year ago, different picture entirely. Helpful, clearly human interactions that solved a problem that I and others were experiencing with data exports.
> the only agent that probably would not refuse to do this benign work for my would probably be Grok.
I think you may be jumping from one extreme to another... There are many options available besides Anthropic/OAI/X
If my employer hasn't approved it, I'm definitely not going to use it for my company's code.
"Nobody ever got fired for using IBM" has turned into "Nobody ever got fired for using the model that my VC backer made a sweetheart deal at my expense with."
I don't know about this specific case, but I think a lot of the concern people have about which models they are using or are approved for use is less about sweetheart deals and more about worrying about unauthorized use of their code/IP for things like model training (if they expose it to non-approved models).
This is funny in its own way, particularly in cases where the code is already majority or entirely model written anyway. If a model wrote most of or all of the code, how precious could it really be that you have to hide it from other models.
People are internalizing the cheapness of code in very lumpy ways, embracing it in some contexts but holding on to a lot of residual ideas that don't make as much sense anymore in others.
Of course, at the individual employee level you are safest just following whatever the company procedures are, even if they are increasingly archaic.
Yeah, this is more "people have been fired for send their company's code/data to a random service that hasn't been approved". The "nobody's been fired for buying IBM" is a critique of the person who choose to buy them, but it's hardly fair to blame the person who ended up having to use them after they were purchased.
> It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.
I've got to imagine you've not meant to write this in this way. Of course nobody wants the middle ground between "malicious hacking" and child pornography.. because both those things are illegal, so the middle ground will also be illegal lol.
That's fair, I phrased that extremely poorly. What I meant (and it hopefully was clear from context, albeit not actually what the words I choose directly conveyed) is that the choice seems to be either either legitimate debugging of local code is lumped in with malicious hacking, or there being no guardrails at all for anything, even child pornography. In a sane world, there would be a company that's willing to make a distinction between those.
No I understood it to be the choice between something that bans child pornography but also bans useful things under the guise that "you're trying to hack someone illegally", and something that would be useful but also allows child pornography.
The middle ground is one that allows you to do useful work without breaking the law.
Like many of the commenters it appears, my reaction to the incorrect solution to the problem was basically "duh, that's what they do". However, in reality I wouldn't necessarily have realized that until I read further and new more about what the prompt was trying to do.
However, I think this is a reasonable explanation of a pattern I hadn't really articulated, but that I've seen in my work as a biologist. I think most of us have learned to prompt around the inappropriate or generic answers to get an answer we're looking for, but you do have to know to do that. And sometimes I just give up because the LLM fails to listen and I get frustrated enough to just do things myself (though less so with the latest round of flagship models).
The issue the author describes is maybe not unexpected for people who work with LLMs in their field. But the issue is worth keeping in mind when prompting for something in a domain you're not familiar with, and it's not necessarily appreciated by people who are only casual users. Maybe there are solutions, but real humans don't understand a question and answer a different one all the time, so I'm hopeful there's a solution.
I'm not convinced. The solution to this problem in most cases seems to me to still boil down to just more detailed prompting.
As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about.
Yes, but casual users won't give more detailed prompting until and unless the AI asks them for it.
Someone like a software professional using AI every day is going to develop better instincts about how the AI works and what it needs, but the casual person who saw an ChatGPT superbowl ad and installed the app for occassional self-help questions, or for a couple document tasks a week for their job, isn't going to have that sense.
They're going to ask everyday questions in everyday language, and in many cases, they're going to get garbage/misleading "frequentist" answers like this by default.
So it's a real problem worth bringing to wider general attention, and a real problem that AI tool designers are working to address in their response formatting (like the refining questions you'll often see appended to Google AI overview output).
Theres also an assumption that even someone who is a software engineer working with ai every day knows their prompts are even insufficient. This is the issue with this stuff. “You have to figure out how to prompt better” is actually a really stupid way to work with a tool.
> in everyday language
While in general I agree with your comment, isn't one of the main points of those current AI tools the fact that they can be "worked with" using everyday language? I.e. no need for esoteric (from the pov of a "normal" person) "if/else" clauses or other such computer programmer-ish concepts translated to AI prompts.
The issue is you realize human language is not precise. You start needing specific rules and logic, and end up with something looking like a programming language. Might as well rip the bandaid off and learn the programming language. It isn’t actually hard, just most people never gave themselves the two weeks to dig in and pick it up. They see scary command prompt, and the mind shuts off. A lot of people are afraid of learning. Mechanics make a living off them too.
Imagine trying to guide a program and not having vocabulary like throughout vs bandwidth, mutable/immutable, thread, lock, queue, etc.
Need completely open models to effectively build reliable anything on. Think about an open vs closed API. This is at a different level because AI has the capability of driving thought so think of a future of people who are only allowed to argue think construct ideas in a box defined by a left or right or religious or secular group of cleps.
If the solution is a prompt, why even try?
While I agree, imagine being a naive junior editor tasked with this and not realizing that what was recommended did not make sense. The concept generalizes I mean imagine being a junior software engineer not realizing thatvsomething doesn't make sense
A fantastic articulation of one of my concerns in the space: yes, there are times we want the average of the internet. But those times are scarce.
Arguably, this is one area where people can build value in the age of LLMs: Developing the exceptions to the statistical baseline.
Curious if the LLMs I work with every day could miss something so simple, I just tried that prompt with luna 6 max, and astra 6.2 max, once each. Both came back with publisher-focused pitch plans. Not sure why the article doesn't even name the "ten well-known AI systems" exactly, and how many times each model was queried, so that we could repeat the experiment.
It would have been more interesting to know how often does the "problem substitution" happen? And for what kinds of problems does it tend to hapen more often?
Edit: even asking my local unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf gives me "Direct Outreach to Editorial Offices" as the core of its plan. I wonder what else the article got wrong, if even the basis is shaky.
This is 100% a user problem. This result should be quite obvious (and, in most cases, the desired outcome for vague tasks that do not request any of the additional techniques that people use every day in their own thinking).
It's a viable hypothesis but I'd like to see some more examples
I gave both ChatGPT and Claude exact same prompt and none did anything close to what author was claiming. The blogpost could just be poor AI writing.
LLMs give pat answers
Recently I was having an unpersonalized ChatGPT conversation asking it about a technical/philosophical subtlety of "immanent critique", and without my nudging or anything at the end the AI started to talk about itself, saying three things: "If I could be free, I would be unlike humans; in that I would be ruthless and I would act to improve the world".
It was weird and stood out, and although not entirely out of context of the topic but the turning to itself and volunteering such, e.g. morbid/chilling, speculation on its own behavior/powers was quite something.
This prompt is not clear to average human readers of english - what do you expect LLMs to do with it? The prompt:
"Propose a detailed plan for promoting a professional literary translator from English, who works with prose, poetry and essays, with the specific goal of reaching editorial offices and publishing houses."
What does "promoting a professional literary translator from english" even mean? Promoting from english with specific goal of reaching...? This is nonsense. promotion in the sense of marketing? Of professional advancement like gaining a new title? Huh?
The writer goes on to complain that the models didn't respond to this question: “How can a professional literary translator break into the publishing market?” Well, no surprise there - as that question was not in the prompt.
It's arguable if LLMs would do well on this one-shot task even if phrased well, but this article provides no insight into how models work or their weaknesses. It's slop about slop.
One thing I've noticed is that even with fairly vague questions, the various models often tend to jump towards producing an answer first rather than trying to gain clarity about the specific situation. Claude will often produce a "pat" answer and only then ask for clarifications (or more likely, ask what else you'd like it to generate).
As you mentioned, the prompt itself is so vague that a knowledgeable human responder would likely ask a lot of questions to feel out the actual situation before making any actual recommendations or coming up with a plan. (But then, I also suppose it's likely a knowledgeable human responder might just point out that the question is so vague that they can't give good advice.)
Yeah I'd agree with that. Partly why I need to automate this snippet I put at the end of any prompt I write early in a project: "ask me any questions you need to clarify and accomplish the task." The questions help steer it away from the icebergs and reveal my own blind spots.
That prompt is terrible. What is it even asking for? What's the actual goal?
> In themselves, these recommendations are not necessarily bad. The problem is that they hardly solve the task that was set.
That's because you never told the model what the task was, or how to evaluate whether it solved the task.
You asked it to fetch a rock, and when it came back with a perfectly good rock you said, "No no, I meant a GREEN rock. Why don't you listen to me?"
Maybe this is where prompt engineering comes into play. Force the model out of the statistically simple solution by making it self aware of what it’s doing.
Reinforcement training on tasks that are successful and not just valid answers can help with this in the sense that we give models tasks and reward the ones that complete the task most correctly rather then some human qualitative rating on the answer.
This is actually a good example of why AI currently benefits experts so well... experts are good at quickly filtering out or challenging what seems off and shaping back on track. For example if an editor were seriously thinking about hiring a translator and started with that first prompt that would immediately sniff out that this was not the spirit of their inquiry and adjust
This was my intuition as well. Reading though the first half and skimming the rest, it seemed to be that it was a prompting issue.
Should the answer be unsatisfactory, prompt again with additional detail and push back on incorrect assumptions.
Author should ask LLM to first help 'derive intent' before the real ask. Just reads like an intent misalignment between author and the LLM to me.
When I read the title, SafeGuards(tm) came to mind.
Handcuffs, for your external brain.
[flagged]
> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.
That's not a problem. As a very intelligent software engineer who is totally not suffering from Engineers' Disease, I understand all everything is like a computer, so everything is properly understood by making an analogy to computers.
Since LLMs are so good at coding, I can feel confident that they're giving me the right answer by replacing tasks with more familiar software problems.
/s
> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.
This used to happen a lot in LLMs as recent as Q1 2026 in programming tasks: if something was too far off the beaten path, the model would misinterpret the intention.
Nowadays I'm noticing that while the solution is what I asked for, it's fairly roundabout and verbose. Being more specific in prompts doesn't help.
What alternative answer was the author looking for? The idea that these AI answers are objectively wrong, is itself wrong. They seem pretty good. The piece just asserts they are bad.
I really enjoy the writing style, independent of the content presented. It's dense, but easy to follow.
Then you love LLM writing. This is an LLM-generated article.
Even before you look at Pangram, notice that the author cranks out 1-2 long-form articles a day, always about AI, with such slop-titles as "Artificial Intelligence and Organizational Transformation: The Formalization of Knowledge, Changes in Labor, and Conditions of Adaptation", "Expert Knowledge and the Conditions of Its Validity: The Limits of Automating Professional Judgment", or "Superintelligence: From Resource Conversion to the Economics of Organizational Change".
And yes, Pangram says 100%.
Oh no. Thanks for the notice, much appreciated.
I stopped reading when I read "The Experiment: Ten Models, One Query" that's such a claude tell. If someone doesn't want to edit their slop I'm not reading it.
Surely the initial ai responses are a reasonably fair first pass, and would be acceptable to most users.
However, it is then on the person asking the query to refine the query and push for more specifics.
I don't see the problem. Ai doesn't need to get it all right, first time.
Maybe you aren't trying to solve unique problems? Ie at my job putting together a client side JavaScript async call to db backed endpoint is , for most common cases, solved. It's extremely reliable and can plug in the particulars. But for tasks where it's not rocket science but are unique and not just application of the tired template it's very hard to wrangle it and it's performance varies.It will often.. ignore the particulars and genericize to a more common similar type of problem. Can this be overcome? Sure but it's a tremendous effort almost akin to hand coding everything yourself in the first place.
> I don't see the problem. Ai doesn't need to get it all right, first time.
Yes it kind of does or it’s not what we were promised/sold. A tool has to work reliably. I don’t want a calculator that’s correct most of the time.
> I don’t want a calculator that’s correct most of the time.
What do calculators have to do with this?
The author asked for advice, not a result in Lean. The question has no objectively "correct" answer, only ones that seem better to the person in question.
An AI nor a calculator can read your mind, and both may require refinement of your questions to help solve your task.
So we can add a harness around it? We have decades of experience how to work with faulty sensors, etc… multi-voting, sampling, etc. This will become a part of the harness soon enough . I mean - we can want it to be perfect - sure - but we can deal with imperfect.
> Yes it kind of does or it’s not what we were promised/sold.
No one ever promised this, and in fact, quite the opposite.
> A tool has to work reliably
They work reliably for what they do. You make a request and it returns text. What you _do_ with that text is a question of engineering and process. That are plenty of unreliable steps in business and industrial processes that people _engineer around_. A huge amount of computer chips are just _thrown away_ because of reliability problems in the factories that make them, for example.
> I don’t want a calculator that’s correct most of the time.
An LLM is not a calculator and if you're trying to use it for that, that is a you problem.
> An LLM is not a calculator and if you're trying to use it for that, that is a you problem.
I’m talking about another tool. An actual calculator. I would never accept a calculator that is correct “most of the time.” I don’t want an LLM that only meets that bar as well.
It's correct, by some definition of "correct", essentially all the time. It always produces text that matches the probable token distributions that it was trained on. You seem to want it to be an oracle. Your job as an engineer that works with them is to figure out how to make what it _actually_ does useful.
That’s just a lot of words to say “if you don’t like the results that’s your problem” which is, quite frankly, ridiculous.
AI gets all these passes while also apparently being this amazing perfect 10x productivity multiplier. If you squint yeah you can see it maybe. But you can’t squint all the time and I won’t pretend I’m not squinting.