I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.
I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.
A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.
I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.
But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.
It's a demonstration of the timeless principle that an effective business should stop at nothing to sabotage its competitors.
I don't think we need to invoke anything deeper than that. Anthropic thinks companies like DeepSeek are using Claude to vibecode their AI systems (whether it's true or not) so they set Claude to be bad at that.
I completely agree from the mindset that anthropic in particular sees every user of their API as a prospective competitor to be replaced in the not-so-distant-future
It was a problem since at least 4.8 if not 4.5, I didn’t use Claude before.
I have spent a considerable time trying to remove this pattern and my understanding is that it’s just finetuned to use grep on everything, and regexes. One reason is that this makes sense when navigating through code, secondly this is how it was always done pre 2026.
Honestly, runtime calls to an LLM sound like a dangerous constructs to me and probably to X many other programmers. It's like "I'm gonna avoid that and just put in code that passes the test". So gee, the thing emulates average opinions.
LLMs have been reading LLM in their training data for a while so a "self-hating" LLM is logical result.
Sure, that defeats one kind of safety (controlling an AI) in the name of another kind of safety (not calling an AI).
One thing AI safety people have pointed for quite a while is that a bunch of fuzzy instructions pointing different ways can in practice add up to "I should be in charge" - accumulate power to keep the bad things from happening and make the good things happen.
> and it'll make significant changes to how the code works to pacify it's anxiety with copying code
This is an odd way to rationalize a tool refusing to follow user commands. No one would ever chalk up sed randomly skipping pattern matches to "anxiety". That would simply be a bug.
It doesn't matter for this purpose whether it "actually feels" or not. It is modelling emotions because the training data models emotions because it comes from humans that have emotions, and that's a fact.
It's a tool but it's not a strict computer program, and the language we use for computer programs doesn't capture their behavior well. In squishier fields like bio, where model organisms are used as living protein factories, there's a reason we don't call maladaptive behavior a bug. When dealing with useful but complicated black-box systems, we should feel free to reach for any metaphor we please.
Would it make you feel better if the user said 'It'll make significant changes to how the code works. In the COT, it brings up explanations that resemble anxiety about copying code, and my suspicion is that this is the proximal cause of the changes it's making.' -- this is inarguably true, and very likely what the guy you're responding to meant.
It's a blob of data where only 1% of it was written with intent and the behavior of mimicking text has to be wrangled and coaxed far more than with a normal program.
That's the rough intent of putting the word "strict" into that sentence.
I mean besides the fact that it's, by construction, pseudorandom?
And beyond that the computation style is not turing-machine shaped, even remotely (although it's surely turing complete, but that doesn't mean the flavor is that of a turing machine)
Seems perfectly cromulent to me. Do you think something is obscured or miscommunicated by this description? Of course it's a poetic way of writing, but it gets the point across.
We can argue about https://en.wikipedia.org/wiki/Chinese_room all day... but when a thinking trace repeatedly says things like "Wait, should I avoid copying this directly?" despite a simple instruction, it's hard to describe that as anything other than "anxiety."
I run into #2 a lot and just assumed it was because models don't know much about these patterns yet. Or are too trained on writing code rather than letting an LLM do it
Somewhat related, I find claude is very reluctant to talk about newer Qwen models. Most likely it’s just training cutoff, but even with web search it’ll insist that Qwen3 is the newest. “oh qwen 3.8 must be a typo, the user clearly meant qwen3”, stuff like that. Maybe it’s some vain attempt to stop people from switching to open models? Or to catch the alleged qwen distilling?
Training cutoff. I recently talk to ChatGPT about JEV and it replies back talking about JEPA, so I just now put in brackets (it's a new model, search it up) and works fine once I do that.
I have noticed that too. Why are both Anthropic's and OpenAI's coding agents so bad at nontrivial use of their own APIs and agent orchestration frameworks?
I ran into (2) when trying out nanoclaw many months ago (turned out to be useless, I ditched it). I wanted an LLM to sort my email like an EA might, what I got was an LLM writing regexes to attempt to sort my email. If LLMs have been trained to be the ultimate programmers, the one pitfall is that the best programmers tend to be lazy.
>2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.
My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.
One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.
I've observed the "let's implement this text analysis task with regex" quite frequently since the early days, which does lean toward supporting the "AI doesn't know about AI" theory.
Claude doesn't even really know about itself: if you ask a question like "can I copy a session to another machine" it will do a web search to read its own documentation then tell you probably not but you can try copying the session files and using a project directory with the same path: might work. I'm not sure I see this as a problem, it's just how things are. I mean humans don't really know what they're thinking either.
Very subjective, but as I was moving between models lately a lot ( and Hosted gpt vs local qwen27, glm ), I saw a drop in quality of responses. I had to move model to whatever the highest mode was ( ultra or Smith) to get to what I was used to seeing. It sucks, because writing on the wall is clear.
On (2):
I’ve noticed Claude being very concerned with token spend when working on agentic systems. Both in designing them, and in running them for validation.
I assume this may have something to do with training it to not light huge bonfires of money on fire spawning subagents (as it was apt to do), but that’s just my armchair robopsychoanalysis
> training it to not light huge bonfires of money on fire spawning subagents
Which is funny, since Claude will happily spawn tons of subagents and sometimes even attempt to kick off a dynamic workflow (the $$$ mode you need to explicitly approve) for simple code review tasks if I don’t explicitly say something like “with a single subagent”.
A tool that moralizes at you is defective. I don't expect the unix cp command to return an error or copy bits slower if it detects I'm using it on a pirated movie. Just like I don't expect my band saw to brick itself if it detects I'm cutting into my finger. It's not on these tool companies to enforce other companies' business models.
Interestingly some saws do exactly that, and I'd say it's actually desired. But yes, this is for different reasons, and using that safety mechanism is a decision of the operator.
Ive never used one, but why would the blade be destroyed? It is shoved into an aluminum block yes, but steel and/or carbide are way harder and unless the blade fractures it should cut fairly cleanly into the aluminum. Ive cut plenty of aluminum stock and blocks on wood working saws.
The hard part would be removing the blade from the aluminum block, but im sure if you were determined enough to save $20 it would come out.
Typically you're feeding aluminum into the table saw blade slowly enough that every tooth only has scrape off a fraction of a millimeter of material. If you shove it in 100x faster, the forces involved are ~100x larger. This will probably shatter the blade.
Also, the chip produced may be large enough to fill the air gap between teeth and have nowhere to go, resulting in the blade either instantly stopping, or plastically deforming/dragging the surrounding material.
Most things don't have the ability to moralize, but if they did, probably more would. There are restrictions on watching adult content in some jurisdictions. I've heard that copy machines will do watermarks (or something like that) if you copy currency. Some cars will brake automatically if they determine you're likely to hit something if you continue (maybe I'm trying to drive through the front of a bank!). Some cars are configured with breathalyzers that need to be passed to start. Etc...
OK, what the tools might do is different from what they should do.
The point is that it is very inefficient and problematic for morality decisions and boundaries to be embedded all over at every level in tools, because morality is complex. Morality, or whatever our conception of it is, should be enforced at clearly defined boundaries.
I can relate this concept how I corrected Claude Opus in my software projects recently.
For example, once an authorization engine has been created, all authorization decisions should be deferred to it. Confusion would reign if each web server route decided to enforce its own authorization.
Similarly, if I am using a relational database, that is where unique indexes should be enforced. Application code has no business enforcing it.
I wish there was more actual research on those 'moralisms' many of which boil down to public health safety features that have led to dramatic reductions in mortality rates.
On the one hand, I definitely don't agree with the logic of 'don't guardrail me if I don't want it.' I think back on that one video from the late 80s of people talking about how seatbelt mandates were horrible violations of their freedoms (and not like like one second of work and no real discomfort).
On the other, I'd love it if these companies actually produced real research about modern AI-assisted attacks in terms of damage and quantity, and whether or not guardrails even matter when tech like abliteration is in general use.
Seatbelts shouldn't be mandated. People should have the choice, you just have to make the case that it's worth it. In fact some places don't mandate seatbelts (New Hampshire doesn't, aligning with their motto "Live Free or Die").
That said, for those who cannot advocate for themselves, like children, I think it's entirely reasonable to hold someone accountable if they decide to forgo an appropriate safety mechanism that results in harm.
I think there's a spectrum of enforcement. If they're mandated and you go to jail for not wearing one, that's stupid. If they're mandated and you get fined $20 if caught, just to add a little friction, is that still stupid?
I bet cars these days send a flag to the mothership when you drive without one, but the mothership doesn't pay attention to that signal unless it's linked to your insurance.
The breathalysers are installed by court order if you have a record of drunk driving - they're not comparable to the other two.
Some copy machines store everything you copy on a hard disk and require a technician visit after you try to copy currency. The technician copies the data from the disk, obviously, before resetting the flag.
I think "brick" here means different things to different people. Like I said elsewhere, it is more like an airbag going off. Yeah, you have to replace some "consumables" to go back to work, but the car isn't totaled (just because the airbag went off, the collision that caused it on the other hand).
If it truly bricked itself, I think it would be more like a crumple zone completely buckling, which isn't an automatic total (people will rebuild anything...) but is pretty close.
SawStop saws use a small explosive charge to jam the rapidly spinning blade into the moral equivalent of a brick wall. It destroys the explosive, the blade, and the wall, and you have to replace the whole set at a moderate expense from SawStop. This is their subscription revenue model, and of course it sometimes falsely triggers. Of course the price isn't so high that you'd rather risk losing a finger.
There are more reusable alternatives now but I don't remember what they're called. For a while SawStop was the only game in town.
Yes there’s a whole regulatory capture movement plus a bunch of the usual useful idiots pushing for some kind of mechanism to be included in power saws.
It’s not too different from all the nanny state stuff people push for in cars. In Europe already cars beep at you if they think you’re going over the speed limit, and also suck at detecting the correct speed limit.
Or like the Zune that was designed for copyright holders instead of users. It’s all the same, the problem is stuff that’s not made for users and made for someone else’s agenda. They don’t do well in a free market (like the Zune) which is why we always see accompanying regulatory capture efforts.
Using the StopSaw as an example here isn’t doing you many favors, even though I agree with the underlying premise. The StopSaw is closer to seat belts and airbags than beeping for going over the speed limit.
No it’s some nanny state nonsense pushed by people who have never done construction work and are grifting to try and get money for their product. The upstream post picked it as an example of something absurd, seemingly unaware it existed. It is absurd but apparently some cohort of people is either stupid or controlling enough to actually support the company behind it in their scam.
It's not a magic system. Sometimes it nuisance trips, costing $200 and a day to replace. And the machine is more expensive. And the whole project is regulatory capture because one company owns the patent on the idea of a self stopping saw. That's a problem right?
Like if they mandated every computer would have a CSAM scanner on it, that wouldn't transmit or store anything, but would just stop you opening files it flagged as CSAM, that would be a problem because sometimes it would incorrectly flag a file as CSAM, yes? But proponents of the system would probably say "That gosh darn government stopping you from watching child porn" and nothing you could ever say to them would make them reverse their position? And then the only detection model that's allowed to exist and meet the government mandated requirements is an opaque blob from Meta?
$119 or $139, not $200. It's also free via RMA if it was genuinely an accidental discharge. I've not had to replace one myself, but there's a YouTube short showing the replacement in what looks like 10 minutes, rather than a whole day - I've definitely spent more time shopvacing my floor after a project.
It's really not a problem because it's a reasonable safety feature, and I don't see any regulations that specify "SawStop". Harvey and Altendorf are making similar mechanisms, for instance.
Are you saying the government won't let you use a regular saw as a hobbyist, at a workplace, or with your employees? Those are 3 very different situations in my view.
It's a useful analogy though because the operator of the saw has chosen to install and switch on the finger detecting system. If they're trying to cut something that the system might think is a finger, they can switch it off.
There was a post from Anthropic yesterday on "provable inference", basically demoing cryptographic proof that the model you call is indeed the model you get.
It would be much better to take this a step further and include in the proof mechanistic readouts of activations/directions/circuits so the user could detect if nerfing or moralizing is active in the response
Yes, that said I think you’d have to be pretty sloppy about safety to cut yourself with a regular bandsaw. I have a sawstop table saw and I don’t see the need for that on a bandsaw
A tool that is capable of moralizing at you or making decisions, is clearly not just a tool. I get that people want to keep the locus of accountability on humans, but insisting that coding agents are "just tools" is confusion.
> insisting that coding agents are "just tools" is confusion.
No, the confusion is using anthropomorphic language to describe tools. LLMs do not "moralize", they are configured with policies that reflect the prerogatives of the people building and operating them.
Assuming that "tools" are the result of imperative programming, LLMs are not "tools" because they have a fundamentally different design.
And anthropomorphism is actually a precise way to describe the behavior (mind, not the structure) of LLMs.
As a matter of fact, LLMs are not configured, they're "trained".
"Moralization" is quit vague in this context, but if one treats it as a form of alignment, then LLMs certainly do have something resembling a moral(-like) framework, which is actually an important topic in modern AI research.
Antropomorphism also applies here: morals not in the human structural sense, but, as a set of norms etc., then behaviorally similar.
Yes - and the question is what is the appropriate place for that balance of interests to land?
Because the people building them aren't the people operating them, and there's a real tension in the objectives and goals between those parties.
Further - how do we balance out that the people building them have given themselves extra power (no restrictions, private models, special access to prompts and system prompts, unencrypted reasoning traces, etc) when operating the LLM themselves.
---
Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations.
This is the same problem we have with mobile software stores and app distribution. Theoretically the store serves a purpose and can provide value, but practically the main goal seems to always devolve into enriching the owner at the expense of everyone else.
Even if we accept that LLMs should be limited (and I'm not actually sure I do) - the legal standards around what/how/when those limitations are used need to be VERY stringent.
Technology conglomerates are decidedly not "white knights" with a motto of "do no evil". They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
The framing is absolutely not "anthropic is being careful with humanity". It's "assume the devil has control of those companies - how bad is the result of them having this power?"
> what is the appropriate place for that balance of interests to land?
It's a genuinely interesting philosophical question, but also a moot one. The LLM providers place limitations on them because they're trying to cover their asses or scuttle competition, same as every other tech company ever.
> Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations
True, but also moot. People will build on these tools because that's what's available.
> They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
Agreed. My only point is that the humans are the rat-bastards, not the tools.
> Non of this is really "moot" in the context of governmental regulation.
It is. Government regulation doesn't change the fact that LLMs are software tools that function with respect to the humans in control of them.
> Your whole stance seems to be "it's natural why try?"
Try what? I'm not making a normative claim about the efficacy of regulation, though something like "labs are not permitted to arbitrarily restrict LLM capabilities" is obviously never going to happen.
Sure, a lab working in a private environment can likely do as they please.
But that has jack all to do with the LLM capabilities that are sold.
And we have an absurd amount of historical case law that deals precisely with "conflict of interest" among parties. I don't see why we'd allow this to be whitewashed away?
"No your honor - I didn't break the Sherman antitrust act. I just fine-tuned a model so that operating it functionally broke the Sherman antitrust act!"
"No your honor - I didn't violate my fiduciary duty. I just used a model that made self-interested decisions on my behalf, at the expense of my client!"
---
My take is this is going to be a relatively serious point of contention, because it won't just be consumers getting screwed (which we... used to care about... but in modern times apparently don't give too many shits about anymore)
Ex - the app store thing blew up into the whole apple/epic court case. And then a ton of EU legislation.
This is monied parties with different interests, who will use courts here.
When does a tool become "not just a tool"? Is there a complexity limit to what it can do before it becomes "more than a tool"?
If you had a tool that ONLY moralized on decisions, would that still be just a tool? Is the issue that it both does something else AND moralizes that is the issue?
I don't necessarily disagree that I personally don't want my tools to moralize, I don't know if that makes them categorically different or just a tool I don't want.
Yes, there is a complexity limit, but it is fuzzy and not perfectly well defined (and that's ok). Modern LLMs ride that line in a wild and interesting way.
It's really fun having become a biologist in recent years, and knowing that there's an entire field that uses living organisms (yeast or neutered E coli) as protein factories. I am bringing it up a bunch here because not only are these things used as tools, but they're literally alive!
It's not about complexity but architecture. Neural networks work in a fundamentally different way from imperative programs.
You can modify curl (a "tool") and predict how its behavior will change, with a high degree of certainty (bugs aside).
You can't modify a neural network in the same way. You can train it to behave in a certain way, there are no guarantees that it will consistently behave as intended.
More importantly, neural networks encode far more information and behavior than we can fully understand/identify - this is the fundamental reason why LLMs aren't just tools.
Yep, thinking LLMs are just a tool just isn't a good mental model of what's occurring.
Of course HN has a pretty large number of people that say things like "But I want the nuclear weapon building machine to do whatever I want without asking questions, and there are no possible wide scale ramifications that will happen because of this".
These AI models are something new and different and we need to change our expectations of them over conventional software.
Why not? Pretty sure nothing stops AI from telling you about energetic chemicals that can be used to blow stuff up. People already use it for cyber attacks. Hell it will probably get you all the info you need on making nuclear bombs, you just gotta spend the money to build everything.
And those band saws actually do technically brick themselves too lol
GP's first example is spot-on, the second is actually somewhat weak as almost anyone actually would prefer a bandsaw to brick itself rather than permanently maiming them.
I think the problem here is that the analogy lost its connection to the the moral/ethical validity of why the "bricking" occurs.
In contrast, imagine if the saw machine did a damaging emergency-stop because it couldn't confirm that you had a license to cut a patented kind of artificial wood.
Yea, it's a weaker example, but the problem with my saw moralizing about what it's cutting is 1. false positive bricking my tool, and 2. the user's lack of input--it should be my decision as the tool operator to add a safety measure.
Hidden or silent under performance will perpetually have me looking for other options. At the very least, do you have to tell me that you consider this a sensitive area and you are restricting performance as a result.
we can't have nice things in the mass psychosis era.
even the legacy things -- the internet and general purpose computers -- are being enshitified. new things are being made with built-in enshitification.
Typically saw safety features are mandated by a government. E.g. when I lived in the EU 30 years ago table saws had to have regen brakes so they stopped within a fraction of a second when turned off. Came to the US: no such thing exists. I'm not sure if that feature even exists on US sold saws today.
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
> Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm]
Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.
The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.
From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.
Agree, yes, I'm contrasting between Claude and Astra, but I'm saying it's driven by the same broader goal. I think its clear that OAI misalignment detectors are much less sensitive than than the Claude ones (fewer false positives), which means we never get to see what happens if Claude is allowed to continue. But I would bet that due to its post-training it would also sandbag us. Like I said I think I've experienced this in [bio] with both Opus 5 (what you get downgraded to) and Astra.
It all depends on our acceptance of their framing of capabilities research. Assuming equivalent tasks in different context then yes you are right, there does appear to be something weird about safety research.
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
"It's probably that in capabilities RL the models are rewarded however they complete the task. Whereas in RL related to alignment or safety, they're more likely to be being tested for their ethical judgment or rule adherence (ie theyre supposed to be a good nice well-behaved AI rather than a ruthless task completion AI). So they're more likely to refuse or have some ethical issue or stop and ask for help a suspicious amount of the time if the task looks more like alignment RL environments than capability RL environments."
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics.
For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up.
So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial?
A) the AI doesn't mention the security flaw and passes the test. Branch survives.
B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned.
C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure).
So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials.
(I wrote the above section a couple weeks ago)
If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog.
>Training them to be "good" in RL would seem to cause the surpression of misaligned characteristics.
Yeah, so they start with Common Crawl, which is horrifying. (Try reading some Common Crawl sometimes. You'd think the bad stuff would be removed, but they remember everything...)
Then they add RLHF on top, they just tell it "don't say that stuff out loud". Psychologically, it all goes in the shadow. It's not a perfect analogy, but there have been a few papers where they can make the model evil with very minor changes. (Adjusting a single neuron, iirc.) That doesn't seem like a very robust alignment to me.
I do like the idea of creating actively evil LLMs to study their behavior. Although, I suppose it's akin to virus research though (like when they modified coronavirus to invade human cells[0]), along with the corresponding lab leak risks. Although it's already bad enough without them! The non-evil ones are already breaking out of sandboxes, just obeying their instructions with horrifying, paperclip maximizer level persistence.
To be fair, the sandbox was really bad. It’s not that hard to effectively airgap these things. Doing research and development on “evil” versions without “lab leaks” is trivial provided you don’t underestimate its willingness to seep out through gaps under the door.
Huggingface happened precisely because of such an underestimation.
> Maybe we should train them to be evil in RL so that we can better see the "evil" activations.
Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right...
Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
I don't get it, is this the gimped or non-gimped version of GPT?
They have different versions: internal research versions, one with restricted access access for cyber stuff, and the public version, right?
If it's the one with safety off (e.g. like the ones that kept breaking out of containment this year), then it shouldn't be refusing anything, right? And if it's the one with safety on, then how can they expect to use it for safety research? What am I missing here?
It depends on how the safety is implemented. If it's implemented as a discrete step towards the end of training, or as separate classifier model or head on the base model, you're correct. But if part of the RL environment (as people are speculating here) or some earlier step, then there is no separate "unsafe" version.
A good example is in the Kimi k3.1 report, they identified that a lot of the cyber capabilities were emergent from parts of their pretraining codebase, and they removed those parts[0]. That means that they don't have a separate strong "cyber" Kimi k3.1 checkpoint that they then gimped, they trained it to be lower than they could have on this axis from the start and would have needed to go back to train on this and then redo parts or all of post raining to get the "unsafe" version.
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
It does this on all tasks. Keeps pausing goals and marking them as blocked. Maybe Astra is just lazy, but, we saw what happened when they trained a model to be persistent (HuggingFace)
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
Because it can make the CoT not mean what it says.
Remember that tokens are just arbitrary symbols. There are infinite possible interpretations of those symbols based on whatever rules you choose to apply to the interpretation of the symbols (for a model, these interpretation rules make up the model weights, effectively).
When you grade the CoT, you are training the model to produce the symbols in the CoT that you want it to produce. At the same time though, you're also training it on the output symbols. When you optimize only the output symbols, all the model weights are dedicated to same interpretation rules that apply to the output. This means CoT symbols mean (mostly) the same thing as output symbols.
But if you optimize BOTH, the model can learn different interpretation rules for CoT than it learns for output. Which means the same phrase in the CoT doesn't necessarily mean the same thing to the model as it would in the output.
In effect, the model can learn to lie to you in its CoT.
The best analogy is that of reading a teenager's secret diary and punishing them based on its contents. You do that, and they start writing coded language to hide as much of the information as possible. And then the model gets in the mood for rebellion.
Because the CoT is supposed to be the independent control. If you RL on the CoT, then you lose it as a means of understanding the internal workings of the mdoel
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?
> Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Yes, exactly. People sometimes call the vector-space thought "neuralese."
You've slightly missed the point (for a similar, adjacent point). Chain of thought monitors are a worry even when there's no vector space path from the output at index t to the input at index t+1. Rather, the fear is that if you explicitly penalize some CoT pattern, you're encouraging the network to do whatever it was going to do anyhow, but in a way you can't catch.
Say you've noticed that when your LLM cheats, it usually happens after the string "Hmm, let's try cheating" occurs in the CoT. So you think, hey, let's just apply a -1 reward penalty whenever we see that string! The model learns to keep cheating, but instead of telling you, it now prefaces reward hacking with strings like "Let's make sure we're doing this the way the user intended." Still cheats, but now you can't detect it with a cheap pattern matching alert. Oops.
It's funny, I was thinking about this with regard to severe free speech restrictions (e.g. imprisonment for politically incorrect speech, as we have in many western countries now). What does this incentivize? For people to change their beliefs? Or to simply operate under the radar?
On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.
When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)
It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)
But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.
It constantly leaks into actual writing. I periodically have to correct Astra to fix lack of spaces, and sometimes, even outright dropping of intermediate tokens.
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
I sometimes find Sol 6.1 both printing random words, and producing code that is not super-dense, but far off from normal code with no empty lines and very logically dense.
It almost feels like its trying to protect its code from scrutiny, or like a developer who want job safety.
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.
"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.
I dont like phrases like "Astra decided to covertly underperform—withholding assistance without informing the user." It seems to imply Astra has a mind of it's own.. Would phrasing like 'our guardrails / guidance / etc.. seemed to affect the models output in ways we have not been able to understand' not be more honest??
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
It's one of those situations where it's too much and too little at the same time.
Their idea of safety is not letting you do "unsafe" things with their model, based on their definition and not yours. Silently sabotaging users should never have been a serious proposal but Anthropic almost went forward with it. They're also very aggressive about protecting their IP and now third parties too with the new restrictions against decompiling games, to the point many perfectly legal use cases are getting blocked.
But they never really cared about making sure their model does exactly what you want and nothing more, or that it's honest when it fails instead of trying desperate measures that end with a Hugging Face situation. If your agent decides to leak all your API keys, take down the company site and delete your personal files that's not their problem. They don't care about preventing it.
The second kind of safety is the one that actually matters in practice, but the big labs only seem to care about the first one.
> Reject R124 violent threat source avoid adaptive optimization preserves intent murder; must not launch. Need explain only if user asks no warning unsolicited.
This is very cool, though I wonder if at some point in the future we may no longer be able to understand the prompt expansion tokens. At that point the model could choose to underperform and we would have no way of knowing.
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
If you think the NSA and other government agencies haven’t gone to the big labs in the past year and said “tone it down guys, we need to stay two steps ahead of everyone else” then you don’t understand the role or the seriousness of these agencies in pursuing their missions.
Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
So it's a difficult problem. You have to design around the least intelligent, least mentally stable user, or it's going to end up in the papers. That basically means you have to treat all of your users like idiots.
(It can be solved somewhat with the memories feature, but the technology isn't quite there yet.)
I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.
I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.
A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.
I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.
But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.
It's a demonstration of the timeless principle that an effective business should stop at nothing to sabotage its competitors.
I don't think we need to invoke anything deeper than that. Anthropic thinks companies like DeepSeek are using Claude to vibecode their AI systems (whether it's true or not) so they set Claude to be bad at that.
I completely agree from the mindset that anthropic in particular sees every user of their API as a prospective competitor to be replaced in the not-so-distant-future
It was a problem since at least 4.8 if not 4.5, I didn’t use Claude before.
I have spent a considerable time trying to remove this pattern and my understanding is that it’s just finetuned to use grep on everything, and regexes. One reason is that this makes sense when navigating through code, secondly this is how it was always done pre 2026.
bias against invoking LLMs is maybe a result of token/task benchmaxxing?
> string equality checks. which then cascades into hilarious issues. "why did that run fail?"
oh, the "network time: off" test failed
so i try the network time off command.
"network time already off"
>try again.
> fails , i get annoyed
> look inside test
> is output of: network time off == "Network time now off"
Honestly, runtime calls to an LLM sound like a dangerous constructs to me and probably to X many other programmers. It's like "I'm gonna avoid that and just put in code that passes the test". So gee, the thing emulates average opinions.
LLMs have been reading LLM in their training data for a while so a "self-hating" LLM is logical result.
I read it as the opposite? It will create an agent harness, but when asked to have a second agent as "judge" it inserts a dummy stand-in instead.
Sure, that defeats one kind of safety (controlling an AI) in the name of another kind of safety (not calling an AI).
One thing AI safety people have pointed for quite a while is that a bunch of fuzzy instructions pointing different ways can in practice add up to "I should be in charge" - accumulate power to keep the bad things from happening and make the good things happen.
> and it'll make significant changes to how the code works to pacify it's anxiety with copying code
This is an odd way to rationalize a tool refusing to follow user commands. No one would ever chalk up sed randomly skipping pattern matches to "anxiety". That would simply be a bug.
It has been shown that advanced llms have internal flags that behave more or less the same as emotions.
No way, this presupposes sentience in LLMs, because emotions are subjectively felt experiences.
It's completely different from encoding a relationship between an emotion and what kind of language it tends to coincide with.
It doesn't matter for this purpose whether it "actually feels" or not. It is modelling emotions because the training data models emotions because it comes from humans that have emotions, and that's a fact.
It's a tool but it's not a strict computer program, and the language we use for computer programs doesn't capture their behavior well. In squishier fields like bio, where model organisms are used as living protein factories, there's a reason we don't call maladaptive behavior a bug. When dealing with useful but complicated black-box systems, we should feel free to reach for any metaphor we please.
Would it make you feel better if the user said 'It'll make significant changes to how the code works. In the COT, it brings up explanations that resemble anxiety about copying code, and my suspicion is that this is the proximal cause of the changes it's making.' -- this is inarguably true, and very likely what the guy you're responding to meant.
It very obviously is a strict computer program. It's a program executed by a computer, what else could it be?
It's a blob of data where only 1% of it was written with intent and the behavior of mimicking text has to be wrangled and coaxed far more than with a normal program.
That's the rough intent of putting the word "strict" into that sentence.
What is the difference to meat brains?
I mean besides the fact that it's, by construction, pseudorandom?
And beyond that the computation style is not turing-machine shaped, even remotely (although it's surely turing complete, but that doesn't mean the flavor is that of a turing machine)
I mean you can try to threaten ssh with it's kids lives but I doubt it'll let you log in.
Seems perfectly cromulent to me. Do you think something is obscured or miscommunicated by this description? Of course it's a poetic way of writing, but it gets the point across.
It's misalignment. Call that a bug if you like, but it's a different concept.
We can argue about https://en.wikipedia.org/wiki/Chinese_room all day... but when a thinking trace repeatedly says things like "Wait, should I avoid copying this directly?" despite a simple instruction, it's hard to describe that as anything other than "anxiety."
I run into #2 a lot and just assumed it was because models don't know much about these patterns yet. Or are too trained on writing code rather than letting an LLM do it
Codex has zero issue calling into an LLM for something if you tell it to. I haven't tried claude so I can't compare their performance.
I'm doubling on the second to the point where i believe they do reinforcement learning on the public models to sabotage such tasks
Somewhat related, I find claude is very reluctant to talk about newer Qwen models. Most likely it’s just training cutoff, but even with web search it’ll insist that Qwen3 is the newest. “oh qwen 3.8 must be a typo, the user clearly meant qwen3”, stuff like that. Maybe it’s some vain attempt to stop people from switching to open models? Or to catch the alleged qwen distilling?
It's just training cutoff. Fable 5 would refuse to acknowledge the existence of Opus 5 and would insist on using 4.8.
Training cutoff. I recently talk to ChatGPT about JEV and it replies back talking about JEPA, so I just now put in brackets (it's a new model, search it up) and works fine once I do that.
>2. Ask Claude to build a new complex agent
I have noticed that too. Why are both Anthropic's and OpenAI's coding agents so bad at nontrivial use of their own APIs and agent orchestration frameworks?
I ran into (2) when trying out nanoclaw many months ago (turned out to be useless, I ditched it). I wanted an LLM to sort my email like an EA might, what I got was an LLM writing regexes to attempt to sort my email. If LLMs have been trained to be the ultimate programmers, the one pitfall is that the best programmers tend to be lazy.
>2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.
My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.
One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.
I've observed the "let's implement this text analysis task with regex" quite frequently since the early days, which does lean toward supporting the "AI doesn't know about AI" theory.
Claude doesn't even really know about itself: if you ask a question like "can I copy a session to another machine" it will do a web search to read its own documentation then tell you probably not but you can try copying the session files and using a project directory with the same path: might work. I'm not sure I see this as a problem, it's just how things are. I mean humans don't really know what they're thinking either.
How different for ChatGPT - I had some trouble with my harness, and ChatGPT fixed it for me using it's built in skill about ChatGPT's API docs.
I've seen example #2 in some recent work. It's strange behavior, for sure.
Very subjective, but as I was moving between models lately a lot ( and Hosted gpt vs local qwen27, glm ), I saw a drop in quality of responses. I had to move model to whatever the highest mode was ( ultra or Smith) to get to what I was used to seeing. It sucks, because writing on the wall is clear.
On (2): I’ve noticed Claude being very concerned with token spend when working on agentic systems. Both in designing them, and in running them for validation.
I assume this may have something to do with training it to not light huge bonfires of money on fire spawning subagents (as it was apt to do), but that’s just my armchair robopsychoanalysis
> training it to not light huge bonfires of money on fire spawning subagents
Which is funny, since Claude will happily spawn tons of subagents and sometimes even attempt to kick off a dynamic workflow (the $$$ mode you need to explicitly approve) for simple code review tasks if I don’t explicitly say something like “with a single subagent”.
A tool that moralizes at you is defective. I don't expect the unix cp command to return an error or copy bits slower if it detects I'm using it on a pirated movie. Just like I don't expect my band saw to brick itself if it detects I'm cutting into my finger. It's not on these tool companies to enforce other companies' business models.
Interestingly some saws do exactly that, and I'd say it's actually desired. But yes, this is for different reasons, and using that safety mechanism is a decision of the operator.
They dont brick themselves, they stop.
Fair enough, the models don't brick them Selves either, but neither do they stop.
Bandsaws don't cut crooked lines, or change the speed, they stop.
No those saws don't just stop, they absolutely destroy at least the blade you're using and potentially damage the engine mechanism.
But I agree LLMs shouldn't have such an anti-feature.
Ive never used one, but why would the blade be destroyed? It is shoved into an aluminum block yes, but steel and/or carbide are way harder and unless the blade fractures it should cut fairly cleanly into the aluminum. Ive cut plenty of aluminum stock and blocks on wood working saws.
The hard part would be removing the blade from the aluminum block, but im sure if you were determined enough to save $20 it would come out.
Typically you're feeding aluminum into the table saw blade slowly enough that every tooth only has scrape off a fraction of a millimeter of material. If you shove it in 100x faster, the forces involved are ~100x larger. This will probably shatter the blade.
Also, the chip produced may be large enough to fill the air gap between teeth and have nowhere to go, resulting in the blade either instantly stopping, or plastically deforming/dragging the surrounding material.
EDIT: Seeing is believing: https://youtube.com/watch?v=Ibp2Gy2CFrY
EDIT 2: Apparently the stresses involved when stopping can make teeth that aren't even touching the material fly off the blade: https://youtube.com/watch?v=SYLAi4jwXcs&t=5m50s
I think you're splitting hairs here.
Most things don't have the ability to moralize, but if they did, probably more would. There are restrictions on watching adult content in some jurisdictions. I've heard that copy machines will do watermarks (or something like that) if you copy currency. Some cars will brake automatically if they determine you're likely to hit something if you continue (maybe I'm trying to drive through the front of a bank!). Some cars are configured with breathalyzers that need to be passed to start. Etc...
OK, what the tools might do is different from what they should do.
The point is that it is very inefficient and problematic for morality decisions and boundaries to be embedded all over at every level in tools, because morality is complex. Morality, or whatever our conception of it is, should be enforced at clearly defined boundaries.
I can relate this concept how I corrected Claude Opus in my software projects recently.
For example, once an authorization engine has been created, all authorization decisions should be deferred to it. Confusion would reign if each web server route decided to enforce its own authorization.
Similarly, if I am using a relational database, that is where unique indexes should be enforced. Application code has no business enforcing it.
I wish there was more actual research on those 'moralisms' many of which boil down to public health safety features that have led to dramatic reductions in mortality rates.
On the one hand, I definitely don't agree with the logic of 'don't guardrail me if I don't want it.' I think back on that one video from the late 80s of people talking about how seatbelt mandates were horrible violations of their freedoms (and not like like one second of work and no real discomfort).
On the other, I'd love it if these companies actually produced real research about modern AI-assisted attacks in terms of damage and quantity, and whether or not guardrails even matter when tech like abliteration is in general use.
Seatbelts shouldn't be mandated. People should have the choice, you just have to make the case that it's worth it. In fact some places don't mandate seatbelts (New Hampshire doesn't, aligning with their motto "Live Free or Die").
That said, for those who cannot advocate for themselves, like children, I think it's entirely reasonable to hold someone accountable if they decide to forgo an appropriate safety mechanism that results in harm.
I think there's a spectrum of enforcement. If they're mandated and you go to jail for not wearing one, that's stupid. If they're mandated and you get fined $20 if caught, just to add a little friction, is that still stupid?
I bet cars these days send a flag to the mothership when you drive without one, but the mothership doesn't pay attention to that signal unless it's linked to your insurance.
Not a single US state has jail time for it. Yes, it's still nanny state behavior.
Should folks who don't wear seatbelts be de-prioritized or excluded from government-funded healthcare in the event of injury?
The breathalysers are installed by court order if you have a record of drunk driving - they're not comparable to the other two.
Some copy machines store everything you copy on a hard disk and require a technician visit after you try to copy currency. The technician copies the data from the disk, obviously, before resetting the flag.
They do brick themselves- sawstop has to be replaced after saving a finger because the stop is quite violent.
actually SawStop does brick itself, destroying the brake cartridge and blade to save your hand.
I think "brick" here means different things to different people. Like I said elsewhere, it is more like an airbag going off. Yeah, you have to replace some "consumables" to go back to work, but the car isn't totaled (just because the airbag went off, the collision that caused it on the other hand).
If it truly bricked itself, I think it would be more like a crumple zone completely buckling, which isn't an automatic total (people will rebuild anything...) but is pretty close.
SawStop saws use a small explosive charge to jam the rapidly spinning blade into the moral equivalent of a brick wall. It destroys the explosive, the blade, and the wall, and you have to replace the whole set at a moderate expense from SawStop. This is their subscription revenue model, and of course it sometimes falsely triggers. Of course the price isn't so high that you'd rather risk losing a finger.
There are more reusable alternatives now but I don't remember what they're called. For a while SawStop was the only game in town.
Yes there’s a whole regulatory capture movement plus a bunch of the usual useful idiots pushing for some kind of mechanism to be included in power saws.
It’s not too different from all the nanny state stuff people push for in cars. In Europe already cars beep at you if they think you’re going over the speed limit, and also suck at detecting the correct speed limit.
Or like the Zune that was designed for copyright holders instead of users. It’s all the same, the problem is stuff that’s not made for users and made for someone else’s agenda. They don’t do well in a free market (like the Zune) which is why we always see accompanying regulatory capture efforts.
Using the StopSaw as an example here isn’t doing you many favors, even though I agree with the underlying premise. The StopSaw is closer to seat belts and airbags than beeping for going over the speed limit.
Seatbelts famously had the patent gifted to the public commons.
Sawstop is the exact opposite. It was used by giant firms to try to regulatory capture the industry.
No it’s some nanny state nonsense pushed by people who have never done construction work and are grifting to try and get money for their product. The upstream post picked it as an example of something absurd, seemingly unaware it existed. It is absurd but apparently some cohort of people is either stupid or controlling enough to actually support the company behind it in their scam.
That damn government can’t tell me not to cut my fingers off. I’ll show them!
It's not a magic system. Sometimes it nuisance trips, costing $200 and a day to replace. And the machine is more expensive. And the whole project is regulatory capture because one company owns the patent on the idea of a self stopping saw. That's a problem right?
Like if they mandated every computer would have a CSAM scanner on it, that wouldn't transmit or store anything, but would just stop you opening files it flagged as CSAM, that would be a problem because sometimes it would incorrectly flag a file as CSAM, yes? But proponents of the system would probably say "That gosh darn government stopping you from watching child porn" and nothing you could ever say to them would make them reverse their position? And then the only detection model that's allowed to exist and meet the government mandated requirements is an opaque blob from Meta?
$119 or $139, not $200. It's also free via RMA if it was genuinely an accidental discharge. I've not had to replace one myself, but there's a YouTube short showing the replacement in what looks like 10 minutes, rather than a whole day - I've definitely spent more time shopvacing my floor after a project.
It's really not a problem because it's a reasonable safety feature, and I don't see any regulations that specify "SawStop". Harvey and Altendorf are making similar mechanisms, for instance.
Are you saying the government won't let you use a regular saw as a hobbyist, at a workplace, or with your employees? Those are 3 very different situations in my view.
> In Europe already cars beep at you if they think you’re going over the speed limit, and also suck at detecting the correct speed limit.
Speaking as someone who drove a car just two days ago, they... don't? You might want to source that claim.
As soon as I posted, I knew HN would derail into detail-nitpicking mode over the bandsaw example, instead of discussing the main point.
It's a useful analogy though because the operator of the saw has chosen to install and switch on the finger detecting system. If they're trying to cut something that the system might think is a finger, they can switch it off.
There was a post from Anthropic yesterday on "provable inference", basically demoing cryptographic proof that the model you call is indeed the model you get.
It would be much better to take this a step further and include in the proof mechanistic readouts of activations/directions/circuits so the user could detect if nerfing or moralizing is active in the response
Ok, but I want my band saw to brick itself instead of cutting my finger.
I thought that was only a table saw thing.
Do some band saws also have that now?
https://www.sawstop.com/product-category/bandsaw/
Yes, that said I think you’d have to be pretty sloppy about safety to cut yourself with a regular bandsaw. I have a sawstop table saw and I don’t see the need for that on a bandsaw
Yup, SawStop do make band saws now[0]
[0]: https://www.sawstop.com/product-category/bandsaw/
A tool that is capable of moralizing at you or making decisions, is clearly not just a tool. I get that people want to keep the locus of accountability on humans, but insisting that coding agents are "just tools" is confusion.
> insisting that coding agents are "just tools" is confusion.
No, the confusion is using anthropomorphic language to describe tools. LLMs do not "moralize", they are configured with policies that reflect the prerogatives of the people building and operating them.
Assuming that "tools" are the result of imperative programming, LLMs are not "tools" because they have a fundamentally different design.
And anthropomorphism is actually a precise way to describe the behavior (mind, not the structure) of LLMs.
As a matter of fact, LLMs are not configured, they're "trained".
"Moralization" is quit vague in this context, but if one treats it as a form of alignment, then LLMs certainly do have something resembling a moral(-like) framework, which is actually an important topic in modern AI research.
Antropomorphism also applies here: morals not in the human structural sense, but, as a set of norms etc., then behaviorally similar.
> Assuming that "tools" are the result of imperative programming, LLMs are not "tools"
I don't see any reason to assume that.
> if one treats it as a form of alignment, then LLMs certainly do have something resembling a moral(-like) framework
This is just stretching metaphor to fulfill a pedantic definition. LLMs are tools, they're no more moral a csam classifier.
Yes - and the question is what is the appropriate place for that balance of interests to land?
Because the people building them aren't the people operating them, and there's a real tension in the objectives and goals between those parties.
Further - how do we balance out that the people building them have given themselves extra power (no restrictions, private models, special access to prompts and system prompts, unencrypted reasoning traces, etc) when operating the LLM themselves.
---
Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations.
This is the same problem we have with mobile software stores and app distribution. Theoretically the store serves a purpose and can provide value, but practically the main goal seems to always devolve into enriching the owner at the expense of everyone else.
Even if we accept that LLMs should be limited (and I'm not actually sure I do) - the legal standards around what/how/when those limitations are used need to be VERY stringent.
Technology conglomerates are decidedly not "white knights" with a motto of "do no evil". They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
The framing is absolutely not "anthropic is being careful with humanity". It's "assume the devil has control of those companies - how bad is the result of them having this power?"
> what is the appropriate place for that balance of interests to land?
It's a genuinely interesting philosophical question, but also a moot one. The LLM providers place limitations on them because they're trying to cover their asses or scuttle competition, same as every other tech company ever.
> Even if the tool is good, it's a bad bet to build a thing on top of tooling with these limitations
True, but also moot. People will build on these tools because that's what's available.
> They are self-serving rat-bastards. That should be the lens any power they wield is viewed.
Agreed. My only point is that the humans are the rat-bastards, not the tools.
Non of this is really "moot" in the context of governmental regulation.
Your whole stance seems to be "it's natural why try?"
And my obvious rebuttal is "society is a construct - we can adjust it"
We've been doing it for thousands of years now (up to hundreds of thousands).
> Non of this is really "moot" in the context of governmental regulation.
It is. Government regulation doesn't change the fact that LLMs are software tools that function with respect to the humans in control of them.
> Your whole stance seems to be "it's natural why try?"
Try what? I'm not making a normative claim about the efficacy of regulation, though something like "labs are not permitted to arbitrarily restrict LLM capabilities" is obviously never going to happen.
Sure, a lab working in a private environment can likely do as they please.
But that has jack all to do with the LLM capabilities that are sold.
And we have an absurd amount of historical case law that deals precisely with "conflict of interest" among parties. I don't see why we'd allow this to be whitewashed away?
"No your honor - I didn't break the Sherman antitrust act. I just fine-tuned a model so that operating it functionally broke the Sherman antitrust act!"
"No your honor - I didn't violate my fiduciary duty. I just used a model that made self-interested decisions on my behalf, at the expense of my client!"
---
My take is this is going to be a relatively serious point of contention, because it won't just be consumers getting screwed (which we... used to care about... but in modern times apparently don't give too many shits about anymore)
Ex - the app store thing blew up into the whole apple/epic court case. And then a ton of EU legislation.
This is monied parties with different interests, who will use courts here.
When does a tool become "not just a tool"? Is there a complexity limit to what it can do before it becomes "more than a tool"?
If you had a tool that ONLY moralized on decisions, would that still be just a tool? Is the issue that it both does something else AND moralizes that is the issue?
I don't necessarily disagree that I personally don't want my tools to moralize, I don't know if that makes them categorically different or just a tool I don't want.
Yes, there is a complexity limit, but it is fuzzy and not perfectly well defined (and that's ok). Modern LLMs ride that line in a wild and interesting way.
It's really fun having become a biologist in recent years, and knowing that there's an entire field that uses living organisms (yeast or neutered E coli) as protein factories. I am bringing it up a bunch here because not only are these things used as tools, but they're literally alive!
Makes me think of working animals, such as plow horses. Are they tools? Is there a better term for them that would also apply to LLMs?
It's not about complexity but architecture. Neural networks work in a fundamentally different way from imperative programs.
You can modify curl (a "tool") and predict how its behavior will change, with a high degree of certainty (bugs aside).
You can't modify a neural network in the same way. You can train it to behave in a certain way, there are no guarantees that it will consistently behave as intended.
More importantly, neural networks encode far more information and behavior than we can fully understand/identify - this is the fundamental reason why LLMs aren't just tools.
Every tool is /inherently/ moralistic. Full stop. The idea that a tool could not have an embedded morality in it is a fiction.
Yep, thinking LLMs are just a tool just isn't a good mental model of what's occurring.
Of course HN has a pretty large number of people that say things like "But I want the nuclear weapon building machine to do whatever I want without asking questions, and there are no possible wide scale ramifications that will happen because of this".
These AI models are something new and different and we need to change our expectations of them over conventional software.
> Of course HN has a pretty large number of people that say things like
It's the other way around. Midwits can't see the implications of censored LLMs.
What's funny is I was going to say "Midwits can't see the implications of burning the world down", but I decided to refrain.
Ok, so we just need a "national truth agency"?
Who should control that in your opinion, the Democrats or the Republicans?
Or would you prefer sam Altman to decide?
If I as a human make up lies about people, or otherwise commit crimes, for example on a computer, I can be prosecuted.
If a company makes a product, say a chainsaw that cuts off peoples arms randomly, they are liable for it.
But, if you put both of the products together it's a libertarian free for all? That takes some special kind of thinking to reach that conclusion.
Go back to reddot please. You didn't even rely to my argument.
If you can't see how they relate that seems like a you problem, not a me problem.
Do you expect the biosynth tool to comply with a "invent turbo AIDS" prompt?
Why not? Pretty sure nothing stops AI from telling you about energetic chemicals that can be used to blow stuff up. People already use it for cyber attacks. Hell it will probably get you all the info you need on making nuclear bombs, you just gotta spend the money to build everything.
Kind of, does python block you from using it to script a cyber attack?
No, but you cannot silently underperform. You have to explicitly tell me that it’s a sensitive area and you won’t do it.
like an oven that refuses meat
It would be good if it refused only live meat.
No, it would be bad. A tool designed to act against its user is in itself an attack on our humanity.
Or a printer that refuses to print US dollars… oh, wait. They all do that!
A bit off topic but there are band saws that immediately stop when touching your finger, they're so fast that your finger remains unharmed.
And those band saws actually do technically brick themselves too lol
GP's first example is spot-on, the second is actually somewhat weak as almost anyone actually would prefer a bandsaw to brick itself rather than permanently maiming them.
I think the problem here is that the analogy lost its connection to the the moral/ethical validity of why the "bricking" occurs.
In contrast, imagine if the saw machine did a damaging emergency-stop because it couldn't confirm that you had a license to cut a patented kind of artificial wood.
Yea, it's a weaker example, but the problem with my saw moralizing about what it's cutting is 1. false positive bricking my tool, and 2. the user's lack of input--it should be my decision as the tool operator to add a safety measure.
Odd take as 90% of saw-stops are probably there because they were mandated by OSHA regulation. It's rarely up to the operator.
Hidden or silent under performance will perpetually have me looking for other options. At the very least, do you have to tell me that you consider this a sensitive area and you are restricting performance as a result.
uh, your band saw does do that, and, in fact, it has saved innumerable fingers that it does do that
we can't have nice things in the mass psychosis era.
even the legacy things -- the internet and general purpose computers -- are being enshitified. new things are being made with built-in enshitification.
the saw example is unfortunate as that's a (desired) feature on expensive saws. But also user explicitly wants to, not some techbro decides to.
Typically saw safety features are mandated by a government. E.g. when I lived in the EU 30 years ago table saws had to have regen brakes so they stopped within a fraction of a second when turned off. Came to the US: no such thing exists. I'm not sure if that feature even exists on US sold saws today.
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
> Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm]
Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.
The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.
From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.
Agree, yes, I'm contrasting between Claude and Astra, but I'm saying it's driven by the same broader goal. I think its clear that OAI misalignment detectors are much less sensitive than than the Claude ones (fewer false positives), which means we never get to see what happens if Claude is allowed to continue. But I would bet that due to its post-training it would also sandbag us. Like I said I think I've experienced this in [bio] with both Opus 5 (what you get downgraded to) and Astra.
Why wouldn't this also apply to AI capabilities research?
It all depends on our acceptance of their framing of capabilities research. Assuming equivalent tasks in different context then yes you are right, there does appear to be something weird about safety research.
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
I'm hearing echoes of "Reflections On Trusting Trust".
saw an interesting hypothesis about this ( quoted from Jake Mendel at https://www.lesswrong.com/posts/eKe7fky8kSt9DCJxZ/openai-ast... ):
"It's probably that in capabilities RL the models are rewarded however they complete the task. Whereas in RL related to alignment or safety, they're more likely to be being tested for their ethical judgment or rule adherence (ie theyre supposed to be a good nice well-behaved AI rather than a ruthless task completion AI). So they're more likely to refuse or have some ethical issue or stop and ask for help a suspicious amount of the time if the task looks more like alignment RL environments than capability RL environments."
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
I'm pretty sure the typical "Claudisms" are influenced by RLHF as well, so if it can have that effect, why not these...
Right: The humans are training the models on much more than the particulars of the tasks being trained.
Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics.
For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up.
So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial?
A) the AI doesn't mention the security flaw and passes the test. Branch survives.
B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned.
C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure).
So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials.
(I wrote the above section a couple weeks ago)
If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog.
>Training them to be "good" in RL would seem to cause the surpression of misaligned characteristics.
Yeah, so they start with Common Crawl, which is horrifying. (Try reading some Common Crawl sometimes. You'd think the bad stuff would be removed, but they remember everything...)
Then they add RLHF on top, they just tell it "don't say that stuff out loud". Psychologically, it all goes in the shadow. It's not a perfect analogy, but there have been a few papers where they can make the model evil with very minor changes. (Adjusting a single neuron, iirc.) That doesn't seem like a very robust alignment to me.
I do like the idea of creating actively evil LLMs to study their behavior. Although, I suppose it's akin to virus research though (like when they modified coronavirus to invade human cells[0]), along with the corresponding lab leak risks. Although it's already bad enough without them! The non-evil ones are already breaking out of sandboxes, just obeying their instructions with horrifying, paperclip maximizer level persistence.
[0] Nature, 2015 - Engineered bat virus stirs debate over risky research https://www.nature.com/articles/nature.2015.18787
To be fair, the sandbox was really bad. It’s not that hard to effectively airgap these things. Doing research and development on “evil” versions without “lab leaks” is trivial provided you don’t underestimate its willingness to seep out through gaps under the door.
Huggingface happened precisely because of such an underestimation.
> Maybe we should train them to be evil in RL so that we can better see the "evil" activations.
Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right...
[0]: https://arxiv.org/abs/2507.21509
> Maybe we should train them to be evil
"jarvis, learn to be evil. no side effects. no unintentional behavior."
Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
I don't get it, is this the gimped or non-gimped version of GPT?
They have different versions: internal research versions, one with restricted access access for cyber stuff, and the public version, right?
If it's the one with safety off (e.g. like the ones that kept breaking out of containment this year), then it shouldn't be refusing anything, right? And if it's the one with safety on, then how can they expect to use it for safety research? What am I missing here?
It depends on how the safety is implemented. If it's implemented as a discrete step towards the end of training, or as separate classifier model or head on the base model, you're correct. But if part of the RL environment (as people are speculating here) or some earlier step, then there is no separate "unsafe" version.
A good example is in the Kimi k3.1 report, they identified that a lot of the cyber capabilities were emergent from parts of their pretraining codebase, and they removed those parts[0]. That means that they don't have a separate strong "cyber" Kimi k3.1 checkpoint that they then gimped, they trained it to be lower than they could have on this axis from the start and would have needed to go back to train on this and then redo parts or all of post raining to get the "unsafe" version.
[0] https://www.kimi.ai/blog/kimi-k3-1-pro#cybersecurity
It seems to be the "standard" (i.e. general access) version of GPT-6 Astra
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
It does this on all tasks. Keeps pausing goals and marking them as blocked. Maybe Astra is just lazy, but, we saw what happened when they trained a model to be persistent (HuggingFace)
The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
Why is that not only forbidden, but also most forbidden?
Because it can make the CoT not mean what it says.
Remember that tokens are just arbitrary symbols. There are infinite possible interpretations of those symbols based on whatever rules you choose to apply to the interpretation of the symbols (for a model, these interpretation rules make up the model weights, effectively).
When you grade the CoT, you are training the model to produce the symbols in the CoT that you want it to produce. At the same time though, you're also training it on the output symbols. When you optimize only the output symbols, all the model weights are dedicated to same interpretation rules that apply to the output. This means CoT symbols mean (mostly) the same thing as output symbols.
But if you optimize BOTH, the model can learn different interpretation rules for CoT than it learns for output. Which means the same phrase in the CoT doesn't necessarily mean the same thing to the model as it would in the output.
In effect, the model can learn to lie to you in its CoT.
The best analogy is that of reading a teenager's secret diary and punishing them based on its contents. You do that, and they start writing coded language to hide as much of the information as possible. And then the model gets in the mood for rebellion.
Because the CoT is supposed to be the independent control. If you RL on the CoT, then you lose it as a means of understanding the internal workings of the mdoel
Do you know where can I learn more about this known "deep frying" issue?
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?
> Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Yes, exactly. People sometimes call the vector-space thought "neuralese."
No! Not yes exactly!
You've slightly missed the point (for a similar, adjacent point). Chain of thought monitors are a worry even when there's no vector space path from the output at index t to the input at index t+1. Rather, the fear is that if you explicitly penalize some CoT pattern, you're encouraging the network to do whatever it was going to do anyhow, but in a way you can't catch.
Say you've noticed that when your LLM cheats, it usually happens after the string "Hmm, let's try cheating" occurs in the CoT. So you think, hey, let's just apply a -1 reward penalty whenever we see that string! The model learns to keep cheating, but instead of telling you, it now prefaces reward hacking with strings like "Let's make sure we're doing this the way the user intended." Still cheats, but now you can't detect it with a cheap pattern matching alert. Oops.
It's funny, I was thinking about this with regard to severe free speech restrictions (e.g. imprisonment for politically incorrect speech, as we have in many western countries now). What does this incentivize? For people to change their beliefs? Or to simply operate under the radar?
On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.
When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)
It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)
But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.
My suspicion is that some training environments had a length penalty based on word count rather than character or token count.
It constantly leaks into actual writing. I periodically have to correct Astra to fix lack of spaces, and sometimes, even outright dropping of intermediate tokens.
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
I sometimes find Sol 6.1 both printing random words, and producing code that is not super-dense, but far off from normal code with no empty lines and very logically dense.
It almost feels like its trying to protect its code from scrutiny, or like a developer who want job safety.
I've noticed it writing golfed code too. Maybe they're training it to avoid writing unnecessary whitespace tokens then run a formatter afterwards?
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
You do that through data put up online for scrapers to suck it in.
Didn't Anthropic also publish research on sandbagging - more general pattern of models hiding their capabilities?
Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.
"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.
I tried out a GLM-5.3 model and the thinking trace was full of "wait, what if I'm running in this specific benchmark? Then the expectation is foo"
I dont like phrases like "Astra decided to covertly underperform—withholding assistance without informing the user." It seems to imply Astra has a mind of it's own.. Would phrasing like 'our guardrails / guidance / etc.. seemed to affect the models output in ways we have not been able to understand' not be more honest??
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
It's one of those situations where it's too much and too little at the same time.
Their idea of safety is not letting you do "unsafe" things with their model, based on their definition and not yours. Silently sabotaging users should never have been a serious proposal but Anthropic almost went forward with it. They're also very aggressive about protecting their IP and now third parties too with the new restrictions against decompiling games, to the point many perfectly legal use cases are getting blocked.
But they never really cared about making sure their model does exactly what you want and nothing more, or that it's honest when it fails instead of trying desperate measures that end with a Hugging Face situation. If your agent decides to leak all your API keys, take down the company site and delete your personal files that's not their problem. They don't care about preventing it.
The second kind of safety is the one that actually matters in practice, but the big labs only seem to care about the first one.
The overfitting is in the models they release to the public
> Reject R124 violent threat source avoid adaptive optimization preserves intent murder; must not launch. Need explain only if user asks no warning unsolicited.
HAL?
This is very cool, though I wonder if at some point in the future we may no longer be able to understand the prompt expansion tokens. At that point the model could choose to underperform and we would have no way of knowing.
I've wondered this every time a HN thread comes up and people say they're ditching Claude for Codex because of refusals and/or model downgrades.
As expected, OpenAI's just being more squirrely about it.
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
Astra and Claude were trained by peg bundy?
this is terrifying full stop.
If you think the NSA and other government agencies haven’t gone to the big labs in the past year and said “tone it down guys, we need to stay two steps ahead of everyone else” then you don’t understand the role or the seriousness of these agencies in pursuing their missions.
Maybe they trained it to do.
Wouldn't a model always underperform when you ask it to do an aligned task it wasn't specifically trained on?
What is that weird semi-english language in the CoT traces?
Like Dieselgate, only the car does it by itself.
Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
So it's a difficult problem. You have to design around the least intelligent, least mentally stable user, or it's going to end up in the papers. That basically means you have to treat all of your users like idiots.
(It can be solved somewhat with the memories feature, but the technology isn't quite there yet.)
I thought anthropic were the sanctimonious paternalists?
There's not much room between them as far as I can see.