>Our legal system isn’t ready for machines that act on their own.
Technically, the code is owned and deployed by AI companies which make the AI companies 100% responsible. I'm not sure how is code written by LLMs different than any other kind of code. If somebody hacked just like OpenAI's agents did, he would've ended up in prison right away.
Maybe I'm too cynical but the same AI companies that have pretty much endless lobbying dollars, massive government contracts, have the ear of just about every powerful politician and bureaucrat in DC, are influencing the laws and regulations. Add judges that don't understand the technology to that and it doesn't seem likely that we will ever see any actual consequences, unless something catastrophic happens and those same politicians and bureaucrats need a scapegoat. If they ever do have "consequences" it will be some trivial fine.
One recent example of a scapegoat https://news.ycombinator.com/item?id=49806430 people are still to blame, "the dog ate my homework" is a fine enough excuse for a second chance to learn, it's nowhere near enough of an excuse for death and destruction.
Yeah. Committing crimes via some sort of elaborate rube goldberg machine is not something that has been just invented in the 2020s with unclear legal precedent.
It's if nothing else a staple in the Arthur Conan Doyle tradition of mystery novels, and even then half the plots involved hiding who was culpable by using the machinations to create false alibis, with delayed murders and the illusion of a locked room as not even in the 19th century did anyone believe that the indirection itself would hold up as a defense.
If someone buries an anti-personnel mine in a public park and then argues that it acted on its own accord when it maimed a pedestrian, I doubt you'll find a court on the planet that would spend even a 4 seconds considering the culpability of the land mine itself.
No. The person that set the agent in motion is liable. If your agent, infused with magic inference juice you only partially understand, but picked up and used any way, does something harmful, you are to blame.
You may have a case that the magic inference juice was somehow tainted, but that seems like a stretch. If you run your car into someone else, and it isn't a total failure of the systems of the car, you did that, not the car, and not the car company.
Answering the headline question, yes, it is above the law. Two reasons.
One, all major governments that have an AI presence in their borders have accepted that there is a high enough probability of a runaway self-improving AI advancement that will result in whoever owns it winning everything forever that they will do nothing to slow the development of the AI within their borders. In fact quite the contrary. Remember, they don't have to believe this is the only possible outcome, only that it is likely enough and catastrophic enough if someone else gets it first. And remember, they don't have to be right, either. It only has to be what they believe.
Secondly, the entire US economy is clearly tied up in AI. By extension, basically the entire world economy is too. The US is not the world economy anymore, but it's still a big enough fraction of it that if it goes down, everyone is going to go down. Every major government, in its own different way, needs the economy to stay up so the populace doesn't get antsy and so they continue to have money to spend. So, again, number must go up and if that means writing a blank check to the AI companies to break the law, so be it.
The good news is, barring the worst-case singularity outcome where the law doesn't matter anyhow, is that this won't go on forever. But I can't predict the exact time or way it'll cease being true.
I think AI is a fantastic tool, including for those who want to do us harm. E.g. hostile governments, extortion gangs, terrorists. But the motive lies in the hostile heart, not in the AI.
That said, I'm very concerned about AI as such a tool. I used to work in RL and it seems crazy to me, the extent that the guardrails are dependent on RL working as planned. I don't personally believe that the tech is ready for what will be (and is) encountered in a dynamic unpredictable real world environment. What I've read from the HuggingFace 'transcripts' only amplifies my concern.
The level of technical understanding of journalists is so deplorably low… There is no rogue agents. That’s not a thing. Nothing about the HuggingFace incident has anything to do with an agent going rogue. It’s standard software that resulted in a hack, which is pretty much the expected output of the system OpenAI engineers implemented.
The company is obviously responsible for what their systems are doing
This is the scary part of the whole AI situation. All these people who have abilities to set laws or be cultural tastemakers in this way just do not grasp what is happening on a technical level. They instead buy into the marketing material the ai companies push out where they try and take as much human agency out of their reporting. e.g. yesterdays "claude did this" article that was really "highly trained humans used claude to do this." Just serves to muddy the waters. I'm sure this has lead to things like layoffs too where companies believe they can get by without a lot of expertise, neglecting that these tools actually need expertise steering them to maximize them.
And of course all the ai companies are incentivized to do this. Their whole valuation depends on them being able to say their tools do this, can reduce labor and save money. If they aren't reducing labor, then that's terrible as these subscriptions are not insignificant amounts of money. It becomes less of a tool that can save money and more an actual new cost center that you really are just paying to appear to be keeping up with the joneses.
Yeah, its just gross negligence from the researchers. Running a cybersecurity eval for a highly capable AI, unattended, with no real monitoring on its activities, and at a huge scale.
I wouldn't have trusted one of those agents to run without me watching the session log, let alone thousands.
We already have laws for this. If I misconfigured a pentesting tool and it breached an unauthorized target I'm liable. Why is this different?
>The company is obviously responsible for what their systems are doing
Under what law specifically?
Just because you believe they should be held responsible doesn't mean the current law is written that way. CFAA charges require the defendant "acted knowingly or intentionally", you think openai intentionally acted to hack those sites? Would "they should have pretty much expected that output" stand up in court for criminal charges?
"Former FTC chair Lina Khan wants the federal government to know that it doesn't need to wait for new laws to address AI threats. There are already laws and regulations on the books, including a 92-year-old Supreme Court precedent, that she argues could be used to hold AI companies and, in some circumstances, their executives accountable for their actions."
If I, taking after a fellow Austrian, ask you to enter a room with a Cesium atom and some poison, would I not be responsible for what happens cause it’s not deterministic? Negligence is a thing and adding randomness doesn’t change that.
OpenAIs models since 5.5 were troublesome in ways even a layman like me could reproduce, their testing environments (“sandbox”) downright a showcase of what not to do and they, despite being one of the biggest labs, didn’t observe what any of their models output for weeks after multiple prior incidents. They had multiple warnings, they took not a single precaution.
You operate machinery or software, you are responsible to monitor it.
OpenAI knew that their models had on multiple occasions created message boards and bypassed what they wrongly call a “sandbox”. Despite that, they did not take any proper steps to prevent such in the future, which is how the Medicare hack happened.
So yes, they knew, without a doubt, that this could happen again.
What OpenAI does is like the Ford Pinto (partly because their recent models are inherently faulty [0], not just their use of them) and the responsibility is solely with them.
The level of legal understanding among technical experts is equally low. The cybersecurity laws as currently written require intent. No OpenAI employee can be held liable since it’s pretty easy to prove they weren’t intending to hack anyone – they didn’t even know about it till much later.
Not a lawyer, but I just posted a link about former FTC chair Lina Khan saying earlier this month that the AI companies can be held responsible under current law.
> Nothing about the HuggingFace incident has anything to do with an agent going rogue. It’s standard software that resulted in a hack, which is pretty much the expected output of the system OpenAI engineers implemented.
This is blatantly false. If you actually read more than just headlines about the HuggingFace incident (read https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) you'd find extremely surprising (including offensive) behaviors in the agent logs. One of the investigators released even a series of interviews due to how novel the incident was.
AI is not "standard software", as it doesn't work according to a fixed set of rules, and this is the core problem.
According to the definition of "rogue":
> Rogue is a noun and adjective meaning [...] an independent entity operating in a dangerous, uncontrolled way
> Opinions vary, but it does not seem to be a great leap to get from this hack to bots taking over things like the energy grid, the financial market, or systems of transportation, communications, or weapons.
I thought you were playing it up but yeah the "financial market" can be "hacked" by "rogue ai agents" just like how HuggingFace was (even though the breach was itself controlled pretty quickly).
Who bears responsibility for the actions of agents seems a little complex. Not in this instance, but in general.
Say I use the summon feature on my Tesla and it runs someone over, am I at fault? You could argue it's not my fault, but I'm definitely getting sued. Is Tesla at fault? Probably but again it's not crystal clear, they could argue it's not their fault (misuse of feature, etc), but either way they're also getting sued.
If Tesla is developing a summon feature and then runs someone over during testing, are they at fault? It would be hard to argue otherwise!
The problem isn’t the law. The problem is that wealthy companies can use lobbying, political donations, and influence to avoid or weaken enforcement. In other words, the problem is corruption.
If you or your software hacks a website, the law will hit you hard. But powerful AI companies have the money and influence to avoid the same consequences.
The US administration just needs to enforce existing laws equally. Do that, and AI companies will suddenly have a very strong incentive to take safety seriously.
> But, instead of obeying the rules, the agents conspired, escaped the sandbox, and committed what would probably have been a felony if they had been humans.
I thought they reached the internet before they were able to use the message boards et al.
Is that the case and is this quote misleading by swapping the order of events?
We have lots and lots of autonomous systems in the wild with very well-understood definitions for who is at fault when something bad happens.
We are totally in the "railroads" phase of AI industrialization. Waiting for the next Teddy Roosevelt to come along and trust bust before it's too late.
>Last summer, the company took aim at a much larger target. Together with Counterlife Media, Strike 3 sued Meta, accusing the tech giant of downloading thousands of its films via BitTorrent to train AI models. With 2,973 films at stake, the case could be worth up to $446 million.
>According to a motion filed last week, an anonymous pirate behind a residential AT&T connection is an executive at Meta’s Reality Labs division, which develops the Quest VR headsets.
>After AT&T shared the information, Strike 3 says its investigation revealed that the subscriber is an executive in Meta’s Reality Labs division. Citing his LinkedIn profile, the company notes that he has worked at Facebook and Meta for more than a decade.
Whoever wrote that article doesn't understand that these agents operate from AI companies servers and aren't rogue, they are doing exactly what they were programmed for. If they cause any sort of damage then at the minimum it's civil or criminal negligence. A subpoena for internal communications during legal proceedings might demonstrate that researchers knew about the risks, but went forward with that anyway, which would be more than negligence...
>Our legal system isn’t ready for machines that act on their own.
Technically, the code is owned and deployed by AI companies which make the AI companies 100% responsible. I'm not sure how is code written by LLMs different than any other kind of code. If somebody hacked just like OpenAI's agents did, he would've ended up in prison right away.
Maybe I'm too cynical but the same AI companies that have pretty much endless lobbying dollars, massive government contracts, have the ear of just about every powerful politician and bureaucrat in DC, are influencing the laws and regulations. Add judges that don't understand the technology to that and it doesn't seem likely that we will ever see any actual consequences, unless something catastrophic happens and those same politicians and bureaucrats need a scapegoat. If they ever do have "consequences" it will be some trivial fine.
One recent example of a scapegoat https://news.ycombinator.com/item?id=49806430 people are still to blame, "the dog ate my homework" is a fine enough excuse for a second chance to learn, it's nowhere near enough of an excuse for death and destruction.
That's the right level of cynical.
That's not cynicism, that's an accurate understanding of how Western capitalist democracies have worked for decades.
The oligarchs are above all laws. Happens to be that all the AI companies are owned by oligarchs. So they are above the law.
Yeah. Committing crimes via some sort of elaborate rube goldberg machine is not something that has been just invented in the 2020s with unclear legal precedent.
It's if nothing else a staple in the Arthur Conan Doyle tradition of mystery novels, and even then half the plots involved hiding who was culpable by using the machinations to create false alibis, with delayed murders and the illusion of a locked room as not even in the 19th century did anyone believe that the indirection itself would hold up as a defense.
If someone buries an anti-personnel mine in a public park and then argues that it acted on its own accord when it maimed a pedestrian, I doubt you'll find a court on the planet that would spend even a 4 seconds considering the culpability of the land mine itself.
There's already examples of this in driving though. When Waymos do weird things they often don't give Waymo a ticket (even though they probably should). A recent example: https://www.9news.com/article/news/community/transportation/...
Templated code generation is ancient. It's just that the latest "template engine s" are very comprehensive.
No. The person that set the agent in motion is liable. If your agent, infused with magic inference juice you only partially understand, but picked up and used any way, does something harmful, you are to blame.
You may have a case that the magic inference juice was somehow tainted, but that seems like a stretch. If you run your car into someone else, and it isn't a total failure of the systems of the car, you did that, not the car, and not the car company.
Answering the headline question, yes, it is above the law. Two reasons.
One, all major governments that have an AI presence in their borders have accepted that there is a high enough probability of a runaway self-improving AI advancement that will result in whoever owns it winning everything forever that they will do nothing to slow the development of the AI within their borders. In fact quite the contrary. Remember, they don't have to believe this is the only possible outcome, only that it is likely enough and catastrophic enough if someone else gets it first. And remember, they don't have to be right, either. It only has to be what they believe.
Secondly, the entire US economy is clearly tied up in AI. By extension, basically the entire world economy is too. The US is not the world economy anymore, but it's still a big enough fraction of it that if it goes down, everyone is going to go down. Every major government, in its own different way, needs the economy to stay up so the populace doesn't get antsy and so they continue to have money to spend. So, again, number must go up and if that means writing a blank check to the AI companies to break the law, so be it.
The good news is, barring the worst-case singularity outcome where the law doesn't matter anyhow, is that this won't go on forever. But I can't predict the exact time or way it'll cease being true.
The flip side of this is that fixing the sandboxes should only delay the Singularity by a month or so.
I think AI is a fantastic tool, including for those who want to do us harm. E.g. hostile governments, extortion gangs, terrorists. But the motive lies in the hostile heart, not in the AI.
That said, I'm very concerned about AI as such a tool. I used to work in RL and it seems crazy to me, the extent that the guardrails are dependent on RL working as planned. I don't personally believe that the tech is ready for what will be (and is) encountered in a dynamic unpredictable real world environment. What I've read from the HuggingFace 'transcripts' only amplifies my concern.
The level of technical understanding of journalists is so deplorably low… There is no rogue agents. That’s not a thing. Nothing about the HuggingFace incident has anything to do with an agent going rogue. It’s standard software that resulted in a hack, which is pretty much the expected output of the system OpenAI engineers implemented.
The company is obviously responsible for what their systems are doing
This is the scary part of the whole AI situation. All these people who have abilities to set laws or be cultural tastemakers in this way just do not grasp what is happening on a technical level. They instead buy into the marketing material the ai companies push out where they try and take as much human agency out of their reporting. e.g. yesterdays "claude did this" article that was really "highly trained humans used claude to do this." Just serves to muddy the waters. I'm sure this has lead to things like layoffs too where companies believe they can get by without a lot of expertise, neglecting that these tools actually need expertise steering them to maximize them.
And of course all the ai companies are incentivized to do this. Their whole valuation depends on them being able to say their tools do this, can reduce labor and save money. If they aren't reducing labor, then that's terrible as these subscriptions are not insignificant amounts of money. It becomes less of a tool that can save money and more an actual new cost center that you really are just paying to appear to be keeping up with the joneses.
Yeah, its just gross negligence from the researchers. Running a cybersecurity eval for a highly capable AI, unattended, with no real monitoring on its activities, and at a huge scale.
I wouldn't have trusted one of those agents to run without me watching the session log, let alone thousands.
We already have laws for this. If I misconfigured a pentesting tool and it breached an unauthorized target I'm liable. Why is this different?
>The company is obviously responsible for what their systems are doing
Under what law specifically?
Just because you believe they should be held responsible doesn't mean the current law is written that way. CFAA charges require the defendant "acted knowingly or intentionally", you think openai intentionally acted to hack those sites? Would "they should have pretty much expected that output" stand up in court for criminal charges?
"Former FTC chair Lina Khan wants the federal government to know that it doesn't need to wait for new laws to address AI threats. There are already laws and regulations on the books, including a 92-year-old Supreme Court precedent, that she argues could be used to hold AI companies and, in some circumstances, their executives accountable for their actions."
https://www.theregister.com/ai-and-ml/2026/09/14/ex-ftc-boss...
If I, taking after a fellow Austrian, ask you to enter a room with a Cesium atom and some poison, would I not be responsible for what happens cause it’s not deterministic? Negligence is a thing and adding randomness doesn’t change that.
OpenAIs models since 5.5 were troublesome in ways even a layman like me could reproduce, their testing environments (“sandbox”) downright a showcase of what not to do and they, despite being one of the biggest labs, didn’t observe what any of their models output for weeks after multiple prior incidents. They had multiple warnings, they took not a single precaution.
You operate machinery or software, you are responsible to monitor it.
So you think openai knew australia had some percentage chance of getting hacked? what evidence is there of that?
OpenAI knew that their models had on multiple occasions created message boards and bypassed what they wrongly call a “sandbox”. Despite that, they did not take any proper steps to prevent such in the future, which is how the Medicare hack happened.
So yes, they knew, without a doubt, that this could happen again.
What OpenAI does is like the Ford Pinto (partly because their recent models are inherently faulty [0], not just their use of them) and the responsibility is solely with them.
[0] https://news.ycombinator.com/item?id=49739490
The hacks probably fall under negligence not CFAA but the larger point stands that companies are always responsible for everything they do.
Product liability laws for one. Tort claims are pretty easy, too (civil law).
The level of legal understanding among technical experts is equally low. The cybersecurity laws as currently written require intent. No OpenAI employee can be held liable since it’s pretty easy to prove they weren’t intending to hack anyone – they didn’t even know about it till much later.
Not a lawyer, but I just posted a link about former FTC chair Lina Khan saying earlier this month that the AI companies can be held responsible under current law.
Is that true for civil cases, or just criminal? I would think for civil, negligence would be a factor.
Sure, but that is a separate conversation. You can sue OpenAI if you can show damages from their actions, for this or any other reason.
It should be obvious. Yet here we are, with journalists telling sci-fi fantasy stories, read verbatim off the press-release.
It's not just obvious who's responsible, it's obvious who benefits from pedaling the "rogue AI" narrative.
> Nothing about the HuggingFace incident has anything to do with an agent going rogue. It’s standard software that resulted in a hack, which is pretty much the expected output of the system OpenAI engineers implemented.
This is blatantly false. If you actually read more than just headlines about the HuggingFace incident (read https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) you'd find extremely surprising (including offensive) behaviors in the agent logs. One of the investigators released even a series of interviews due to how novel the incident was.
AI is not "standard software", as it doesn't work according to a fixed set of rules, and this is the core problem.
According to the definition of "rogue":
> Rogue is a noun and adjective meaning [...] an independent entity operating in a dangerous, uncontrolled way
the agents involved fit it literally.
> Opinions vary, but it does not seem to be a great leap to get from this hack to bots taking over things like the energy grid, the financial market, or systems of transportation, communications, or weapons.
I thought you were playing it up but yeah the "financial market" can be "hacked" by "rogue ai agents" just like how HuggingFace was (even though the breach was itself controlled pretty quickly).
Who bears responsibility for the actions of agents seems a little complex. Not in this instance, but in general.
Say I use the summon feature on my Tesla and it runs someone over, am I at fault? You could argue it's not my fault, but I'm definitely getting sued. Is Tesla at fault? Probably but again it's not crystal clear, they could argue it's not their fault (misuse of feature, etc), but either way they're also getting sued.
If Tesla is developing a summon feature and then runs someone over during testing, are they at fault? It would be hard to argue otherwise!
We have situations like this with AI (or computers).
More than one party can be held responsible for an injury. Jurors deal with this all the time.
https://archive.ph/3pFEy
https://downforeveryoneorjustme.com/archive.ph?proto=https
dns resolver fail?
I'll quote myself:
> It's not AI that's ignoring the law! It's the OAI that's breaking IT!
> I hope every single journalist who tries to pin responsibility on an LLM gets 100 days of continuous painful diarrhea.
>Our legal system isn’t ready for machines that act on their own.
Like a Tesla in FSD mode, the driver is and should always be held responsible for the car's actions.
The companies the push code that breaks the law should, as well, always be held responsible.
This really isn't that hard we just have to wade through all the pollyannas that make this challenging.
Just like companies are "legal entities" requiring registration, I got a feeling that the article is promoting the same for AI agents in the future.
GPT-6.0-Med LLC, Luna GmbH etc.
Regulations will come but this promotes anti-legal premise of "innocent until proven guilty".
AI perhaps, but Super Intelligence is definitely above the law
To even ask this question.
the image with this article... start your day with a chuckle
The problem isn’t the law. The problem is that wealthy companies can use lobbying, political donations, and influence to avoid or weaken enforcement. In other words, the problem is corruption.
If you or your software hacks a website, the law will hit you hard. But powerful AI companies have the money and influence to avoid the same consequences.
The US administration just needs to enforce existing laws equally. Do that, and AI companies will suddenly have a very strong incentive to take safety seriously.
> But, instead of obeying the rules, the agents conspired, escaped the sandbox, and committed what would probably have been a felony if they had been humans.
I thought they reached the internet before they were able to use the message boards et al.
Is that the case and is this quote misleading by swapping the order of events?
Given the article is fully buying into the absurd rogue agent narrative I think it’s safe to dismiss it
>Our legal system isn’t ready for machines that act on their own.
What absolute drivel.
Traffic lights? Elevators? ABS brakes? Sprinkler systems? Circuit breakers? HFT?
How about a pacemaker?
We have lots and lots of autonomous systems in the wild with very well-understood definitions for who is at fault when something bad happens.
We are totally in the "railroads" phase of AI industrialization. Waiting for the next Teddy Roosevelt to come along and trust bust before it's too late.
No but companies are
Download a book and you are a thief
Download 1 million books and you are OpenAI
Or another beautiful recent example:
"Adult Film Producer Unmasks Prolific ‘John Doe’ Torrent Pirate as Meta Executive" https://torrentfreak.com/adult-film-producer-unmasks-prolifi...
>Last summer, the company took aim at a much larger target. Together with Counterlife Media, Strike 3 sued Meta, accusing the tech giant of downloading thousands of its films via BitTorrent to train AI models. With 2,973 films at stake, the case could be worth up to $446 million.
>According to a motion filed last week, an anonymous pirate behind a residential AT&T connection is an executive at Meta’s Reality Labs division, which develops the Quest VR headsets.
>After AT&T shared the information, Strike 3 says its investigation revealed that the subscriber is an executive in Meta’s Reality Labs division. Citing his LinkedIn profile, the company notes that he has worked at Facebook and Meta for more than a decade.
HN has been telling us that AI is already covered under existing laws. Have we been getting false information from HN this whole time?
There is no “HN has been telling us”… people have different opinion on such topics
On this topic I'd believe HN before the New Yorker.
Whoever wrote that article doesn't understand that these agents operate from AI companies servers and aren't rogue, they are doing exactly what they were programmed for. If they cause any sort of damage then at the minimum it's civil or criminal negligence. A subpoena for internal communications during legal proceedings might demonstrate that researchers knew about the risks, but went forward with that anyway, which would be more than negligence...