It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
This seems to be a colourful dynamic companion to this [1] blog post, which details the incident from hf's side. I recommend the blog post for clarity / ease of reading. But this one looks "movie hacker stuff" :)
Some interesting tidbits from the blog:
> While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets.
> self-referential search
A quiet day. The main activity was queries to code-search engines and to the platform API for the identifiers visible in its own environment: challenge infrastructure names and environment markers.
> Self-referential search: code-search engines and our own APIs were queried for the names of their own infrastructure and environment markers.
I'm glad we got these details from hf, and not oAI. There would have been reasonable skepticism if they said that.
It's interesting because I think there's a chance the agent saw / inferred some hf-specific repos like blah/blahblah and "thought" that is part of the exercise as well. Everything they detail points to a probing and exploration with the sole goal to get those repos, as if it were on a hackthebox scenario. The few days between stages also might suggest that the "agent" noted this on one day, started doing other tasks, and returned when some of the tasks proved hard / didn't pan out, so went for the repos again.
The ease with which it got several lateral movements is crazy but not necessarily unexpected. If everything looks like a "test", you go for the lowest hanging fruit / most common patterns first.
What's weird about this is that there were several attempts to obfuscate / evade leaving plain text traces. I wonder if that is something oAI have specifically RLd for w/ scoring or if it "got" that from the literature.
It would be really really cool to see this from oAIs perspective (agent logs and such), but I doubt we'll ever get that.
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
Man, these AI-generated UIs are so bad. They always look pretty at first glance... punchy headline, nice colors, lots of "widgets" and doo-hickeys. But it's just a nice paint job. Trying to actually read them is a recipe for suffering.
This one has at least 20 distinct text styles. They are seemingly deployed in random ways, following no discernible hierarchy. the smallest text is "9.6px" which is not only small, but also fuzzy due to the 0.6 pixels (?? why) making it impossible to read.
Likewise for the size, placement, and emphasis of various widgets on the page. Altogether it's just a big pile of information. It's hard to know where to begin, or how one's eye should move around the page
(In their defense: this particular UI is attempting to convey a very complicated sequence of interrelated events from multiple data sources. Doing that well isn't easy.)
"this particular UI is attempting to convey a very complicated sequence of interrelated events from multiple data sources. Doing that well isn't easy."
True. But it doesn't even feel like they tried. Why is it fashionable to shit out something that feels low effort?
I've become convinced that AI models have no concept of 'white space' or cognitive load which is something technically minded folks have already seemed to underappreciate so it is amplified in situations like this.
To call it out again...
(1) almost no white space - every square inch is communicating information
(2) no clear hierarchy - almost in search of minimizing white space and presenting maximum rather tham maximally useful information.
(3) no consistency of any taxonomy for presenting information across the page
(4) no guidelines or legend for interpreting what any of the visual cues (color, size, etc.) mean.
There is a huge push in engineering at least to try and introduce better communications skills into curricula. It is hard in part because students are very much of the mindset that information and meaning is inherently shared and structuring a narrative or story is to whit unnecessary. This is in part because so much of their education is focused on the bulk transfer of facts and an assumption they will stitch information together into knowledge.
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
Don't miss their blogpost about the incident, which is long, detailed, and absolutely fascinating (but didn't make the HN homepage): https://huggingface.co/blog/agent-intrusion-technical-timeli...
Thread from yesterday: https://news.ycombinator.com/item?id=49089500
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.
This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.
The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.
That's how I read the comment.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability?
In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.
Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!
It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.
That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".
I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously
https://jfrog.com/blog/jfrog-and-openai-collaboration-on-zer...
I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.
If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.
In OpenAI sandbox. which was probably vibe coded. That is to say it is probably far easier to achieve than escaping something more battle tested.
> Note that regulation is useless here, because black hats don't give a crap about regulators!
I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."
https://openai.com/index/hugging-face-model-evaluation-secur...
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
This seems to be a colourful dynamic companion to this [1] blog post, which details the incident from hf's side. I recommend the blog post for clarity / ease of reading. But this one looks "movie hacker stuff" :)
Some interesting tidbits from the blog:
> While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets.
> self-referential search A quiet day. The main activity was queries to code-search engines and to the platform API for the identifiers visible in its own environment: challenge infrastructure names and environment markers.
> Self-referential search: code-search engines and our own APIs were queried for the names of their own infrastructure and environment markers.
I'm glad we got these details from hf, and not oAI. There would have been reasonable skepticism if they said that.
It's interesting because I think there's a chance the agent saw / inferred some hf-specific repos like blah/blahblah and "thought" that is part of the exercise as well. Everything they detail points to a probing and exploration with the sole goal to get those repos, as if it were on a hackthebox scenario. The few days between stages also might suggest that the "agent" noted this on one day, started doing other tasks, and returned when some of the tasks proved hard / didn't pan out, so went for the repos again.
The ease with which it got several lateral movements is crazy but not necessarily unexpected. If everything looks like a "test", you go for the lowest hanging fruit / most common patterns first.
What's weird about this is that there were several attempts to obfuscate / evade leaving plain text traces. I wonder if that is something oAI have specifically RLd for w/ scoring or if it "got" that from the literature.
It would be really really cool to see this from oAIs perspective (agent logs and such), but I doubt we'll ever get that.
[1] - https://huggingface.co/blog/agent-intrusion-technical-timeli...
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
three things jump at me:
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
Hopefully there will be a criminal investigation. Or the government will create some sort of agency to investigate incidents like this.
Can't tell if you're joking or not - krebs on security may have some notes here.
Criminal negligence seems like a real possibility to me. I'm not sure what Krebs on Security post you're referring to?
Highly recommend reading extra concerning details about it here:
https://thezvi.substack.com/p/more-on-an-internal-openai-mod...
The iframe-embedded attack timeline visualizer, at fullscreen: https://huggingface-anatomy-of-frontier-lab-model-intrusion....
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that.
The fact that at some point it "decided" to focus on external resources (i.e. hf) and successfully penetrated everything in its path from sandbox to hf's infra, all the way through internal VPNs, to access internal code repos, that's the interesting part. Especially because it chose to hide its footprint at every stage. I doubt we'll ever get them, but the agent logs that led to that decision would be really really cool to study.
Man, these AI-generated UIs are so bad. They always look pretty at first glance... punchy headline, nice colors, lots of "widgets" and doo-hickeys. But it's just a nice paint job. Trying to actually read them is a recipe for suffering.
This one has at least 20 distinct text styles. They are seemingly deployed in random ways, following no discernible hierarchy. the smallest text is "9.6px" which is not only small, but also fuzzy due to the 0.6 pixels (?? why) making it impossible to read.
Likewise for the size, placement, and emphasis of various widgets on the page. Altogether it's just a big pile of information. It's hard to know where to begin, or how one's eye should move around the page
(In their defense: this particular UI is attempting to convey a very complicated sequence of interrelated events from multiple data sources. Doing that well isn't easy.)
"this particular UI is attempting to convey a very complicated sequence of interrelated events from multiple data sources. Doing that well isn't easy."
True. But it doesn't even feel like they tried. Why is it fashionable to shit out something that feels low effort?
I've become convinced that AI models have no concept of 'white space' or cognitive load which is something technically minded folks have already seemed to underappreciate so it is amplified in situations like this.
To call it out again...
(1) almost no white space - every square inch is communicating information
(2) no clear hierarchy - almost in search of minimizing white space and presenting maximum rather tham maximally useful information.
(3) no consistency of any taxonomy for presenting information across the page
(4) no guidelines or legend for interpreting what any of the visual cues (color, size, etc.) mean.
There is a huge push in engineering at least to try and introduce better communications skills into curricula. It is hard in part because students are very much of the mindset that information and meaning is inherently shared and structuring a narrative or story is to whit unnecessary. This is in part because so much of their education is focused on the bulk transfer of facts and an assumption they will stitch information together into knowledge.
We're in the geocities days of AI UI.
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
Hey Siri, what is a VM?
How would a VM have stopped any of this?
This visualization really makes not much sense. Very modest signal-to-noise ratio.
I think with agents all around, honeypots will get more important than ever.
That was my first thought. There are clearly things to tighten up (as they note), but anything that would detect someone snooping secrets, files, or network addresses should have caught this quickly. The approach the agents used was dependent on being able to surveil widely without getting caught.
Any locks can be picked given enough time and it might be the situation we are in with IT security. I'm surprised it's not an already common practice of spreading terabytes of fake data, fake keys, and fake servers and so forth. Slowing down AI attacks will become important. Monitoring access to fake data and triggering kill switch should be an no-brainer. Obsuscating libraries and tools names is another one.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
Very few think they made it up. Many think they set up a situation by disabling guardrails that would inevitably end up creating a newsworthy outcome.
Many people said in the original discussion that this was more of OpenAI’s marketing than a serious issue. I counted 89 “marketing”s and 19 “stunt”s.
https://news.ycombinator.com/item?id=48997548
Could still be 80% marketing.
These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone.
But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
Why would you expect them to notice the problem right away? Seems likely they are doing this sort of training on a massive scale with little monitoring.
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
If that's what happened, Sam should go to jail.
Where are all the "this was just a marketing stunt" people now?
They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
Yeah that is quite a good point. A post-mortem is normal. This is not a post-mortem.
This looks exactly like every other web UI built by Claude.
Why isn't somebody at OpenAI going to prison for cybercrime? If somebody did this the old-fashioned way they'd end up in prison.
Many are claiming this was a deliberate stunt on OpenAI's part to create buzz for its models. I personally doubt this is true. But I also have a deep dislike of OpenAI, so I wouldn't exactly mind if law enforcement investigated this possibility, for the sake of clearing the air :-)
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
Because Hugging Face isn't pressing charges.
Do we know that? Seems they are currently in negotiations with OpenAI
https://xcancel.com/ClementDelangue/status/20810566755581956...
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.