Wow, this is serious. Makes you think, that even though QubesOS attack surface is so tiny (well-designed to be secure) there are still vulnerabilities to be found.
Worth noting that (as I understand) this vulnerability occurs only when doing copy-to-VM from Dom0:
>Note that the VM variant of `qvm-copy-to-vm` is not affected, as its
version of the error reporting function does not use `system()`:
Since you should not use Dom0 for regular work, and definitely not for interacting with likely-to-be-infected VMs, the scope of this attack is smaller than it sounds. On the flip side, when it works, it elevates privileges straight to Dom0.
You are right, copying to dom0 is not best practices and warned against since anno dazumal, but given the user groups I remember not always being technically minded (journalists, dissidents, etc.) and ensuring qubeses isolation holds even when users do things they are discouraged from has always been part of the philosophy. Don’t trust users, don’t trust userland, don’t trust software and all that yazz.
I believe this is a vulnerability that occurs when copying data from dom0, which is a more common task. Generally the Qubes model recommends copying data from more trusted VMs to less trusted VMs, and dom0 still runs some system-wide processes in many default configurations.
For example when you take a screenshot with xfce4-screenshooter, the file is saved to dom0, and you have to use qvm-copy-to-vm to move it to a (less trusted) qube to do something with it. That's the most frequent use case, at least for me.
Theos is a very opinionated and not necessarily wrong position, but I feel also a bit too reductive given we are eternally having to deal with compromises of some form. Also, lest we forget, it has been two decades in the interim and oh so much has changed. In any case, this originated from their code, not virtualization, so it doesn’t really apply either way…
I think it's pretty much spot on myself and applies to more than virtualization based on the last point. It really suggests that further complexity and abstraction is not a good security posture. And I agree with this from extensive experience (embedded, defence).
Regarding the two decades since and the numerous exploitable x86-64 and hypervisor bugs suggests he wasn't wrong and that the tone was appropriate for the severity of the problem.
Yeah, I'm with Theo on this one. Conventional OS security between Ring-0 and everything else is well understood; the problem has become too much code in Ring-0, a great fraction of which has its own interfaces across the security boundary, and the Unix security model just doesn't scale.
No capabilities, or even a sane and useful way of adding capabilities with everything in ring 0, and the flat integer namespacing of users and groups just doesn't work for what userspace needs to do today - hence namespaces, which have introduced their own problems, because (no surprise) trying to graft a tree structure onto a flat integer namespace after the fact is a mess.
Virtualization tried to sidestep all that, but to make it fast the cost has been more driver interfaces to host ring-0 - remember what the original was? - and screwing around a whole bunch with particularly arcane facets of the core ring-0 security boundary, e.g. page tables.
The bug here is not related to virtualization, but a footgun as old as C stdlib: system() that doesn't take arguments separately, and instead relies on shell escaping by the application.
I hope when people read this though they understand this is a communication style; they're clearly trying to strongly discourage people from thinking they are suddenly protected. Effective? Maybe at one time, where "macho dev energy" was a thing. Today, not so much. You can tell they mean well because the intro sentence is actually pretty cheeky!
Being right is not the most important part. If you're right but don't convince anyone, you've made no difference.
Theo was right that virtualization is a comparatively shoddy security boundary. At the same time, it's flexible and capable in ways that now define the shape of modern IT.
Could we have replicated that by other means? If yes, then it's on Theo and other knee-jerk critics that they never proposed a better approach and settled for insulting people. If not, then maybe virtualization was a necessary evil. Or maybe everyone else is an irredeemable idiot, but again - if we reach that conclusion, is the world better off?
I don't really know as the argument is mainly about how stupid people are... The technical argument is one paragraph ended with an insult, not much to make an educated and civilized opinion.
This is less of a virt/x86 bug and more of a "don't call system() on arbitrary user input" bug.
.. incidentally, OpenBSD also provides one of the clearest examples of how the excuse "calling system() is fine in my case, its totally not arbitrary user input" is deluded just the same, see CVE-2020-8794.
It has? News to me. Go on any major thread on this page, you’ll witness similarly strong pushback visa-vi buying into corporate backed hype, akin to the overconfidence in virt security he pointed at back then.
The founder Joanna Rutkowska left QubesOS in 2018. All the code involved in this bug was committed by her successor Marek Marczykowski-Górecki.
Joanna seems to be a genuine good guy, she once wrote a paper titled "Intel x86 considered harmful". That's why Huawei and the Chinese government aren't even trying any more to make western CPU architectures secure, it's a hopeless cause.
I am still impressed by QubesOS track and I use it for my dedicated 'financials' laptop.
IMHO the thing that is holding back QubesOS is the lack of hardware acceleration for graphics - maybe now when dual monitor setups are getting popular this could be a workaround for the security considerations?
I dropped QubesOS once exactly for that reason in the past but now I’m running it again on a separate computer.
Even with all its drawbacks, there is something really nice about being able to run different applications over Tor, VPN or plain internet simultaneously, the ability to isolate non-safe binaries and being able to backup your VMs easily.
I wish a similar distro would be made based on KVM so that the standard kernel could be used. It would be great for compatibility.
> being able to run different applications over Tor, VPN or plain internet simultaneously, the ability to isolate non-safe binaries and being able to backup your VMs easily.
These are all possible using light containers. For example, on FreeBSD I will spin up a jail which runs wireguard, and then I'll bridge that to another a jail. That 2nd jail is running entirely off wireguard without any other way to access the network. Since it is a jail, it is isolated. And backing up is as simple as a zfs snapshot and zfs send. I assume the same is possible on Linux.
Problem is not that nobody wants to do it, it is that GPU stack is exactly the kind of enormous driver surface Qubes exists to keep away from dom0. Second monitor does not really change that, somebody still has to trust the driver.
Mini tangent: Could someone explain to me why Qubes is used for security, when (from what I understand) Jails on BSD is significantly more robust/safe/has a much smaller exposed area? Is it just "everyone's using linux already; here's a safer linux"?
Qubes can be viewed as a Xen distribution, rather than a Linux distribution [1]. You may find the Qubes FAQ a good starting point (I'm reading it now because of your question, so thanks).
The worst part is that in 2020 they explicitly documented that the remote filename is attacker controlled,but still allowed it to reach system()
That is C security 101: never pass untrusted input through a shell. This should have been caught in review!
That is sphincter tightening to read. Have to point out how amazingly well their bulletins handle communication. Clearly describes the issues, how users are to act, etc. in, what I feel, is an easy to grasp language, even if one’s not in the weeds that much. In fairness though, I do still have some past memories concerning Qubes architecture from way back, so maybe my assessment is wrong and this is still not that straight forward to grasp for most.
> Important: At this point, you still don’t know whether the key you just imported is the genuine QMSK or a forgery. In order for this entire procedure to provide meaningful security benefits, you must authenticate the QMSK out-of-band. Do not skip this step! The standard method is to obtain the QMSK fingerprint from multiple independent sources in several different ways and check to see whether they match the key you just imported. For more information, see How to import and authenticate the Qubes Master Signing Key.
It looks like Qubes is ran by people who take security seriously, which is refreshing.
I own a Qubes T-shirt which I bought in person at FOSDEM. The design on the T-shirt consists of many copies of the QMSK in hexadecimal. All of their merch is like this.
The worst part is that in 2020 they explicitly documented that the remote filename is attacker controlled,but still allowed it to reach system()
This is C security 101: never pass untrusted input through a shell. This should have been caught in the review!
Isn't it the case for all bugs? If they appear in the production software, it means that they passed the review. And obviously bugs shouldn't pass the review, but that's easier said than done.
Another example for why system() is so dangerous to use.
I also don't understand why it needs to show the dialog in dom0. If you have the option to handle attacker controlled input on the unprivileged side, you should do that instead of putting a lot of logic on the privileged side.
The code is sloppy. They check existance of kdialog binary using full path; next step they rely on PATH search by shell. If would've been much safer to just do execve directly.
If you're going to put the graphics and NIC into separate VMs, surely the secure screen can be another of those semi-privileged VMs rather than part of dom0
Peculiar stuff. I'm always skeptical of these security Linux distributions, but this bug is so bad that it seems like an infiltration of Qubes at best or Qubes being a honeypot at worst.
Wow, this is serious. Makes you think, that even though QubesOS attack surface is so tiny (well-designed to be secure) there are still vulnerabilities to be found.
Worth noting that (as I understand) this vulnerability occurs only when doing copy-to-VM from Dom0:
>Note that the VM variant of `qvm-copy-to-vm` is not affected, as its version of the error reporting function does not use `system()`:
Since you should not use Dom0 for regular work, and definitely not for interacting with likely-to-be-infected VMs, the scope of this attack is smaller than it sounds. On the flip side, when it works, it elevates privileges straight to Dom0.
You are right, copying to dom0 is not best practices and warned against since anno dazumal, but given the user groups I remember not always being technically minded (journalists, dissidents, etc.) and ensuring qubeses isolation holds even when users do things they are discouraged from has always been part of the philosophy. Don’t trust users, don’t trust userland, don’t trust software and all that yazz.
I believe this is a vulnerability that occurs when copying data from dom0, which is a more common task. Generally the Qubes model recommends copying data from more trusted VMs to less trusted VMs, and dom0 still runs some system-wide processes in many default configurations.
For example when you take a screenshot with xfce4-screenshooter, the file is saved to dom0, and you have to use qvm-copy-to-vm to move it to a (less trusted) qube to do something with it. That's the most frequent use case, at least for me.
Reminds me of Theo DeRaadt again: https://marc.info/?l=openbsd-misc&m=119318909016582
I feel that, in fairness, one should at least read Adam’s response, though ideally all subsequent mails: https://marc.info/?l=openbsd-misc&m=119320496730314&w=2
Theos is a very opinionated and not necessarily wrong position, but I feel also a bit too reductive given we are eternally having to deal with compromises of some form. Also, lest we forget, it has been two decades in the interim and oh so much has changed. In any case, this originated from their code, not virtualization, so it doesn’t really apply either way…
That's an impressive amount of maturity and composure Adam demonstrates there after receiving a response like that.
I think it's pretty much spot on myself and applies to more than virtualization based on the last point. It really suggests that further complexity and abstraction is not a good security posture. And I agree with this from extensive experience (embedded, defence).
Regarding the two decades since and the numerous exploitable x86-64 and hypervisor bugs suggests he wasn't wrong and that the tone was appropriate for the severity of the problem.
In all fairness also read this: https://taviso.decsystem.org/virtsec.pdf
Yeah, I'm with Theo on this one. Conventional OS security between Ring-0 and everything else is well understood; the problem has become too much code in Ring-0, a great fraction of which has its own interfaces across the security boundary, and the Unix security model just doesn't scale.
No capabilities, or even a sane and useful way of adding capabilities with everything in ring 0, and the flat integer namespacing of users and groups just doesn't work for what userspace needs to do today - hence namespaces, which have introduced their own problems, because (no surprise) trying to graft a tree structure onto a flat integer namespace after the fact is a mess.
Virtualization tried to sidestep all that, but to make it fast the cost has been more driver interfaces to host ring-0 - remember what the original was? - and screwing around a whole bunch with particularly arcane facets of the core ring-0 security boundary, e.g. page tables.
It is a mess.
The bug here is not related to virtualization, but a footgun as old as C stdlib: system() that doesn't take arguments separately, and instead relies on shell escaping by the application.
The only point of system(3) is to invoke the OS shell, if you do not want that use exec(3).
Looks like this has nothing to do with the hypervisor, it's not a traditional VM escape
Theo is a very insightful guy, but also very opinionated. I think the truth is somewhere in between.
Especially as more and more virtualization functions move into hardware, not using them as a second security barrier seems foolish.
Brutal
I hope when people read this though they understand this is a communication style; they're clearly trying to strongly discourage people from thinking they are suddenly protected. Effective? Maybe at one time, where "macho dev energy" was a thing. Today, not so much. You can tell they mean well because the intro sentence is actually pretty cheeky!
All I see is rudeness, insults, and arrogance sparkled with inklings of technical arguments. Worthless.
But, and this is the important part, is he wrong?
Being right is not the most important part. If you're right but don't convince anyone, you've made no difference.
Theo was right that virtualization is a comparatively shoddy security boundary. At the same time, it's flexible and capable in ways that now define the shape of modern IT.
Could we have replicated that by other means? If yes, then it's on Theo and other knee-jerk critics that they never proposed a better approach and settled for insulting people. If not, then maybe virtualization was a necessary evil. Or maybe everyone else is an irredeemable idiot, but again - if we reach that conclusion, is the world better off?
Maybe if we didn't virtualize everything at machine level we'd have portable software that runs on the original virtualization method: processes.
Stares at Go as about the only step in that direction...
LLM slop account. Admittedly this one was harder to spot.
You're chasing ghosts, son.
Oh shit, you're right!
I don't really know as the argument is mainly about how stupid people are... The technical argument is one paragraph ended with an insult, not much to make an educated and civilized opinion.
This community lives on not understanding that...form over function always...
Can I insult you and then complain that you focus too much on form?
Only If I deserved it :-)
What if that isn’t the most important part
This is less of a virt/x86 bug and more of a "don't call system() on arbitrary user input" bug.
.. incidentally, OpenBSD also provides one of the clearest examples of how the excuse "calling system() is fine in my case, its totally not arbitrary user input" is deluded just the same, see CVE-2020-8794.
Peak Theo! This refreshing truth telling has been eradicated in 2026.
It has? News to me. Go on any major thread on this page, you’ll witness similarly strong pushback visa-vi buying into corporate backed hype, akin to the overconfidence in virt security he pointed at back then.
Vi doesn't require a visa but the trip is one way only. (Vis-a-vis)
You know, given my French grades, I really should stop using such phrases…
The founder Joanna Rutkowska left QubesOS in 2018. All the code involved in this bug was committed by her successor Marek Marczykowski-Górecki.
Joanna seems to be a genuine good guy, she once wrote a paper titled "Intel x86 considered harmful". That's why Huawei and the Chinese government aren't even trying any more to make western CPU architectures secure, it's a hopeless cause.
I am still impressed by QubesOS track and I use it for my dedicated 'financials' laptop.
IMHO the thing that is holding back QubesOS is the lack of hardware acceleration for graphics - maybe now when dual monitor setups are getting popular this could be a workaround for the security considerations?
I dropped QubesOS once exactly for that reason in the past but now I’m running it again on a separate computer.
Even with all its drawbacks, there is something really nice about being able to run different applications over Tor, VPN or plain internet simultaneously, the ability to isolate non-safe binaries and being able to backup your VMs easily.
I wish a similar distro would be made based on KVM so that the standard kernel could be used. It would be great for compatibility.
> being able to run different applications over Tor, VPN or plain internet simultaneously, the ability to isolate non-safe binaries and being able to backup your VMs easily.
These are all possible using light containers. For example, on FreeBSD I will spin up a jail which runs wireguard, and then I'll bridge that to another a jail. That 2nd jail is running entirely off wireguard without any other way to access the network. Since it is a jail, it is isolated. And backing up is as simple as a zfs snapshot and zfs send. I assume the same is possible on Linux.
> I wish a similar distro would be made based on KVM
There is an Issue for that: https://github.com/QubesOS/qubes-issues/issues/7051
Problem is not that nobody wants to do it, it is that GPU stack is exactly the kind of enormous driver surface Qubes exists to keep away from dom0. Second monitor does not really change that, somebody still has to trust the driver.
If it's `untrusted_filename`, it should at least accept the length of the string as an argument.
I'm still very impressed by qubes, and glad I'm not such a target that I feel I need the level of opsec it affords (on all my laptops).
Maybe someday AI-assisted killchains will be so widespread that Qubes is the minimal level for the (few?) still-local users of compute.
I would not have copied anything from dom0 to any another qube, the impact is low.
Mini tangent: Could someone explain to me why Qubes is used for security, when (from what I understand) Jails on BSD is significantly more robust/safe/has a much smaller exposed area? Is it just "everyone's using linux already; here's a safer linux"?
Qubes can be viewed as a Xen distribution, rather than a Linux distribution [1]. You may find the Qubes FAQ a good starting point (I'm reading it now because of your question, so thanks).
[1] https://doc.qubes-os.org/en/latest/introduction/faq.html#is-...
its kind of fascinating that all this paranoia falls to a shell escaping issue with system().
The worst part is that in 2020 they explicitly documented that the remote filename is attacker controlled,but still allowed it to reach system() That is C security 101: never pass untrusted input through a shell. This should have been caught in review!
Most security bugs are due to improper string validation and use.
That is sphincter tightening to read. Have to point out how amazingly well their bulletins handle communication. Clearly describes the issues, how users are to act, etc. in, what I feel, is an easy to grasp language, even if one’s not in the weeds that much. In fairness though, I do still have some past memories concerning Qubes architecture from way back, so maybe my assessment is wrong and this is still not that straight forward to grasp for most.
I do really like the following in the bulletin:
> Important: At this point, you still don’t know whether the key you just imported is the genuine QMSK or a forgery. In order for this entire procedure to provide meaningful security benefits, you must authenticate the QMSK out-of-band. Do not skip this step! The standard method is to obtain the QMSK fingerprint from multiple independent sources in several different ways and check to see whether they match the key you just imported. For more information, see How to import and authenticate the Qubes Master Signing Key.
It looks like Qubes is ran by people who take security seriously, which is refreshing.
I own a Qubes T-shirt which I bought in person at FOSDEM. The design on the T-shirt consists of many copies of the QMSK in hexadecimal. All of their merch is like this.
How well is QMSK protected from a serious attacker?
The worst part is that in 2020 they explicitly documented that the remote filename is attacker controlled,but still allowed it to reach system() This is C security 101: never pass untrusted input through a shell. This should have been caught in the review!
Isn't it the case for all bugs? If they appear in the production software, it means that they passed the review. And obviously bugs shouldn't pass the review, but that's easier said than done.
This should be made structurally impossible through type safety rather relying on code review.
This os is supposed to be run on bare metal AFIK for same reason
My username checks out
Another example for why system() is so dangerous to use.
I also don't understand why it needs to show the dialog in dom0. If you have the option to handle attacker controlled input on the unprivileged side, you should do that instead of putting a lot of logic on the privileged side.
The code is sloppy. They check existance of kdialog binary using full path; next step they rely on PATH search by shell. If would've been much safer to just do execve directly.
> why it needs to show the dialog in dom0
I think it's the "secure screen" that cannot be manipulated by the malware in a VM. I'd expect a password entry dialog to also be handled like that.
If you're going to put the graphics and NIC into separate VMs, surely the secure screen can be another of those semi-privileged VMs rather than part of dom0
It's for an the "File copy/move error" error dialog.
it's kinda doomed at this point
It's kinda futile to seek any kind of security in the modern scene of 2026. Kind of disappointing.
Peculiar stuff. I'm always skeptical of these security Linux distributions, but this bug is so bad that it seems like an infiltration of Qubes at best or Qubes being a honeypot at worst.