Ssl.com: DCV bypass and issue fake certificates for any MX hostname

(bugzilla.mozilla.org)

216 points | by xPaw 3 days ago ago

65 comments

  • btown 3 days ago ago

    Public service announcement: CAA records exist and allow you to whitelist the CAs you trust to issue certificates for your domain.

    https://letsencrypt.org/docs/caa/

    You can use https://www.entrust.com/resources/tools/caa-lookup (or e.g. `dig caa paypal.com`) to see if any domain is protected.

    https://isc.sans.edu/diary/26738 is a cautionary study from 2020 indicating only 3% of the Alexa top 1M had CAA records. And just now, I've seen numerous news and government sites that do not have CAA enabled... making them vulnerable to issuance bugs like this on CAs they may never have heard of, and thus making their readership/constituencies vulnerable to misinformation and fraud, especially in the context of a potential multifaceted attack against router infrastructure to perform MITM attacks at scale.

    Of course, you'll want to make sure you don't accidentally disavow an important subdomain where an engineer used a different CA than your usual suspects. But looking at all historic issuers for your domain hierarchies on transparency logs using e.g. https://crt.sh/ might be a good place to start.

    It's also good to monitor certificate transparency logs, but then the onus is on your security team to react if an incident occurs. Proactive controls are vital as well, and IMHO CAA avoids many of the downsides of pinning.

    • tgsovlerkhgsel 3 days ago ago

      The CAA whitelist is still enforced by the CAs themselves, so a malicious, compromised or buggy CA could ignore it. You still have to monitor CT. CAA mostly does two things:

      1. It makes sure that nobody accidentally issues a cert from another CA (giving you better control, avoiding the "an engineer used a different CA" scenario, and meaning that if you see a cert from another CA, you know it's something Very Not Good).

      2. It gives you a chance that an attacker able to bypass some but not all controls on a crappy CA won't be able to use that CA to get a cert for your site (if they don't manage to somehow also bypass the CAA check).

      I'm not sure whether CAA would have prevented this CA from issuing for this domain. I think it's more likely than not, but not certain, that it would have helped in this case.

      • jchw 3 days ago ago

        Unfortunately the best solution there was for this problem was probably HPKP, which fell out of favor years ago. Would be nice to have some kind of solution for this some day; I think it would compliment CT very well.

      • mcpherrinm 3 days ago ago

        CAA plus DNSSEC also provides significant defense against some types of attacks on domain validation.

    • agwa 3 days ago ago

      Domain owners may find my CAA record generator <https://sslmate.com/caa/> useful, as it can automatically generate a CAA policy that covers all the certificates found in CT logs for your domain. It's not always obvious how to translate from issuer name to CAA domain (due to white labeled intermediates); my tool consults CCADB data to determine the correct CAA domain.

    • LinuxBender 2 days ago ago

      It may also be worth mentioning that when using CAA and also using something like LetsEncrypt one can specify which account is permitted to create and update certs and which method is approved DNS in this case. [1]

      Example using DNS validation:

          0 iodef "mailto:domainowner@example.net"
          0 issue "letsencrypt.org; validationmethods=dns-01; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/xxxxxxxxxx"
          0 issuewild "letsencrypt.org; validationmethods=dns-01; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/xxxxxxxxxx"
      
      Only useful for non-rogue CA's of course and maybe some day crt.sh will be less after-the-fact on all browsers and API clients.

      [1] - https://www.rfc-editor.org/rfc/rfc8657

    • cortesoft 3 days ago ago

      So CAS records are supposed to keep a CA from issuing a certificate if the CAA record exists and doesn't have that CA.

      However, this is relying on the CA to properly check the record. If the CA has a bug where it isn't validating properly, they could also fail to check the CAA properly. Also, this doesn't help against a malicious or compromised CA.

    • m_sahaf 3 days ago ago

      I always wonder who/what checks if CAs respect CAA. I know some browsers now check the certificate transparency log, but are there any that check the CAA record against the issuer of the certificate?

      • agwa 3 days ago ago

        No, because the CAA record only has to be in place at the time of issuance, rather than the whole lifetime of the certificate.

        Even if the semantics of CAA were changed, the challenges described in paragraph 3 of this post would apply: https://www.imperialviolet.org/2015/01/17/notdane.html

        • londons_explore 3 days ago ago

          > No, because the CAA record only has to be in place at the time of issuance, rather than the whole lifetime of the certificate.

          could we change this? Ie. if the CAA record disappears, it would be a reason to revoke a certificate?

          Then 3rd parties could scan transparency logs and CAA records and flag discrepancies.

          • mcpherrinm 3 days ago ago

            It would be possible to change, though that would be a pretty big change.

            Personally I think this is another good argument for short lived certificates and reducing reliance on revocation systems.

      • 9dev 3 days ago ago

        Wouldn’t that be an obvious quick win?

  • 0x0 3 days ago ago

    So I guess you couldn't get certificates for any random (MX) domain, only for those where you can obtain an inbox / user account. Still really bad, especially for things like gmail.com, but also larger enterprises. Intense.

    • remram 3 days ago ago

      Or any domain for which you can read an email sent to an inbox. I remember a few years ago an attack where the attacker would read email because a ticket would be created for incoming emails, and he could guess the next ticket ID to read it. A lot of platform that aren't email providers still allow emails in (e.g. GitHub, GitLab). This looks like a rather widely-applicable attack.

      edit: I was thinking about this: https://news.ycombinator.com/item?id=41818459

    • tptacek 3 days ago ago

      It is unlikely that SSL.com would issue a certificate for any major mail host; it would be malpractice for them not to have some kind of exclusion list.

      Issuing a Google certificate is a good way to get your whole CA killed.

      • AdamJacobMuller 3 days ago ago

        Sure, gmail.com might be excluded, but its still a massive hole for a few reasons.

        This would affect ANY email provider who offers public email addresses. While I agree gmail.com is probably excluded (and maybe this doesn't bypass CAA -- maybe it does) there's a whole additional surface of anyone who has an email at any big enterprise getting a certificate for their domain.

        Even if I work at google.com, therefore have a google.com email, I should absolutely not be able to get a certificate for google.com just by getting an email at that company.

        I doubt it's even /that hard/ to buy an email account at a big company like that in the underground world, it seems like they are valuable generally and any company with 200k employees is going to have some leaks. This massively increases the attack surface of a simple leaked email account (which might otherwise have very little or no access).

        Crazy crazy oversight that has huge implications and is so easy to carry out that I would not be surprised if this was actually exploited by bad actors.

        • londons_explore 3 days ago ago

          plenty of companies have mailing lists which are listname@companydomain.com

          Getting on those lists is often easy. Same with support ticketing systems, etc.

          • thayne 3 days ago ago

            Someone else on the list might figure out something was fishy when the verification email came through though.

      • bawolff 3 days ago ago

        > Issuing a Google certificate is a good way to get your whole CA killed.

        Surely what happened here is a good way to get your CA killed? The linked bug seems pretty bad.

        • tptacek 3 days ago ago

          Less clear on that. Bugs happen. I'm not an expert on browser root policies.

          • thayne 3 days ago ago

            From what I understand one of the factors is how often things like this happen, and how well they handle it when it does.

        • agwa 3 days ago ago

          Historically, singular domain validation bugs have not killed CAs.

      • unit149 3 days ago ago

        [dead]

    • cperciva 3 days ago ago

      Or potentially one where you could subscribe to a mailing list. Which includes a lot of very important open source software projects.

    • mukesh610 3 days ago ago

      Even then, use of a DNS CAA record should mitigate this, right?

      • AdamJacobMuller 3 days ago ago

        Maybe?

        I wouldn't assume that the bug doesn't bypass CAA checking.

        Very important question to answer.

      • jsheard 3 days ago ago

        Yeah - unless you're an actual SSL.com customer, in which case your CAA records would allow it. That's a much smaller blast radius at least.

    • mcpherrinm 3 days ago ago

      I couldn’t reproduce the attack with a pair of my own domains, so I think it might be even narrower in scope than the initial post suggests. But I suppose we will just have to wait to see what the CA says.

      • thayne 3 days ago ago

        > Out of an abundance of caution, we have disabled domain validation method 3.2.2.4.14 that was used in the bug report for all SSL/TLS certificates while we investigate.

        I think they have already addressed the bug.

        • mcpherrinm 3 days ago ago

          I tested before they acknowledged or disabled the method (I was able to use a 3.2.2.4.14 validation the “normal” way)

  • cmeacham98 3 days ago ago

    This is a ... pretty serious oversight.

    But at least it initially appears SSL.com is taking it seriously, we'll have to see what the report says.

  • jenny91 3 days ago ago

    Wow... this is the most serious TLS issue I've seen since following these things.

    • tptacek 3 days ago ago

      It's bad, but the WebPKI of the oughts featured CA certificates issued to random big enterprise IT teams that could simply issue arbitrary certificates. We've come a long ways.

  • AdamJacobMuller 3 days ago ago

    > We will provide a preliminary report on or before 2025-04-21.

    Bunch of engineers just got their easter weekend ruined. Sucks.

    • sneak 3 days ago ago

      [flagged]

  • CrimsonRain 3 days ago ago

    I guess they can check logs and find how many times this has been abused already? Can we trust them to release full transparent report?

    • bawolff 3 days ago ago

      > Can we trust them to release full transparent report?

      Generally browser vendors take a pretty dim view of CA's not being transparent when bad things happen. Given the seriousness of this issue,i suspect being aggressively transparent is their only hope of saving their business.

    • thenickdude 16 hours ago ago

      They've released their report now, 10 further certificates were mis-issued:

      https://bugzilla.mozilla.org/show_bug.cgi?id=1961406

    • toast0 3 days ago ago

      I would expect them to be able to report on certificates issued based on this validation method. That's a basic CA capability and other CA incidents often include these kinds of reports.

      Depending on what was logged during the validation, it might be tricky to determine if it was abuse or not. If the DNS content wasn't logged, they could pull a live record and report if the current record would support validation or not.

      My guess is that use of this method should be low... If you're updating DNS to add a TXT record, you might be more likely to add a direct verification value rather than an email. But that's speculative; I'm not a CA, I've just been a customer of several... IIRC, I've validated domain control by controlling postmaster@ (or the whois address when that was public) or adding direct TXT verification records or ACME http validations.

      • agwa 3 days ago ago

        This method may be more popular than you'd think, since it only requires the TXT record to be published once, whereas using the DNS method requires periodically updating the DNS record. Yes, that can be automated or delegated, but for a legacy/manual/dysfunctional organization, email to TXT record contact is an easy alternative to the now-banned email to WHOIS contact method that they were likely using previously.

      • thayne 3 days ago ago

        You could at least narrow it down to certs with multiple domains, since it sounds like the email domain was added as an additional domain.

    • thayne 3 days ago ago

      All such certs should be in transparancy logs, so I think it should be possible for a third party to verify.

    • aaomidi 3 days ago ago

      They will need to most likely do a full mass revocation at this point.

    • gruez 3 days ago ago

      >We will provide a preliminary report on or before 2025-04-21.

  • Galanwe 2 days ago ago

    The whole PKI concept is dead anyway. Users have been trained to bypass certificate warnings by CA store wars between browsers and OSes, MitM corporate SSL proxies a-la ZScaler / Intune, corporate self signing CAs for intranets and whatnot, blindfolded public CAs giving certs to whatever.

    • peanut-walrus 2 days ago ago

      What? When was the last time you had to bypass a cert warning because of "CA store wars" (whatever that means)? What examples can you give for public CAs giving certs to "whatever"?

      • Galanwe 2 days ago ago

        > When was the last time you had to bypass a cert warning because of "CA store wars" (whatever that means)?

        Essentially all the time for the last 10 years...

        Did you ever try to deploy a website with a certificate from a non public CA? Like, say, your company CA?

        If you want it to be valid for Java users, you will have to store your CA cert on the Java trust store.

        Want it available for users of Firefox ? Store it in the OS certificate store.

        Want it available for Chrome users? Store it in the Chrome certificate store.

        Want it available for Python users? Add it to certifi.

        And so on.

        No single piece of software validating certificates agree on a single CA certificate store.

        So, essentially, no company out there supports all these stores, and you just train users to bypass these warnings.

        > What examples can you give for public CAs giving certs to "whatever"?

        There have been dozens of CAs removed from widespread trust stores for failing to do proper diligence or reporting leaked keys.

        Not only that, but essentially I never myself to do any kind of diligence for whatever certificate I requested from public CAs beyond proving I had TXT records update powers at some point in time.

        I'm not even mentioning fortune 500 websites running with expired certs.

        • LiamPowell 2 days ago ago

          > So, essentially, no company out there supports all these stores, and you just train users to bypass these warnings.

          This just sounds like a problem with your company. The barrier for getting certs from a public CA is lower than ever now that Let's Encrypt and others exist. If you really must have a non-public CA then your company needs an IT team that can properly manage that.

          This isn't an issue for normal users.

          • Galanwe 2 days ago ago

            > This just sounds like a problem with your company.

            I have seen that pattern in enough companies to be convinced this is widespread.

            > The barrier for getting certs from a public CA is lower than ever now that Let's Encrypt and others exist

            I don't think you understood what I'm talking about. Public certificates are cute for your public website, but any sizeable company is _also_ hundreds of internal websites and services, especially for the non IT departments. Think legal, compliance, accounting, HR, etc.

            Most companies use a private CA for these, and that makes sense:

            - You want subsigning CAs for your VPN, contractor services, websites, teams, etc.

            - You want private DNS to private IPs (lots of ISPs won't even serve your private IPs through their DNS caches)

            - etc

            > If you really must have a non-public CA then your company needs an IT team that can properly manage that.

            On the contrary, managing private CAs is what most companies do _well_. What they don't (and honestly nobody can) do well is distribute CA certs to user devices. This is often not done right on work devices, but BYOD made it even worst. No company can distribute its CA certs on the hundreds of different stores that one could think of, so after 2 years, some benign change of default corporate browser for users ends up with them learning to auto bypass certificate warnings.

            • LiamPowell 2 days ago ago

              > You want private DNS to private IPs

              Nothing stops you getting a cert while pointing your DNS records to internal addresses. The DNS-01 challenge exists to serve exactly that kind of configuration.

              > lots of ISPs won't even serve your private IPs through their DNS caches

              I have never seen this, could you give an example? However, if this is an issue then there's nothing stopping you from just using your public DNS for DNS-01 challenges and using your internal DNS for everything else.

              It is also impossible for your ISP to do this if you're using DoH or DoT, which you really should be, especially if you already know that your ISP is messing with DNS traffic.

              > You want subsigning CAs for your VPN, contractor services, websites, teams, etc.

              You can't do this, but you can have your own ACME server that forwards requests to a public CA if you really need to let different teams manage their own certs. A better option is probably to use one of the paid CloudFlare tiers where you can create scoped API keys that provide DNS editing access scoped to a subdomain, or you could of course host your own DNS server or find a different DNS provider that offers this service.

              • cj 2 days ago ago

                We have a team who uses a ".dev" domain for local development (with a publicly issued SSL cert), with an A record of 127.0.0.1.

                We had someone new join the team and couldn't get the dev environment working. Turns out his ISP's DNS wouldn't resolve to an internal IP. Simple fix was updating his system DNS away from his ISP. We only saw this happen to one person, so wouldn't say it's common but it happens.

                • andrewaylett 2 days ago ago

                  That's protection against DNS Rebinding attacks -- you don't want external domains to be able to make same origin requests to internal domains, and while it suffices to only block changing resolution, that's harder than blocking internal IPs altogether.

              • devrand 2 days ago ago

                You could also use the `_acme-challenge` CNAME record to delegate cert acquisition, assuming you're using separate subdomains for each.

  • thayne 3 days ago ago

    Have they started revoking invalid certs?

    • voxic11 3 days ago ago

      You can see the cert was revoked here https://crt.sh/?id=17926238129

      • progbits 3 days ago ago

        Unclear who revoked that but I think it likely was the reporter who discovered the bug. They only needed it issued & logged as evidence, and would be good practice to revoke immediately.

        • mcpherrinm 3 days ago ago

          The certificate remained unrevoked in OCSP until after SSL.com acknowledged the issue, so I don’t think the reporter was the one who had it revoked.

          It is also possible I was being served a stale/cached OCSP response.

  • 0xbadcafebee 3 days ago ago

    And remember kids: there's hundreds of CAs, they all implement validation independently, and you just need one to do one of the three validation methods wrong to make any cert you want. And there's two dozen different attacks that work aside from bugs in validation. Cert validation is swiss cheese.

    But there's a fix: have the registrars confirm authority to issue certs using a public key uploaded by the domain owner. The CSR contains a request signed by the same domain owner key. The CA sends the CSR to the registrar, the registrar verifies it was signed by the same key they have, then they send back a newly signed object (that the eventual-https-end-user can later confirm was signed by the registrar, so everybody knows that every step of the way was confirmed cryptographically). This ensures a single secure authorization by the actual domain owner, verified by the registrar of the domain. All you have to change is how the CA validates, and the registrar needs to handle an extra step or two. Solves 95% of the vulnerabilities.

    ....but nobody's going to do that, because the fact that Web PKI is swiss cheese hasn't threatened a big enough business yet. Once money or politics is threatened, they'll fix it.

    • peanut-walrus 2 days ago ago

      CAA account binding is basically this. Not cryptographically verified but similar idea that the CA confirms you possess a secret (account) before issuing. https://www.rfc-editor.org/rfc/rfc8657

      • 0xbadcafebee 2 days ago ago

        Nope, it's not the same. The point of having the Registrar involved is to side-step the problem of validating a cert request is allowed to request the cert. All the CA validation methods are supposed to be verifying your authorization to request a cert, but they don't do that.

        They instead verify your authorization to control DNS records, or IP space, or an e-mail address. And there's dozens of exploits to compromise each of those. And they can be chained. And they can be CA-specific.

        That's not domain authorization, and each of those verification methods lacks cryptographic authentication. Only the Registrar controls the domain, so that is the only way to know that the request is genuinely authorized. We're playing a game of telephone, it's not secure, and it's unnecessary. Just get the registrar involved.

    • cmeacham98 3 days ago ago

      With your solution, we end up with the same problem just one layer down. Browsers have to contain a list of 'trusted' registars, and an attacker only needs to find one buggy registrar that will incorrectly sign for a domain the attacker doesn't own.

      • 0xbadcafebee 2 days ago ago

        That's a much simpler problem to solve than the current one. One attack vector to cover, one set of organizations, one trust list. It's definitely no worse than our current predicament.

        Basic math shows how much safer the new model would be:

          - Assume there are 350 CAs, 3 validation methods, and 12 kinds of exploit per validation method (there are more in some combinations but for simplicity I'll say 12).
            (350 x 3 x 12) leaves *12,600* possible attack vectors.
        
          - Now assume there's 2,650 domain registrars, 1 validation method, and 1 kind of exploit.
            (2650 * 1 * 1) leaves *2,650* possible attack vectors.
        
        So 4.75x fewer possible attack vectors.

        Add to this that with only 1 validation method and 1 feature to support, that's way less code, which means much fewer bugs.

        Add to that a cryptographic chain of proof that the key of the domain owner was used to sign the request all the way, which we don't have today.

        • cmeacham98 2 days ago ago

          This math only looks good because you're adding an arbitrary unsubstantiated 12x multiplier to the CA numbers.

          Of course, neither of us have actual numbers but my gut instinct is that registars are probably about as secure if not less secure than CAs, and there are nearly 10x as many of them.

          • 0xbadcafebee a day ago ago

            Registrars are less secure because CAs have to follow a strict guideline for security. We could just apply that same standard to the registrars. It would be easier for them to follow since it would be far simpler to use 1 validation method, and using cryptographic verification it's easier to automate validating that it was done successfully.

            The 12x multiplier comes from the number of attack vectors to validation methods. The whole system is public knowledge; if you know how each part works, and you know about all the possible security exploits out there, you can just count them.

            Here's a brief list off the top of my head, it's not exhaustive:

              - DNS validation. The CA looks up TXT records of a given subdomain.
                Attack 1. DNS cache poisoning
                Attack 2. Compromise the credentials of a DNS admin
                Attack 3. Zero-day exploit in the DNS server
                Attack 4. Zero-day exploit in the DNS web management interface
                Attack 5. Incorrectly configured DNS zone transfer settings.
                Attack 6. BGP spoofing attack on the target DNS nameserver.
                Attack 7. BGP spoofing attack on the CA's DNS resolver.
            
              - HTTP validation. The CA requests a specific URL over HTTP and verifies the contents.
                Attack 1. MITM the HTTP request/response. (Can be done anywhere across the network, from the CA internal network to the target internal/external network)
                Attack 2-8. Every single DNS attack. You just replace the A/AAAA record when looking up the target HTTP host, with an attacker-controlled http host.
                Attack 9. BGP spoofing attack on the IP of the target HTTP host.
                Attack 10. Zero-day exploit in the target HTTP server.
                Attack 11. Stealing credentials to remotely login to the target HTTP server.
            
              - Email validation. The CA sends an e-mail to the domain and confirms the reply.
                Attack 1-7. Every DNS attack.
                Attack 8. BGP spoofing attack on the IP of the target MX host.
                Attack 9. MITM the e-mail. This one is extra easy as intermediate unencrypted relays are common/expected.
                Attack 10. Steal the credentials of the mail server admin to remotely log in and intercept/fake emails.
                Attack 11. If it's a site that allows users to register an email address, there are six different email addresses they can try to register; if one works, they can use that to validate certs for that domain.
                Attack 12. Zero-day exploit in the mail server software.
                Attack 13. Zero-day exploit in the mail server's web management interface.
            
            And this is just the run-of-the-mill exploits most experienced hackers can pull off remotely. Haven't gotten into more advanced things like supply-chain attacks, timing attacks, protocol/algorithm flaws, espionage, social engineering, etc.

            The big problem is that since validation uses no cryptography whatsoever, all the attacks are fairly trivial. A BGP attack is so easy that it happens by accident on a monthly basis. MITM is easy. Stealing credentials is easy (ask any botnet admin). Attacking DNS is easy (most people don't uses DNSSEC and even if they do their clients/resolvers aren't enforcing validation).

            Do an end-run around all this bullshit by asking the people who actually know who owns the domain (the Registrar) to validate it cryptographically.