Active incident at Dreamhost forcing password reset

(dreamhoststatus.com)

164 points | by greenchair 2 days ago ago

79 comments

  • duskwuff 2 days ago ago

    Reading the tea leaves: given that the outage started with "connectivity issues affecting the File Manager", which was an install of Monsta FTP [1] - a commercial PHP application - I suspect the file manager got popped by one of the recently published PHP vulnerabilities, and users' FTP credentials may have been exfiltrated.

    [1]: https://www.monstaftp.com/

    • traeregan 2 days ago ago

      20 year customer here. This morning I asked their support chatbot if this is related to the PHP vulnerabilities and it assured me that's not the case. I'm skeptical.

      These status updates from the PHP incident would also have us believe that they were never directly affected: https://www.dreamhoststatus.com/pages/incident/575f0f6068263...

      They're being awfully vague about this issue so far. 8 hours ago I submitted a ticket requesting more information and they've not responded yet. I guess that they're still in the thick of it.

      • pixl97 2 days ago ago

        Why would their support chatbot know?

        Having worked in support for years quite often higher layers of the company will give T1 support almost no information on what happened intentionally. That's pretty standard corporate CYA policy in litigious societies.

        • traeregan 2 days ago ago

          I didn't think that it would know but their support UX funnels you through it so that's where I started. I followed it up with a ticket.

      • nkrisc 2 days ago ago

        I would never expect first tier support to know, let alone ever admit, something like that.

      • nikanj a day ago ago

        Asking the chatbot is as useful as asking the pet parrot.

        • paranoidrobot a day ago ago

          You're being unkind to the parrot. Parrots are known for mimicking others in their environment.

          One might reasonably expect it to scream/shout and swear if one mentions PHP.

    • MuffinFlavored 2 days ago ago

      hopefully their hashed/salted passwords, right? not plaintext?

      • duskwuff 2 days ago ago

        Internally, sure. But the application that was (probably) compromised was an FTP client, so it would have received the plain text of those passwords from users as they logged in.

        • spopejoy 2 days ago ago

          > the application that was (probably) compromised was an FTP client

          Thank you, this actually makes some sense. It's disturbing to get these emails with zero explanation what's going on, and very frustrating that user community has to guess. But PHP vuln in a web client intercepting user logins is the only thing that makes sense, otherwise that would mean passwords in the clear or some other 1990s-era nonsense.

        • trollbridge 2 days ago ago

          For those customers who are still logging in with FTP…

          As a precaution they reset everyone’s email passwords, presumably from an assumption people use the same ones for both.

          I’m surprise they still let people use FTP.

          • jacquesm 2 days ago ago

            I came across such a thing recently. The counterparty was a bank, and they point blank refused to update their processes so the people I talked to had no choice but to continue to use something they knew was hopelessly outdated. SFTP was released what, 3 decades ago or so? There is a good chance that whatever was on the other side of that upload was not capable of any kind of secure connection.

          • bawolff 2 days ago ago

            I dont really see a problem with ftp over TLS. The protocol might be dated, but it still works fine.

        • Alex13192 2 days ago ago

          [flagged]

    • 2 days ago ago
      [deleted]
    • juliusdavies a day ago ago

      DH patched all their PHP versions extremely quickly.

  • paulbjensen 2 days ago ago

    Dreamhost - gosh that brings back memories…

    • blocked_againnn 2 days ago ago

      000webshost anyone? Also .tk domains.

      • phantompeace 2 days ago ago

        Only real ones remember 110MB

      • LoganDark 2 days ago ago

        Hey, I used both of those! Those were fun days. When DotTK moved to "Freenom" though, that's when it really started to go to shit.

      • SteveNuts 2 days ago ago

        Freewebs!

  • joshstrange a day ago ago

    I got the email about this and realized I’ve been using them for just shy of 20 years. It was one of the original places I hosted things and where I hosted projects for other people. I haven’t actively used it years but migrating the last few things off has never been a priority. This might be the push I need to re-evaluate everything hosted there and shut down or migrate it all.

    • zoidb a day ago ago

      Yeah same here! I dont pay them any money but I still have an account there. For awhile I let my domains sit there until I realized that it was cheaper to move them to Cloudflare

  • xmddmx 2 days ago ago

    Super displeased with DreamHost's handling of this. I have multiple master accounts. One of them got the email around 12pm today (over 48 hours after the incident started) saying...

    "As a precaution, we have expired the passwords for some email and FTP/SFTP users on your DreamHost account. Their old passwords no longer work, and no new password has been set."

    But none of those email passwords were actually expired - all were working. This is confusing - did they only expire passwords that were under threat? If a password wasn't expired, should it be trusted?

    The same email suggests you change all email passwords you control.

    Another account received no email warning from DreamHost so I assumed it was fine, but 7 hours later it also received the same generic email.

    My complaints:

    - sucks that it took them 48 hours to identify the breach - perhaps understandable

    - but once the breach was identified, how the fuck does it take them over 7 rolling hours to email their customers? That's terrible.

    - confusing messaging: DreamHost are claiming that passwords are expired, but in testing multiple accounts, all of them are still working. Do I need to rotate these? (I probably will just for safety).

    I'm a 25+ year DreamHost customer and generally satisfied, but this feels like amateur hour.

    • dannyw a day ago ago

      For emails over 7 rolling hours, unless you're e.g. Google or Microsoft, you kinda have to, otherwise those systems will automatically shadowban (send-to-spam) you for excessive/abnormal email sending rates. It's not great.

      It also doesn't help that a lot of scammers use this, or fake cyber incidents, for phishing; so it's not like they have an easy solution for exempting "security emails" either.

      • xmddmx a day ago ago

        Sure, but it also took them 7 hours to email their own customers on the email infrastructure run by DreamHost themselves. Since they run the IMAP and SMTP servers, seems like they could have done that in seven seconds.

        Of course, if they are nuking everyone's email password that doesn't help for notification purposes -- but as of this morning, all my email logins are still working fine, so it seems they didn't actually do that? Confusing.

        • pushcx a day ago ago

          Yeah. And if it's not all passwords under the account, the email should've had a list of the affected ones. Instead Dreamhost says both that they've already deleted the at risk passwords, and also that customers need to reset all passwords. Maybe when they mean "reset all" they meant "reset all [the deleted] passwords", but being so unclear about scope and remediation steps is a painful mistake.

          A single sentence like, "If a password wasn't reset as of (timestamp), we don't believe it could have been at risk." would go a long way to fixing these incident comms, if that's the case.

          • xmddmx 20 hours ago ago

            Agree. And this is all made worse by the fact that there seems to be a bug in iCloud keychain syncing - I can change the password on my mac, verify it works, but when I go over to iPhone or iPad, the old password is still being used. Not sure if there's a way to force the sync process on both ends?

  • lexwhite 2 days ago ago

    This has been a real annoyance for me lately since I run a lot of my music production business out of my website lexwhite.net hosted on Dreamhost and Dreamhost VPS. I love and trust Dreamhost a lot, though, as they have been an exceptionally solid company that I have worked with for, dang, nearly 20 years now I think! All of this new AI stuff has been exposing vulnerabilites at a rate that is hard to keep up with for anyone. We just have to really harden our cybersecurity as much as possible now.

  • hmokiguess 2 days ago ago

    I migrated all my stuff from dreamhost about 5 years ago, got this email today and felt validated from having made that decision.

    • rationalist 2 days ago ago

      Dreamhost is no longer the hosting company that I originally signed up for. I'm also looking for a different host.

      I might have been willing to stick it out, but their panel/website-management is so slow.

      • elevation 2 days ago ago

        I love their shared hosting; I have a $100/year account with dozens of little low traffic apps in their own subdomain/user isolation. They maintain the OS, I maintain the app, I can pull code with git and even compile a python interpreter if I need.

        I love that I have infinite/unmetered email accounts. In practice I run 3-4 accounts on 2-4 domains, but they don't nickel and dime you on it, I've literally setup 1200 forwarders in the past. They have a good sending reputation.

        What other provider does this as well?

        • hmokiguess a day ago ago

          have you tried digital ocean? some friends I have swear by them as good

          • elevation 11 hours ago ago

            Yes, they're in my rotation as well (I never want to have just one provider.) But I liked that DH would maintain the OS, the shared hosting performance never bit me (my sites are all highly performant, no wordpress dogs) so all of my time would be in patching vulns. DH giving me no-root access to a hardened, performant box is ideal all around.

      • dieselgate 2 days ago ago

        Porkbun has been fine, used dreamhost for years prior

    • mistersquid 2 days ago ago

      > I migrated all my stuff from dreamhost about 5 years ago

      May I ask to where you migrated?

      • hmokiguess a day ago ago

        I re-architected most of what I had to fit in a serverless deployment model on AWS and now they run well within the free-tier in AWS + Neon DB

  • whyage 2 days ago ago

    What's going on with Dreamhost lately? Is it falling apart?

    • AznHisoka 2 days ago ago

      I havent seen anyone talk about it for the last 10-15 yrs. I would have guessed it fell apart already

      • PaulDavisThe1st 2 days ago ago

        They abandoned all support for email lists a couple of months back. Will likely move after 20+ years on DH.

        • trollbridge 2 days ago ago

          Chat support works well. Email is nonexistent.

          • PaulDavisThe1st 2 days ago ago

            I don't mean "support for using mailing lists". I meant "they no longer provide the services (i.e. mailman) to run mailing lists"

            • trollbridge 2 days ago ago

              Nobody wants to run a service like that anymore.

              • bigiain 2 days ago ago

                i wouldn't be so sure. I know I move in privacy and "anti big tech" circles, but I (and a lot of people I know) am moving a lot of my personal/recreational internet time to rss feeds of personal websites, mailing lists, and Mastodon. And there's a fair bit of discussion and small but growing use of private and very small group federated Mastodon hosts.

                Having said that, very little of that is being done by the sort of people who'd host it on a service like Dreamhost.

                • cowlevel 2 days ago ago

                  Mailing lists are inherently dangerous as you can be marked as a spammer.

                  • LoganDark 2 days ago ago

                    Everyone (and I mean everyone) is a spammer by default these days, so this is no problem in practice.

              • james_in_the_uk 2 days ago ago

                Topicbox.com, from the team behind Fastmail

              • danielmeskin 2 days ago ago

                I mean, clearly the person you’re replying to does

    • trollbridge 2 days ago ago

      No, they’re still doing well.

      Support is primarily via chat these days.

      Pricing wise they don’t beat Herzner anymore but are somewhat competitive with Digital Ocean.

      I am gradually moving everyone to my own self hosting mostly for supportability reasons. We want to react in seconds, not hours, when a customer has an issue.

      • 99harry 11 hours ago ago

        I've been using Contabo, and so far, the service has been quite stable and professional. Updates are handled automatically, and I haven't encountered any major issues DigitalOcean seems a bit slower and more expensive in comparison, especially when Contabo offers a VPS with 4 CPU and 8 GB of RAM for just 4.50 euro/month.

      • beej71 2 days ago ago

        I'm still at DH (have been for 20 years), but taking a peek at Hetzner... seems compelling at 30% cheaper than I pay now.

        • cowlevel 2 days ago ago

          Just be warned Hetzner will delete your account at the slightest sign of controversy. It's fine for Minecraft but I wouldn't want to host a political website there.

    • forgotmypw17 2 days ago ago

      Working pretty well for me.

      • donohoe 2 days ago ago

        Same. Into my second decade with them. Great value, no real complaints.

  • monster_truck 2 days ago ago

    I used them for entirely too long, and only continued to do so because a classmate in high school used my referral code to sign up for the account that would become his career of selling and managing websites for local churches.

    I could not believe what the hell they were charging for a year of hosting with a dedicated ip (which they at the time required if you wanted https, real scumfuck move), so I didn't pay.

    I guess I hope they actually deleted my shit because I sure didn't get this email

    • rstupek a day ago ago

      How long ago was that because at the beginning you did have to dedicate an IP for https prior to SNI

  • forgotmypw17 2 days ago ago

    Ah, I noticed this yesterday, figured it was just me.

  • therein 2 days ago ago

    Now that's a blast to the past. What's next, Rapidshare.de is having issues?

    • tuwtuwtuwtuw 2 days ago ago

      Angelfire shut down this year so not them at least.

    • pixel_popping a day ago ago

      Remember when they put that 25GB daily limit vs MU unlimited, that was the start of its death.

    • dvdyzag 2 days ago ago

      Dang, memory unlocked

      • jamesfinlayson a day ago ago

        Counting down until you can download again...

        I do not miss those days.

  • dgiorgaras a day ago ago

    [flagged]

  • hvalero2001 2 days ago ago

    [flagged]

  • bone_421 2 days ago ago

    [flagged]

  • kagouhui 2 days ago ago

    [flagged]

  • sunjester 2 days ago ago

    [flagged]

    • jawns 2 days ago ago

      I've been a user for more than 20 years. With their shared hosting plan, which was about $5/mo. when I joined and is now $13/mo, I've been able to host my personal blog and professional site, as well as probably 3 dozen other sites over the years, including several that have hit the HN front page and never got hugged by the squeeze of death.

      Honestly, compared to some of the professional environments I've worked in, it's a joy to have an idea, create a few PHP files, and in a matter of seconds have that idea live.

    • rpdillon 2 days ago ago

      Been a customer for decades, love them.

      • 0898 2 days ago ago

        Do you remember the cool newsletters they used to do?

    • ceejayoz 2 days ago ago

      A shitload of small business sites that were built years ago and haven't been touched since.

      • SoftTalker 2 days ago ago

        If they are static sites that shouldn't be a problem. Many are probably not though.

        • ceejayoz 2 days ago ago

          My suspicion is at least half are unpatched WordPress installs.

          • Jgrubb 2 days ago ago

            My suspicion is closer to 95%

            • Kwpolska 2 days ago ago

              And the remaining 5% are unpatched Joomla and Drupal installs.

    • pockybum522 2 days ago ago

      Me, for all the reasons listed so far in comments

    • Arainach 2 days ago ago

      It's cheap and reliable with predictable costs. It hasn't caused me any problems in a decade. Most of the websites I want to host don't need cloud anything, they're statically generated.