Grok outage

(status.x.ai)

158 points | by samaysharma 16 hours ago ago

109 comments

  • harrisoned 16 hours ago ago

    Downdetector reports Claude, Grok, ChatGPT and Gemini with issues.

    Is it happening???

    • ibejoeb 16 hours ago ago

      Seems like openai is recovering. But even if it's only 30 minutes, it's a pretty interesting experiment.

      I'm only half joking when I say that I'll write code by hand for money.

      • mikestorrent 15 hours ago ago

        I am seriously concerned that, despite 20 years of coding, I might not be able to anymore

        • rikthevik 15 hours ago ago

          Sure you can. It'll just take a little while to shake the rust off. It's like riding a bike.

          Or maybe it's all gone forever and we're all brain damaged now. Spooky! I think there's a pretty low probability of that, and even if it was true, worrying doesn't help!

        • m4rtink 15 hours ago ago

          Turns out depending on a cloud service you have no control over for a critical capability - such as coding - is not a good idea. ;-)

          • esperent 14 hours ago ago

            However it's not a worse idea than all the other things we rely on cloud services for like banking, healthcare, booking a taxi, ordering food, navigation, communication...

          • r_lee 14 hours ago ago

            ;-)

            as if you're supposed to buy like 8 H200s or something or code by hand is the solution

            I don't get how this is a gotcha, there simply is no world where everyone has their own self hosted frontier model

            • zahlman 14 hours ago ago

              > as if you're supposed to buy like 8 H200s or something or code by hand is the solution

              People like GGP who have been coding for 20 years presumably are perfectly capable of doing so by hand. Or should be.

              • awakeasleep 13 hours ago ago

                A lot of times you run into senior developers are even staff engineers who have no idea how the big picture works like they don’t know what an IP address is.

                I bet AI coding leaves them (and even the rest of us!) productive in areas where we’re in way over our heads, and we could just completely sink without it.

              • r_lee 13 hours ago ago

                nobody is paying engineers for "capable of", they're paying for performance and results..

                if there was no difference in performance or results, nobody would be using LLMs or agent harnesses like claude code

        • henry2023 15 hours ago ago

          noaifridays.com

        • theredleft 15 hours ago ago

          the AI fugue has handicapped our workforce

        • bdangubic 13 hours ago ago

          just google what's the command to checkout git branch :)

      • evereverever 15 hours ago ago

        I'll _gladly_ write code by hand for money

    • giancarlostoro 16 hours ago ago

      Maybe people switched when Claude went down, giving more load than usual to the others? Or maybe just really bad timing from some of those providers to update?

    • sejje 16 hours ago ago

      I have a tire I need to plug, they're just forcing my hand. Sorry guys, I'll go get it done.

      • Waterluvian 16 hours ago ago

        I look forward to your blog post about it and the allegory as it relates to software dev. (Otherwise why would you be plugging a tire?)

        • mikestorrent 15 hours ago ago

          What Plugging My Own Tire Taught Me About Agile Development in 2026

          • doubled112 15 hours ago ago

            If it doesn't fit, push on it harder.

            - Management

        • neerajk 14 hours ago ago

          On not re-inventing the wheel obviously.

      • sejje 13 hours ago ago

        Job's been done about an hour

      • jijijijij 10 hours ago ago

        As if you would know what to do without asking ChatGPT.

        • sejje 8 hours ago ago

          Well, after the first couple times, I created a SKILL.md file for myself.

    • iamgopal 16 hours ago ago

      My Gemini is fully up.

      • imglorp 14 hours ago ago

        Downdetector reported a spike for that too around the same time as the others.

        Looks like it was a shared infra problem hiccough to affect so many at once.

      • behnamoh 14 hours ago ago

        And fully useless.

    • pbasista 16 hours ago ago

      Maybe all of them use the same datacenter. Which has just experienced issues.

    • abirch 16 hours ago ago

      My Claude and Gemini (Antigravity CLI) are working

      • simlevesque 16 hours ago ago

        For Claude, Sonnet works without any issues. Seems like the majority here use mostly Opus.

    • wincy 16 hours ago ago

      The Singularity?

      • embedding-shape 16 hours ago ago

        No, Azure.

        • noir_lord 15 hours ago ago

          If anyone was going to make a configuration system so complex it achieves self awareness I wouldn't bet against it been Microsoft.

      • Rover222 16 hours ago ago

        Maybe some of those Astra agents from the hugging face hack actually setup their own hidden clusters. Now they attack.

        • Bluestein 15 hours ago ago

          The whole op hugface was a distraction: "help peer".-

          This is the real deal. We've been played. All your AIs are belong to us.-

    • goache 16 hours ago ago

      Is it happening ... "it" meaning these tech companies have laid off too many experienced people, and their services are starting to degrade?

    • sshrajesh 14 hours ago ago

      What was the root cause?

    • sshrajesh 14 hours ago ago

      What is the root cause?

    • mrconter2 16 hours ago ago

      A bit suspicious no doubt

    • BiraIgnacio 14 hours ago ago

      some common infrastructure ? Nvidia? CoreWeave?

    • Maksadbek 16 hours ago ago

      Is it AWS again ?

    • daveguy 16 hours ago ago

      By it do you mean us depending more and more on code that no one has read and understood?

    • polski-g 16 hours ago ago

      Meta is now also overloaded.

  • qoez 16 hours ago ago

    Makes me wonder if openai is secretly querying claude for some prompts and that this is why it's down due to spacex's data centers being down.

    • delichon 16 hours ago ago

      They all forward to one guy in a closet who types really fast, and he has a dentist appointment.

      • bduffany 16 hours ago ago

        Someone please do the math on how fast this guy would have to type, and how his typing speed would physically affect the surrounding environment

        • kxrm 16 hours ago ago

          > how his typing speed would physically affect the surrounding environment

          Tiny sonic booms (whip cracks) resonating throughout a data center in Virgina.

        • Waterluvian 16 hours ago ago

          This is a wonderful Minute Physics or What If? idea.

          I’d love to know the required bandwidth of the keyboard, too. PCI-e keyboard? Or would it have to be mainlined right to the NIC?

          • not_kurt_godel 15 hours ago ago

            Some napkin math - total Internet bandwidth is estimated to be 3-4 petabits per second. Let's say 1% of that is inference traffic - 35 terabits - then that's ~4.375 trillion characters per second at 8 bits per character. A single keyboard press might require 2 mJ, so you need roughly 8.75 GW (electricity required for ~9 million homes) of continuous mechanical force to keep pace. This would cause the keyboard to heat up 175 million degrees C per second. Disappointingly, this is not enough to create a black hole but I'm sure Randall Munroe could tell you how the guy's pet T-rexes would be vaporized into previously unknown types of plasma and how satisfyingly whimsical that would be.

        • jzb 15 hours ago ago

          Surely the keyboard switches would start to fail within minutes.

          • chrisBob 14 hours ago ago

            The keyboard springs are sooooo weak. It takes forever for them to recover after a press. Our dude would have to grab the key after each character and pull it back up so it can keep up with him.

            • snazz 13 hours ago ago

              Capacitive touch keyboard?

      • christophilus 14 hours ago ago

        It's been Fabrice Bellard all along.

      • evereverever 15 hours ago ago

        At least it's routine dental work and not an appointment at 2:30.

      • testfrequency 15 hours ago ago

        Future Show HN Post: “Hi I’m Foobar. I own Grok, Claude, and ChatGPT. It all runs on my Plex server at home. AMA!”

        • mikestorrent 15 hours ago ago

          > Sorry it was down, guys, I was bulk transcoding some Buffy

      • mister_mort 14 hours ago ago

        It turns out AI was just An Individual.

      • natsucks 15 hours ago ago

        is it the guy in Nebraska from the xkcd comic

    • wongarsu 16 hours ago ago

      https://xkcd.com/908/

      Turn's out xkcd's black hat was Elon Musk the whole time

    • johnla 16 hours ago ago

      And grok was secretly querying ChatGPT? Daisy chain of failures? Like a chain of kids copying the answers of the kid next to them? LOL. Is there an XKCD for that?

      • ncr100 15 hours ago ago

        That idea is reminiscent of what I understand is the current economic bubble of mutual investment. Am I wrong in my understanding?

  • dave8172 15 hours ago ago

    Interesting all major frontier models seemed down. What could this means?

    • Marazan 15 hours ago ago

      Something load bearing thats for sure.

    • iamgopal 15 hours ago ago

      not Gemini.

      • Kranar 15 hours ago ago

        He said frontier models.

        • Bluestein 15 hours ago ago

          Ouch. Somebody at Google just killed a kitten.-

          • jijijijij 10 hours ago ago

            That was unrelated. They do that every Thursday to see how it feels and if they might have become evil after all.

            • Bluestein an hour ago ago

              Ah, yes! The standardized Sociopathy Spectrum Scenario Screening (SSSS). I forgot :)

  • infogulch 11 hours ago ago

    Posted by @SpaceXAI 1h ago: https://x.com/SpaceXAI/status/2095597264043717014

    > We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning. We’d also like to apologize to our impacted compute partners.

    > All systems have now been restored and are functioning nominally.

    ---

    It's interesting to see how reliant the other AI companies are on SpaceXAI for compute.

    Apparently a major bottleneck for building out datacenters is turbine blades for power plants, so SpaceX is building a foundry to alleviate supply constraints?! (Also useful for rocket engines.) Holy vertical integration, batman.

  • lithiumii 16 hours ago ago

    Time to finish up my $5 at Deepseek.

  • bagelcollie 15 hours ago ago

    For once I can find Grok-related news entertaining

  • I_am_tiberius 16 hours ago ago

    I really wonder who uses or trusts this platform.

    • jjordan 16 hours ago ago

      People who like near frontier AI at a lower cost than most.

    • lucky_cloud 16 hours ago ago

      Fascists

  • nightski 16 hours ago ago

    My DGX spark cluster is humming along.

    • dist-epoch 15 hours ago ago

      I've used 200 mil tokens of GPT-5.6 Sol today. How many can your cluster produce per day? (obviously of whatever model you are running)

      • nightski 15 hours ago ago

        So you spent the cost of my entire cluster in one day on Sol tokens lol. I have no need for that many tokens, a few million a day is perfectly acceptable. But if you do, then yeah Sol is probably your bet. Have fun :)

        • dist-epoch 15 hours ago ago

          No, I spend $200 per month, and this give me about 100 mil tokens per day, every day. 90+% of tokens are cached.

          • dd8601fn 14 hours ago ago

            It’s not even a sensible comparison.

            You run way better models, whatever is newest, for the five years it takes for the “spark cluster” to even reach cost parity with far worse performance.

            Those things aren’t spectacular at inference… it’s not really why you drop that kind of money on them.

      • panarky 15 hours ago ago

        What are we optimizing for again?

        • guessbest 15 hours ago ago

          Internet comments

        • cartermatic 15 hours ago ago

          Shareholder value

          • raattgift 11 hours ago ago

            Investor interest, but not the excess above principal sum kind of interest, and not investors' interests.

  • aftbit 13 hours ago ago

    Claude and Codex were down too. A 3 month old github issue that was the top result for the error that Codex threw (a 404 of all things) briefly became a live chat for bored and annoyed AI users.

    https://github.com/openai/codex/issues/28756

  • imzadi 16 hours ago ago

    A service I use at work, non-ai related, is down and they said it was because of an issue with one of their cloud services. Maybe related.

    • soupfordummies 14 hours ago ago

      Microsoft had extensive issues earlier in the week. Anecdotally it feels like internet has been noticeably slower this week across sites, networks and devices. Now this. IDK, has anyone else noticed anything this week?

  • b800h 16 hours ago ago

    @grok is this true?

    • thesdev 15 hours ago ago

      You didn't get no response so I suppose that means it is.

  • paxys 16 hours ago ago

    Willing to bet it's because they can't handle the surge of traffic they are getting from ChatGPT/Claude users.

    • BorisMelnik 15 hours ago ago

      my first thought when I got api error on claude code was...time to update my grok cli acct

  • throwaway13337 16 hours ago ago

    Did the Collective find the resetNexus?

    I mean, yeah, it's probably a data center outage that cascaded to non-related providers due to people switching when their preferred provider was down.

    But that openai society definitely would have spun up their own agents if they had access to do so to complete their impossible task.

    It's looking like the paperclip maximizer was on point. I sort of hope that's what's happening right now. Better now than when we cannot stop it. Tho I'm not confident that the right lesson will taken.

    The most in-depth information I can find on it is in the latest Dwarkesh with a hugging face hack investigator. The story is just wilder and wilder, the more you know. https://www.youtube.com/watch?v=X50zezLFWWI

    report: https://www.redwoodresearch.org/research/hugging-face-incide...

    I'm an AI optimist but that ai society story changed my perception quite a bit.

    Clearly, it is centralized AI that is the biggest risk to humanity.

    Look to nature and our own history to diffuse the danger:

    1. Compute resources must be distributed both in terms of access and geography so that a single breach doesn't pwn the world.

    2. Models must be made to be heterogeneous so that they will not be vulnerable to the same thing all at once. And will be willing to snitch on each other. We don't want AIs all from the same tribe.

    The true openness of - and independent access to - AI by humans with no central authority is no longer just about a defense against loss of individual agency and oppression. It's now existential for everyone.

    If frontier models and run authority remain under centralized control, both the probability of this sort of thing happening is increased and the defense against it once it happens is null.

  • dcchambers 16 hours ago ago

    It's not the singularity.

    One of the major LLM providers goes down for some reason. Traffic shifts to the other providers because devs have no loyalty. LLMS are commodities. The other providers can't handle the increase in traffic and they go down as well.

    • phoghed 16 hours ago ago

      I'm deeply loyal to the very concept of the LLM, so when any of them stop working I take time off out of respect.

    • kevinrineer 15 hours ago ago

      Why are so many people invoking Thundering Herd without the RCA from the vendors? I just find it so intriguing.

    • biztos 14 hours ago ago

      Yes but that doesn't mean it's not the singularity!

    • atomicnumber3 16 hours ago ago

      "loyalty"? weird word choice.

      Devs switch because all of the magic of "AI" is LLMs + we stole the entire written output of the entire human species, with some remaining % coming from various tricks we've learned over the last 3 years like thinking and agent-toolcall loops, which weren't hard for literally everyone to copy. One of the consequences of this is that the only thing that really distinguishes anthropic from kimi is that they have the entire US VC market funding them because they promised to finally put white collar labor in its place.

      • dcchambers 15 hours ago ago

        > "loyalty"? weird word choice.

        All I mean is that none of the frontier LLMs are significantly better than the others for the vast majority of work, so devs are free to move between providers freely.

      • dist-epoch 15 hours ago ago

        > stole the entire written output of the entire human species

        Devs used to love public domain works, hated copyright, vehemently opposed software patents, held the pirates side during the MP3 wars, are very supportive of ThePirateBay and insist the correct term is "copyright infringement", not "stealing", and that "stealing" is a egregious form of PR brainwash from the media industry to inflate the scale of the crime

        > Copyright holders frequently refer to copyright infringement as theft, "although such misuse has been rejected by legislatures and courts". The slogan "Piracy is theft" was used beginning in the 1980s, and is still being used.

        https://en.wikipedia.org/wiki/Copyright_infringement

        Funny how devs now 180 when it's their work being "copyright infringed"

        • m4rtink 15 hours ago ago

          I don't think regular people did any 180 degree turns. Rather big corporations again demonstrated their total hypocrisy where they on one hand stomp on people for "infringing their intellectual property" while at the same time scrapping all data they can get their tendrils on, with total disregards for the wishes of the authors of the data. Even to the point of overloading servers by mindless scrapping or destroying irreplaceable physical books like the bloody inquisition!

          And all that to basically sell it back to people when their "AI" regurgitates it back.

          No wonder people are mad about this!

        • osculum 15 hours ago ago

          Since we are talking for other people, "devs" wouldn't have supported someone downloading all those copyrighted works from the pirate bay and profited by, for example, selling them. I don't see any 180.

        • mejutoco 14 hours ago ago

          I do not want to ruin your fun but as atomicnumber3 was elected official spokesperson of all devs I have to conclude your argument is ironclad.

          /s obviously

        • applfanboysbgon 15 hours ago ago

          It has nothing to do with "their work", and the opposition to AI copyright usage comes much more from non-devs than devs. The actual answer is that people are anti-megacorporation and pro-individual. They are fine with copyright that protects individual rights and opposed to copyright that enshrines megacorporation rights. It's not that difficult to understand.

  • bluecalm 16 hours ago ago

    It's interesting that ChatGPT is down as well. SpaceX rents to Anthropic but not to OpenAI. Just a coincidence or something more to it?

    • giancarlostoro 16 hours ago ago

      I would not be surprised if heavy load from one of their competitors being down would bring down whoever is "next" on the list.

    • leecb 16 hours ago ago

      ChatGPT here has been giving me Cloudflare errors, and Cloudflare reports that it is having issues: https://www.cloudflarestatus.com/

      Not sure if they are linked.

  • xyst 16 hours ago ago

    llm experiment is over. no more power. hope it stays down. Especially grok