Is AWS S3 having an outage?

91 points | by GGO a day ago ago

44 comments

  • mopatches a day ago ago
  • gslin a day ago ago

    Something related/non-related, it's still painful to read specific timezone not UTC.

    • nnf a day ago ago

      I've long wished for built-in browser functionality that converts times to the user's preferred time zone, with perhaps a dotted outline indicating that a change was made by the browser to the page.

      • crgwbr a day ago ago

        I’d always hoped this is what HTML’s time tag would become. Unfortunately it does almost nothing.

        • yellowsir 12 hours ago ago

          same with gps - the user should deside how to open direction / location links.

      • daniel_sim a day ago ago

        This is worth proposing. A tag where fallback text is provided within which can be overridden by the browser with a formatted date string would be excellent.

        • colanderman a day ago ago

          It exists: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ti...

          As a nearby comment indicates, it's not clear any browser makes use of it.

          • spartanatreyu a day ago ago

            I used the <time> element in a subscription dashboard.

            All times on the page are in UTC and are wrapped in a <time> element. The raw UTC time is on the datetime attribute and a fallback more easily readable time is inside the tags. Some JS on the page removes whatever text is inside the tag and replaces it with the user's locale specific format (no libraries required, it's a part of the browser standards).

            User friendly times wasn't actually the reason we implemented it.

            Our largest concern was our E2E testing. We needed to make sure that the dates/times displayed were always right (and since subscriptions involve money, we wanted to make sure that part of the website was the most tested.) The E2E testing simply ignores whatever is inside the tag and reads the datetime attribute instead. Then we can fastforward or rewind the simulated time inside the test to make sure everything is working as expected.

      • krick a day ago ago

        Yeah, but, I mean, the least you could do is to say UTC-7. I'm sure everybody living in PDT knows it's, well, their time, but how the fuck should everyone else know what time PDT is.

        • quesera a day ago ago

          Also, some timezone labels are ambiguous, e.g. "CST".

          • pxx a day ago ago

            What I learned recently is the way Python strptime deals with this. It only parses the current time zone or "GMT" or "UTC" and all have the same effect (of returning a tz-naive object). This behavior is amazing because I don't think it's ever what you want.

            $ TZ=Asia/Shanghai python -c "import datetime; print(datetime.datetime.strptime('4CST', '%H%Z').astimezone(datetime.timezone.utc))"

            1899-12-31 19:54:17+00:00

            $ TZ=America/Chicago python -c "import datetime; print(datetime.datetime.strptime('4CST', '%H%Z').astimezone(datetime.timezone.utc))"

            1900-01-01 10:00:00+00:00

            $ TZ=America/Havana python -c "import datetime; print(datetime.datetime.strptime('4CST', '%H%Z').astimezone(datetime.timezone.utc))"

            1900-01-01 09:29:36+00:00

            $ TZ=America/Havana python -c "import datetime; print(datetime.datetime.strptime('4CDT', '%H%Z').astimezone(datetime.timezone.utc))"

            1900-01-01 09:29:36+00:00

            $ TZ=America/Los_Angeles python -c "import datetime; print(datetime.datetime.strptime('4CST', '%H%Z').astimezone(datetime.timezone.utc))"

            ValueError: time data '4CST' does not match format '%H%Z'

            That last error was real fun to debug when something worked in production but not locally.

          • OhMeadhbh a day ago ago

            How is CST ambiguous? Are you thinking of all those counties in Indiana that don't observe daylight savings? In theory at least, a specific instant in CST maps to a single specific instant in UTC, GMT, CUT or Zulu Time, whichever you set your watch to.

            And it does seem like people sometimes forget that CST is not the same thing as CDT. Several months ago (after the 2024 PST -> PDT crossover) I had someone across town tell me they wanted to set up a meeting at something like 9AM. But when the invite came, the time zone was set for 9AM PST instead of 9AM PDT. I assumed it was PDT since we were in the same locality. But I did make sure my schedule was clear for the hour before on the off chance they had some weird software bug that picked time zones at random.

            If your point is something similar, then I heartily agree. In theory, converting between time-zones and daylight and standard times is easy, but in practice there are several situations where it's made more difficult than it should be.

            • quesera 15 hours ago ago

              I was referring to the multiple timezones that are abbreviated to "CST", as described in a sibling comment.

              But you bring up another good example. I've worked with people who write times as "PST" or "EST" all year long. This bothers me in its simple incorrectness -- I do know what they mean but I hate autocorrecting them.

              But I've also worked with people in parts of Indiana, and Mexico. When they say "CST" at any time of the year, they might mean it precisely!

              My strategy is to write, e.g. "8:30 AM US/Pacific". Which probably annoys other people, but at least it's precise and unambiguous. I think it's less awkward than the equivalent "America/Los_Angeles".

              Full list of US lower 48 timezones: US/Arizona, US/Central, US/East-Indiana, US/Eastern, US/Indiana-Starke, US/Michigan, US/Mountain, US/Pacific.

              And the +2s: US/Alaska, US/Aleutian, US/Hawaii

            • chucksmash a day ago ago

              > How is CST ambiguous?

              - CST: Central Standard Time UTC−06

              - CST: China Standard Time UTC+08

              - CST: Cuba Standard Time UTC−05

    • quesera a day ago ago

      Yes, this can be super painful, especially when correlating information from several sources, or when traveling. It can become almost impossible when reviewing historical data.

      My solution to inconsistent time rendering in AWS (sometimes UTC, sometimes localtime) is to run a separate browser profile for AWS (and third party status pages) that that's always running in UTC.

      But seriously, status page people: there's no excuse for being vague about timezones!

  • szvsw a day ago ago

    us-east-1 gang rise up!!

    For once I can feel mild pleasure at seeing the tables turned…

    To be honest I don’t know why all my projects are always in USE1, I guess it’s just because that’s where we have always had them for my lab so I’ve stuck with it for no good reason…

    • jedberg a day ago ago

      US East 1 was the default region until a few years ago. If your account is older than that, then most likely all your stuff is there.

      • szvsw a day ago ago

        Yeah the account is from like 2019 or so. But still, I’m spinning everything up with Terraform or AWS Copilot, so it is still an active choice on my end to put it in us-east-1.

      • averageRoyalty a day ago ago

        Unless - like most people - you and your audience are not in the US.

        • jedberg a day ago ago

          This wasn't an "everyone is in the US" type of post. Even for non-US folks, for a long time the default region was us-east-1, until they switched it to eu-west-1, and about 1/2 of all of AWS customers are in the US, or at least used to be until recently.

          • EvanKnowles a day ago ago

            "Where's my new server gone? ... oh, it's hanging out in us-east-1, damnsit."

    • cr125rider a day ago ago

      They launch all the new fun toys there!

    • iJohnDoe a day ago ago

      It’s usually always cheaper there.

  • gp5dgk 6 hours ago ago

    I am seeing on multiple AWS accounts CloudWatch log groups are not loading. It even says you have no log groups, that's not the case.

  • synhare a day ago ago

    Elastic Beanstalk, Lambda, and CloudWatch returning errors for us. us-east-2

  • aertmann a day ago ago

    Can confirm, S3 backed CloudFront experiencing random errors in us-east-2 for the past 15 minutes at least.

  • chromatin a day ago ago

    Cloudfront is: my small, not even yet profitable SaaS landing page and javascript frontend, served by CloudFront, are down since about 17 minutes ago (3:28 US Eastern)

    the API whcih sits behind ELB is working fine

    edit: us-east-2

  • mastry a day ago ago

    I wonder if this is related to the NHC outage?

    https://news.ycombinator.com/item?id=41771629

  • austinpena a day ago ago
  • GGO a day ago ago

    S3 on us-east-2 stabilized for us as of 2 minutes ago

  • mopatches a day ago ago

    We're seeing CloudFront and S3 issues in us-east-2 in multiple accounts. No errors in us-east-1 or and ap-south-1.

    • paulddraper a day ago ago

      No issues in us-east-1 (for once....)

      • mopatches a day ago ago

        Thank you! I was misreading our logs. Can confirm no errors in us-east-1 for us as well.

  • dn0 a day ago ago

    Next time you can also check https://cloudstatus.page/cloud/aws/issues

    I guess I'll increase the frequency of some probes to get better aligned with the official timeline :)

  • wbobeirne a day ago ago

    Also seeing Cloudfront failures on my end, both in us-east-1 and us-east-2.

  • MuffinFlavored a day ago ago

    https://health.aws.amazon.com/health/status

    Rhetorical question (I know why, humans manually involved, incentive to not report SLA breakage, etc.): How does their status page not auto-update when one of their core APIs goes to basically 99% 500 (or even above 5% 500) status error?

    • gottorf a day ago ago

      "Show me the incentive, and I'll show you the outcome."

  • sdv0389 a day ago ago

    Seeing CloudFront + S3 errors in us-east-2 on our end as well.

    • sdv0389 a day ago ago

      And now we're back up.

  • jamroom a day ago ago

    us-east-2 seeing tons of S3 errors.

  • MuffinFlavored a day ago ago

    Yes. I can't tell if listing/reading is fine but putting (uploading) for sure seems to not be working (500 error).

    • jerjerjer a day ago ago

      Have issues reading (500).

  • sgt a day ago ago

    This is why I use Hetzner's Object Storage. Proven and tested! /s

  • 7874cole a day ago ago

    Next time my boss complains about uptime, I will tell him, even AWS have downtime. Boom!