It's tempting to interpret this as a proxy for over-all damage but it's perhaps a bit unwise to do so. Alternative possibilities are that internet related infra (power lines, cell towers, roof-mounted dishes, etc.) was disproportionally damaged or that cities in the projected path saw a lot of evacuations. Of course all of these things could be true at the same time. It's an interesting way to try to assess damage in any case because it's automagically collected; you don't need to wait for people on the ground to report damage. So here's hoping that a) those affected can get back on their feet quickly and b) there's some follow-up to see how useful this data was as a proxy for assitance needs.
I spend most of my time in the path of hurricanes, we lose power 5-10x per year for between 1 to 8 hrs on average.
When we lose power the battery backup for the modem takes over(15hrs runtime), and we have other portable batteries to recharge our phones multiple times.
If things get really bad, we can pull out the camping solar panels and keep the phones charged. At that point, I expect the fiber will be down anyways and cell service is probably all that would be available.
Works quite well. I can usually still complete my work day on my laptop without issue.
and many are busy prepping, checking in with loved ones, checking weather updates, and then cleaning up, which means less time doing normal Internet things
One of my go to power outage devices is a cheap car jump starter with USB charging. 50$ and you can run a cellphone with wireless hotspot and a tablet for days and it’s far less of a hassle than jumper cables.
Actually having solar or a generator is definitely a good idea for homes, but apartments also lose power.
There are pretty much constantly deals for battery backups from companies like Jackery and Bluetti that can run modems and routers for many hours for under $300.
There's probably better figures. Internet providers themselves probably know what subscribers dropped at what times, so you'd be able to distinguish from evacuations and have a good signal as to whether it happened from power outages or key pieces of their infrastructure failing.
It would be really neat to figure out the cooperation necessary to effectively use it. Rapid damage assessment has a ton of value even if the data is somewhat unreliable.
Evacuations are, unfortunately, often accompanied by tweets. As early as 2012 or 2014 a team I was on was able to localize the epicenter of an earthquake by examining the Twitter fire hose, half an hour before the earthquake was announced on any of the major news networks.
Almost on topic: I have this vague memory of some very interesting blog posts of a datacenter sysadmin who bravely dealt with hurricane Katrina and the aftermath, in 2005.
I really wanted to reply here with a link to something I recall reading from back when Katrina happened, but I can't find it.
If I remember correctly it was someone called Usurper or The Usurper and they chronicled their journey at the time looking after a DC during Katrina. It may not have been a whole DC, possibly it was a business and their (smaller set of) servers, but my memory fails me.
Unfortunately I can't remember what medium it was written on - a blog of some sort? Heck, it could even have been on the Something Awful forums given the year.
I remember stories of struggling to find fresh sources of gas for the generator and all the fun involved in getting it from A to B.
Anyone else remember that? I'd love to have another read of it now, and I think some of you might too. It'll offer a bit of insight of what's to come this year too.
There's also the infamous bucket brigade from Hurricane Sandy. There was a Stack Overflow podcast about it but it looks like it disappears, here's a read though:
Honestly not really. Some of the older facilities took great care on site selection and some hardened building features, but modern sites are more based on power grid availability and are simply standard amazon distribution warehouse style buildings.
The redundancy is in having your data and infrastructure in multiple buildings and geographic locations.
This is due to a myriad of reasons but they all boil down to cost and practicality. The scale of modern facilities typically eclipses anything built 20 years ago by a huge margin which further limits site selection.
The days of AT&T long lines are long gone. Simply driving a standard SUV through the right wall or two in most facilities would be enough to cripple them for a long while, much less flooding or a direct hit by hurricane force winds.
You could be in the basement of a high-rise though, so that's not quite as helpful. My dad was in commercial construction, and I got to visit several sites. One in particular had a basement with concrete islands dotting the floor. They were support for the computer racks to be raised so if the sprinklers were to open the racks would be above the flooding water. They however would not protect from 12'-15' storm surges
They raise the floor not just to keep the racks above any water but also the air handlers typically blow chilled air under the floor to be delivered via perforated floor tiles to the "cold aisles" this saves having to run ductwork for cooling. Also they can run power, networking, and chilled water for any systems needing it, under the floor.
Getting all the equipment 24 inches off the floor saves you from a pretty big fraction of events. You can often mostly protect against fairly severe events with sump pumps and flood barriers, but some water almost always gets in. If nothing important is on the ground you'll often still be OK.
For a datacenter that does not exist on a coastline susceptible to hurricanes, you might have a point. However, we're specifically discussing a cluster of datacenters near Tampa which just missed a direct hit from a very powerful hurricane that had a forecast of a storm surge of 12'-15'. The thing to remember about storm surge is the predicted height does not include the height of wind driven waves on top of that surge. So unless your data center racks are on the 3rd floor, you are screwed
Yes, I am not saying that you can withstand any event.
I have seen >3' of water next to a building with infrastructure in the ground level that were fine, though. Seepage barriers and sump pumps and elevating things a little bit can do more than you'd think.
It's also worth noting that however high the storm surge is -- your infrastructure is probably at least a little above sea level.
It's tempting to interpret this as a proxy for over-all damage but it's perhaps a bit unwise to do so. Alternative possibilities are that internet related infra (power lines, cell towers, roof-mounted dishes, etc.) was disproportionally damaged or that cities in the projected path saw a lot of evacuations. Of course all of these things could be true at the same time. It's an interesting way to try to assess damage in any case because it's automagically collected; you don't need to wait for people on the ground to report damage. So here's hoping that a) those affected can get back on their feet quickly and b) there's some follow-up to see how useful this data was as a proxy for assitance needs.
Many science fiction works begin with the premise "we've lost contact with the colony". It's a clear signal that something is amiss.
LV426
we try not to use our phones - if the power goes out, you can't charge again for how long? You never know.
another problem are power flickers. a short flicker resets modems and routers. so you read a book or talk rather than waste battery
I spend most of my time in the path of hurricanes, we lose power 5-10x per year for between 1 to 8 hrs on average.
When we lose power the battery backup for the modem takes over(15hrs runtime), and we have other portable batteries to recharge our phones multiple times.
If things get really bad, we can pull out the camping solar panels and keep the phones charged. At that point, I expect the fiber will be down anyways and cell service is probably all that would be available.
Works quite well. I can usually still complete my work day on my laptop without issue.
and many are busy prepping, checking in with loved ones, checking weather updates, and then cleaning up, which means less time doing normal Internet things
One of my go to power outage devices is a cheap car jump starter with USB charging. 50$ and you can run a cellphone with wireless hotspot and a tablet for days and it’s far less of a hassle than jumper cables.
Actually having solar or a generator is definitely a good idea for homes, but apartments also lose power.
There are pretty much constantly deals for battery backups from companies like Jackery and Bluetti that can run modems and routers for many hours for under $300.
Ok why is this being downvoted? Isn’t this helpful info?
There's probably better figures. Internet providers themselves probably know what subscribers dropped at what times, so you'd be able to distinguish from evacuations and have a good signal as to whether it happened from power outages or key pieces of their infrastructure failing.
It would be really neat to figure out the cooperation necessary to effectively use it. Rapid damage assessment has a ton of value even if the data is somewhat unreliable.
Evacuations are, unfortunately, often accompanied by tweets. As early as 2012 or 2014 a team I was on was able to localize the epicenter of an earthquake by examining the Twitter fire hose, half an hour before the earthquake was announced on any of the major news networks.
Almost on topic: I have this vague memory of some very interesting blog posts of a datacenter sysadmin who bravely dealt with hurricane Katrina and the aftermath, in 2005.
Anyone know what I'm talking about?
https://en.wikipedia.org/wiki/Interdictor_(blog)
Yes! Thanks. (Unfortunately the wayback machine is down...)
Nevermind: The actual livejournal posts are still up.
Start here: https://interdictor.livejournal.com/2005/08/27/
Then just move forward day by day. (next day link in the lower right corner)
There are several DCs in the Tampa area. How are they? https://www.datacentermap.com/usa/florida/tampa/
I used to be a customer of Hivelocity with dedicated servers in their Tampa facilities. Top notch operation, never had an issue. Supposedly they remained unscathed through Milton: https://www.hivelocity.net/blog/hurricane-milton-service-not...
I really wanted to reply here with a link to something I recall reading from back when Katrina happened, but I can't find it.
If I remember correctly it was someone called Usurper or The Usurper and they chronicled their journey at the time looking after a DC during Katrina. It may not have been a whole DC, possibly it was a business and their (smaller set of) servers, but my memory fails me.
Unfortunately I can't remember what medium it was written on - a blog of some sort? Heck, it could even have been on the Something Awful forums given the year.
I remember stories of struggling to find fresh sources of gas for the generator and all the fun involved in getting it from A to B.
Anyone else remember that? I'd love to have another read of it now, and I think some of you might too. It'll offer a bit of insight of what's to come this year too.
Yes, that was the interdictor blog!
https://interdictor.livejournal.com/2005/08/27/
Interdictor! That's the one!
Looks like the posts start at https://interdictor.livejournal.com/?skip=340 and newer ones can be found via the 'Next 10' link at the lower right.
Thanks Julian!
There's also the infamous bucket brigade from Hurricane Sandy. There was a Stack Overflow podcast about it but it looks like it disappears, here's a read though:
https://www.datacenterdynamics.com/en/analysis/reliving-sand...
Aren't datacenters usually pretty hardened buildings? I would think they're okay but disconnected.
Honestly not really. Some of the older facilities took great care on site selection and some hardened building features, but modern sites are more based on power grid availability and are simply standard amazon distribution warehouse style buildings.
The redundancy is in having your data and infrastructure in multiple buildings and geographic locations.
This is due to a myriad of reasons but they all boil down to cost and practicality. The scale of modern facilities typically eclipses anything built 20 years ago by a huge margin which further limits site selection.
The days of AT&T long lines are long gone. Simply driving a standard SUV through the right wall or two in most facilities would be enough to cripple them for a long while, much less flooding or a direct hit by hurricane force winds.
Of course there are exceptions.
> redundancy is in having your data and infrastructure in multiple buildings
aka "RAIDC" -- redundant array of inexpensive data centers. :)
need to get the electricity from somewhere. Need to feed the people who work there.
Wow. I didn't expect to see so many so close to the water. They might be in high-rises though.
You could be in the basement of a high-rise though, so that's not quite as helpful. My dad was in commercial construction, and I got to visit several sites. One in particular had a basement with concrete islands dotting the floor. They were support for the computer racks to be raised so if the sprinklers were to open the racks would be above the flooding water. They however would not protect from 12'-15' storm surges
They raise the floor not just to keep the racks above any water but also the air handlers typically blow chilled air under the floor to be delivered via perforated floor tiles to the "cold aisles" this saves having to run ductwork for cooling. Also they can run power, networking, and chilled water for any systems needing it, under the floor.
Getting all the equipment 24 inches off the floor saves you from a pretty big fraction of events. You can often mostly protect against fairly severe events with sump pumps and flood barriers, but some water almost always gets in. If nothing important is on the ground you'll often still be OK.
For a datacenter that does not exist on a coastline susceptible to hurricanes, you might have a point. However, we're specifically discussing a cluster of datacenters near Tampa which just missed a direct hit from a very powerful hurricane that had a forecast of a storm surge of 12'-15'. The thing to remember about storm surge is the predicted height does not include the height of wind driven waves on top of that surge. So unless your data center racks are on the 3rd floor, you are screwed
Yes, I am not saying that you can withstand any event.
I have seen >3' of water next to a building with infrastructure in the ground level that were fine, though. Seepage barriers and sump pumps and elevating things a little bit can do more than you'd think.
It's also worth noting that however high the storm surge is -- your infrastructure is probably at least a little above sea level.
Anything of value should be above the first floor.
https://archive.nytimes.com/www.nytimes.com/2012/11/01/healt...
https://nitter.poast.org/CloudflareRadar/status/184444428691... (not X-walled link to thread)
Starlink has been providing emergency internet service in the affected areas: https://news.ycombinator.com/item?id=41803857
That article is about areas affected by Helene, not Milton
Based on what people are posting, it appears Starlink's emergency services are continuing with Milton: https://x.com/search?q=starlink%20milton&src=typed_query
There's nothing on that link.
It’s to a WSJ article, but apparently users flagged it.
Any non-Xitter link? Also, how much of this is less TikTok and similar?
Wouldn't you expect this even if there was no actual damage since so may people evacuated, as they were told?