Failure analysis of the Arecibo 305 meter telescope collapse

(nap.nationalacademies.org)

176 points | by mhb 10 hours ago ago

66 comments

  • magicalhippo 9 hours ago ago

    In the summary they state:

    All the reported experimental zinc electroplasticity (EP) data were developed at current densities orders of magnitude higher than those possibly present in the Arecibo Telescope but measured in laboratory experimental periods that were orders of magnitude shorter than the telescope's socket zinc service.

    There are no reported experimental data concerning low-current, long-term EP, which the committee has lumped together under the term "LEP", affecting zinc's creep mechanisms over decades.

    The timing and patterns of the Arecibo Telescope's socket failures make the LEP hypothesis the only one that the committee could find that could potentially explain the failure patterns observed.

    Accelerated aging is, as far as I know, pretty much standard in the industry. Nobody can wait 20 years to find out if a certain material is good enough or not.

    However, the real failure seems to be the lack of urgency when they signs started to show up:

    Upon reflection, the unusually large and progressive cable pullouts of key structural cables that could be seen during visual inspection several months and years before the M4N failure should have raised the highest alarm level, requiring urgent action. The lack of documented concern from the contracted engineers about the inconsequentiality of the cable pullouts or the safety factors between Hurricane Maria in 2017 and the failure is alarming.

    • deskr 9 hours ago ago

      > The lack of documented concern from the contracted engineers about the inconsequentiality of the cable pullouts or the safety factors between Hurricane Maria in 2017 and the failure is alarming.

      This is crazy. Basically cables were pulling out for months and years and no one raised the alarm? In many industries that could be a criminal or career ending malpractice. Are the "contracted engineers" liable?

      • kreims 3 hours ago ago

        You go to school and learn that 2+2=4. You get a consulting job and learn 2+2= whatever the client says it is.

        I wouldn’t jump to the conclusion that the consulting engineer was incompetent. Sometimes a bureaucrat tells you to sharpen your pencils and come back with the answer That fits the budget they have if you want to keep your professional services contract going.

      • londons_explore an hour ago ago

        Since the design had a factor of safety of 2, and no other cables exhibited pull out at such high safety factors, checking such things might not even have been on the checklist.

      • magicalhippo 7 hours ago ago

        In chapter 5 they go into how the ownership was transferred to the University of Central Florida (UCF) at the start of 2018, after the hurricanes in 2017.

        It seems unlikely that UCF had adequate time and resources to review and understand the Arecibo Telescope's original 1963 design, the 1974 upgrade, the structural inspection and maintenance records produced for nearly 50 years, [...] and the key factors, such as the wire breaks and cable pullout of the sockets and their significance on the strength and integrity of the structure.

        The measured cable pullout may have appeared "normal" to [the UCF staff] and was not on their radar as signs of structural distress. The lack of concern may be because a small cable pullout was present from the beginning, and no one in authority had previously raised an alarm

        But yeah, seems weird no-one of the contractors tried to raise an alarm.

        • hinkley 5 hours ago ago

          I bet there’s a r/MaliciousCompliance in Spanish out there somewhere about how UCF cancelled existing maintenance contracts and hired new companies to keep an eye on the equipment, and did it so rudely that the old company didn’t hand over important warnings like this.

          Discontinuity could explain problems like this readily.

      • bsder an hour ago ago

        > Basically cables were pulling out for months and years and no one raised the alarm?

        And what if they did? Then what? My guess is the conversation went something like this:

        "Okay, cables are pulling out. Raising this issue will not magically make money appear to fix it as nobody wants to fund this. Tell me, how much do you like your job? If you flag this, you may lose your job directly and if the project gets shut down you may lose your job indirectly. So, how about we bury this as much as possible, cross our fingers and bank as much money as possible in the meantime, eh?"

      • LorenPechtel 3 hours ago ago

        But that would have required spending money. And why would the engineers be liable? They weren't being paid to fix it.

      • renewiltord 7 hours ago ago

        These are Professional Engineers, not your run of the mill fake 'engineer' in software development. They have licenses and certifications. They don't just ship bugs. When they say that something is fine, it is fine. They have certifications. They have licenses. They are professionals. If something failed, it must have been management. Professional Engineers do not make mistakes. They are real engineers.

        • feoren 6 hours ago ago

          As someone who works in a company with 10,000+ professional engineers*: yes, yes they do make mistakes. Despite what you seem to think, they are actually human beings. Good processes understand this and have multiple layers to catch and correct mistakes. But it's hard to fight limited data. Plenty of engineering decisions are still made based on numbers in photocopied tables from two or three studies from 50 years ago, each having a dozen or two data points. It's surprising how limited some of the data is that these Professional Engineers are basing their decisions on.

          *Edit: okay, I don't know exactly how many are officially PEs vs. junior engineers or something, but at least that many are "PE-track".

        • LaffertyDev 6 hours ago ago

          > Professional Engineers do not make mistakes. They are real engineers.

          My _literal_ engineering 101 case studies on "engineering failure":

          https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge_(1940)

          https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse

          https://en.wikipedia.org/wiki/I-35W_Mississippi_River_bridge

          Are all failures traced down to the underlying design and engineering. Granted, they are famous because they are rare. I almost linked the Columbia disaster but that is sufficiently murky between engineer and management.

          Stop being elitist.

        • mden 6 hours ago ago

          I think people are missing that you're being satirical (maybe there is a better word for it?). Unless I'm the one being wooshed.

          • rootusrootus 6 hours ago ago

            I agree. Maybe on the internet we really do require /s just to make sure everyone understands, but it looks to me very much sarcastic.

            • PhilipRoman 2 hours ago ago

              It's 100% sarcasm. The capitalization should have made it obvious to anyone. The only place I've seen struggling with recognizing sarcasm is hackernews.

              • rootusrootus an hour ago ago

                Software developers are very sensitive to being attacked as 'not real engineers' and it turns off the humor detector.

        • mNovak 6 hours ago ago

          I wouldn't overinvest your confidence in these licenses and certifications. A "Professional Engineer" is just a Bachelors level engineer who took one extra test. It's extremely common for MechE and CivE. As far as certifications go it's not quite the same as, say, passing the bar exam or finishing med school.

          • albrewer 3 hours ago ago

            > A "Professional Engineer" is just a Bachelors level engineer who took one extra test

            At least for mechanical, it's 4 years' working experience, documentation of sufficiently advanced projects, and sign-off from 3 other registered professional engineers who are familiar with your work.

            A guy I worked with took the mechanical design variant. I don't know how the HVAC variant is. The test is over the most advanced stuff you learn in your degree; the "specialist" portion of the test (what your concentration is in, e.g., HVAC, thermal/fluids, or machine design and materials) is ~25 questions and you have 4 hours to complete it, in addition to the other 4 hour comprehensive portion that's the same for all takers.

            On top of that, to keep your license, you need to complete some amount of classroom education every year.

            As far as coursework goes, the coursework for some of my upper-level mech E classes was usually identical to a graduate-level course, the only difference being graduate students do a research paper+experiment for a final exam and undergrads do a traditional written exam since they had access to funding and we did not.

            imo this comment is dismissive and glosses way too heavily over the rigor that goes into even the "lowliest" of engineering programs.

            • wpollock 33 minutes ago ago

              This is how PEs should be certified. Sadly, licensure of PEs varies by state.

              In Florida, the title of "software engineer" requires a license. To obtain the license, you only need to pass the national engineering (I forgot the exact name of the organization) exam for software engineering. The national exam was withdrawn some years ago. I wrote the Florida Engineering Board about it, suggesting that they needed to update the requirements.

              Their response? Florida doesn't distinguish between PE specialties; prospective software engineers should just get a PE license in another field. They suggested electrical engineering.

              So in Florida, the PE signing off on your buildings's wiring might be a chemist, or the PE supervising a bridge construction might be only trained in electrical engineering.

          • dylan604 5 hours ago ago

            But it is different than that cert you receive after taking a 2 week bootcamp on Python.

        • haccount 7 hours ago ago

          Anyone in the field will likely say that they wouldn't trust half of their colleagues or themselves.

          It's pretty much how professional prestigious positions work. Beneath the gilding there's duct tape repaired with super glue.

          • renewiltord 7 hours ago ago

            That's how software works. But professional engineers are professionals. They have certifications. They have an iron ring. It's real.

            • feoren 6 hours ago ago

              I'd like to live in the world you imagine we live in.

        • tiahura 5 hours ago ago

          I'm not sure by your tone if this is sarcasm, but it has been shown by formal proof that if something failed the root cause must be an MBA.

    • mkesper 9 hours ago ago

      And further:

      The reliance by the consultants (before and after the first cable failure in 2020) on a perceived allowable pullout of one-sixth of the cable diameter, which should only be seen at loading at 80 percent of ultimate cable strength, does not align with the AASHTO M 277 standard guidance. The committee, therefore, disagrees with the suggestion made in the Thornton Tomasetti, Inc. (TT) 2022 report, Arecibo Telescope Collapse: Forensic Investigation,13 to use the D/6 limit as a threshold for slip monitoring.

    • UniverseHacker 2 hours ago ago

      As a sailor that uses similar fittings in sailboat rigging… the idea of not treating a cable pullout from a terminal as an emergency is absurd. If that happened on a sailboat rig I’d assume it had a remaining strength of zero and will fall in seconds or less- and take the most rapid possible action to keep the structure from coming down by deloading the shroud and rigging an emergency replacement.

    • Iolaum 5 hours ago ago

      I would be curious if there are any accounts/reports of people working there - especially engineers - that could verify the claims of no alarm being raised when the first structural failures started showing up. It is often the case that people on the ground report such things but when "reports" are written for the higher ups, facts get "massaged" according to expectations.

      • 77pt77 5 hours ago ago

        Bearers of bad news get shot.

        Everyone working in large corporations knows this.

        • lazystar 2 hours ago ago

          being a bearer of bad news is a sign that you are not disagreeing and committing.

    • nick3443 9 hours ago ago

      Sounds like it was a blind spot for the engineers due to the normally bulletproof service history of zinc spelter cable terminations in structures.

      • potato3732842 9 hours ago ago

        >normally bulletproof service history of zinc spelter cable terminations in structures.

        They're relatively new, they're relatively expensive (which saves them from being used in the worst of the applications) and like most things inherent to steel cable and whatnot they tend to get employed in situations where safety factors are generous. Don't count your chickens before they hatch.

        • nick3443 5 hours ago ago

          I am not an expert, I read this in the report: The long-term zinc creep failure of the Arecibo Telescope sockets and the subsequent cable pullout has never been documented elsewhere despite a century of zinc-filled cable spelter sockets use. 17 The type, size, length, and fittings of the cables used in the Arecibo Telescope (whether the original cables constructed in the 1960s or the auxiliary cables installed in the 1990s) were catalog-selected items, not at all unusual, with decades of proven performance. Exten-sive structural modeling of the Arecibo Telescope, independently confirmed by laser cable sag surveys, validated that under all static and cyclic loading conditions, the cable loads barely exceed half the nominal cable strength. While PLC can occur at stresses well below yield, such a creep failure has never been reported in spelter socket zinc.

          • londons_explore an hour ago ago

            > Exten-sive structural modeling of the Arecibo Telescope, independently confirmed by laser cable sag surveys, validated that under all static and cyclic loading conditions, the cable loads barely exceed half the nominal cable strength.

            I have a feeling such measurement and modelling might have been wrong.

            It's easy to miss some oscillation mode, particularly high frequency longitudinal wave's in the cables, which couldn't be picked up by a laser survey due to the kilometers per second these waves travel.

          • hinkley 4 hours ago ago

            I think we might need to send some microbiologists to do their own forensics.

            You have a mechanical failure that can’t be seen in the labs and isn’t reproducible at other sites. Suggests something environmental.

            Like when natural fiber army gear starts dissolving in the jungles of Vietnam.

            Could be direct action, or a side effect of a chemical in plant or animal detritus. Like pigeon poop on car paint.

            • madaxe_again 2 hours ago ago

              The paper posits that it’s due to low frequency electromagnetic induction in the tethers from the observatory operations, which is a pretty unique environmental factor.

              • hinkley 11 minutes ago ago

                Oooh.

                Active galvanic action will ruin your day.

    • thimkerbell 5 hours ago ago

      Is this flaw something an LLM could have caught? (Or do I have an inflated understanding of their abilities?)

      • magicalhippo an hour ago ago

        I think absolutely not, for several reasons. First off, the report mentions this failure of the zinc filled sockets never been reported elsewhere before.

        Secondly, even if it had, the report mentions that written documentation of inspections and so on was spotty and inconsistent.

        That said, if the failure mode had been published before, and inspections and maintenance was written down in a consistent and detailed manner, then perhaps an LLM-based tool could have linked the two. I'm thinking of a RAG-like search thing that compared inspection reports with published documents. Alternatively just assisting a human in finding relevant published reports.

  • Wololooo 8 hours ago ago

    Even if I am not a radio astronomer, this instrument was unique in nature and still in use for very specific observations and until the very end of its operation was able to deliver data. It is disheartening to learn that it was indeed due to gross maintenance negligence (as assumed originally by many) that the final fatal failure occurred, when potentially the structure could have been salvaged.

    • magicalhippo 7 hours ago ago

      Losing Arecibo was, as I understand it, a big blow to the NANOGrav experiment, which is looking for very low frequency gravitational waves by measuring pulsar timing variations[1].

      Albeit having limited field of view the Arecibo Telescope was very sensitive[2], and so could see pulsars that the other telescopes they used could not. And the longer they could collect data from a set of pulsars the lower frequency waves they could probe.

      [1]: https://nanograv.org/news/15yrRelease

      [2]: https://pirsa.org/20100068 Moving Closer to a Detection of nHz-frequency Gravitational Waves with NANOGrav (Arecibo details at around 10:20)

    • axus 7 hours ago ago

      My uninformed recollection from when the warning signs were raised, is there was not even budget to perform emergency maintenance, and they were resigned to the inevitable collapse.

      • jdhwosnhw 7 hours ago ago

        I worked and did research atobservatory for my PhD - you are exactly correct. It was well known to the scientists and onsite engineers that a collapse was imminent, for years. Ironically, prior to the collapse the agency responsible for funding performed a study to see would need to do to stop supplying money to the observatory, and found that due to environmental impact, it would actually be more expensive to stop funding. After that study we all suspected that the agency was just waiting for decay or a natural disaster to do the dirty work for them

        • trogdor an hour ago ago

          Would you link the study you are referring to?

      • WillPostForFood 5 hours ago ago

        That's not what the report says. There was ~$14 million allocated for repairs in 2-18 after Hurricane Maria, but the cable sockets weren't identified as needing repairs until the first one failed. The problem wasn't the money, it was the failure to identify the problem. See page 28/29.

    • SirMaster 8 hours ago ago

      What was unique about it compared to this one for example?

      https://en.wikipedia.org/wiki/Five-hundred-meter_Aperture_Sp...

      • superkuh 7 hours ago ago

        Arecibo did high power transmit for radar imaging of asteroids/comets/the sun/etc and ionosphere studies, among other things. FAST is not capable of transmit (it was not designed for it and the suspended receiver mass is much lower). Also, FAST has a RF-noise emitting theme park built just outside of it's grounds.

        The uniqueness of Arecibo was this transmit coupled with such a large aperture (so small pattern on sky).

        • dylan604 5 hours ago ago

          > Arecibo did high power transmit for radar imaging of

          the Sun? that's something I never thought about, and now that I'm thinking about it, still seems very interesting. It seems like there would be so much noise radiating from the burning ball of fusion that radar signals would just get absorbed/lost. And now I've sat here for 5 minutes doing nothing but imagining this.

        • SirMaster 7 hours ago ago

          Thanks! To a layman it seemed like they were really similar devices.

  • ctippett 2 hours ago ago

    Practical Engineering on YouTube has an episode covering the collapse of this structure[1]. It's very good and worth a watch.

    [1] https://youtu.be/3oBCtTv6yOw

  • sideshowb 4 hours ago ago

    Analysis of recorded audio contains the phrase "for England, James?"

    • fijiaarone 2 hours ago ago

      Finally a cogent post. Kids these days.

  • pantulis 5 hours ago ago

    What an imposing document: the details are overwhelmingly detailed, it really does justice to the marvel of engineering that the Telescope was.

  • bell-cot 6 hours ago ago
  • nonfamous 9 hours ago ago

    From the summary:

    >>> The only hypothesis the committee could develop that provides a plausible but unprovable answer to all these questions and the observed socket failure pattern is that the socket zinc creep was unexpectedly accelerated in the Arecibo Telescope's uniquely powerful electromagnetic radiation environment. The Arecibo Telescope cables were suspended across the beam of "the most powerful radio transmitter on Earth." The other investigations failed to note several failure patterns and provided no plausible explanation for most of them. To answer these questions with empirical evidence instead of only the inferences that can be made from the existing data, a more comprehensive and widespread forensic analysis of "good" and "bad" socket workmanship and the low-current, long-term effect on zinc creep is required.

    • shagie 9 hours ago ago

      (There was a comment that I was replying to about the realization that this was a transmitter and not just a receiver that was deleted ...)

      It was both.

      The Arecibo message was transmitted from there - https://en.wikipedia.org/wiki/Arecibo_message

      > The entire message consisted of 1,679 binary digits, approximately 210 bytes, transmitted at a frequency of 2,380 MHz and modulated by shifting the frequency by 10 Hz, with a power of 450 kW.

      There's also the field of radar astronomy (not radio astronomy) https://en.wikipedia.org/wiki/Radar_astronomy where you bounce of a signal off of an object in the solar system (such as an asteroid)

      > Radar provides the ability to study the shape, size and spin state of asteroids and comets from the ground. Radar imaging has produced images with up to 7.5-meter resolution. With sufficient data, the size, shape, spin and radar albedo of the target asteroids can be extracted.

      > ...

      > In August 2020 the Arecibo Observatory (Arecibo Planetary Radar) suffered a structural cable failure, leading to the collapse of the main telescope in December of that year.

      > There is one remaining radar astronomy facility in regular use, the Goldstone Solar System Radar.

    • userbinator 9 hours ago ago

      DC can definitely cause a lot of electrolytic damage, but AC usually doesn't. Perhaps something was acting as a diode (these are easily formed from natural processes) and rectifying the signal.

      • whoopdedo 6 hours ago ago

        The old "Foxhole radio" trick of using a rusty razor blade as a diode comes to mind. Some metal oxide coatings are semiconducting and under the right conditions will form a Schottky barrier.

    • Syonyk 9 hours ago ago

      Fortunately, more novel research comes out of "Huh, that's weird... why did that happen?" than out of things behaving as expected. Though this one does seem rather baffling.

    • ricksunny 9 hours ago ago

      I'm interested in hearimg the most speculative hypotheses as to why Arecibo Telescope's socket zinc creep proceeded so quickly compared to expectation.

  • alexnewman 7 hours ago ago

    Puerto Rican here. Everything on our island is like that. Preventive maintence is ignored and our insurance companies go bankrupt every big hurricane

    • wileydragonfly 6 hours ago ago

      Everything in the Caribbean is like that if you want to be honest.

    • BobaFloutist 5 hours ago ago

      If we ever make you a state proper hopefully that will offer an opportunity for that to change.

      • ordu 2 hours ago ago

        Aren't you afraid that US insurance companies will go bankrupt?

      • ljsprague 2 hours ago ago

        Statehood? Why not independence?

        • BobaFloutist an hour ago ago

          Because they voted for statehood in a referendum.

        • leptons an hour ago ago

          How would independence be better?

      • dylan604 5 hours ago ago

        Because insurance payouts in Florida are so spectacular? Please, don't lead them along with false hope! /s