> It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
No, the mere act of picking imbues their ticket with a bit of guarantee that you haven't maliciously selected numbers that don't win to cheat them. And I'd expect people's psychologies to pick up on that even if they can't explain why.
Even if that actually was the study that was conducted, it still wouldn't support belief in juju as an explaination.
Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being? Either way, we know more about what's likely to work for future interventions. It's the scientific method.
These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?
Edit: I wrote the above before reaching the end of the article. By the end, the article reaches a similar conclusion, albiet with less science and more mysticism.
Maybe that means it's a good article, leading me to reach its conclusion before taking the stance itself. On the other hand, the article's leading sections gave such a strong impression that the author thought something needs to be drastically changed about the current system — the opposite stance from its conclusion — on seemingly shakey grounds, that I almost didn't finish reading it. I'm not sure what to think, aside from please leave innocent goats alone.
I think most sociological interventions don't follow the scientific method because you need a control group. You can't just use random people you haven't helped elsewhere as a control, because you need to identify all of the external variables that you can so as to properly assess the effectiveness of your method (ruling out changes external to your intervention).
There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
Instead we have this reality where we throw money at problems and claim credit whenever something good happens, and claim we did good work when nothign changes.
That has to carry with it the assumption that the response to similar interventions is long lasting over time. I'm shooting from the hip here, but I'd expect culture, life experiences, etc to play a role in how interventions would play out.
If my hunch were to price right, anything we'd learn there would have a short half live and may not hold out past, say, a generation or two.
I can say anecdotally that my grandparents would respond differently to many situations and interventions than I would. They grew up during the Great Depression and he flew in the war, those two events alone very much impacted how they viewed things for the rest of their life.
Looking at those only a generation behind me and even they seem to respond very differently than I'd expect those in my generation would.
I read the whole article, and in the beginning part the only thing I felt the author wanting to change was my own mind. He is drilling in the idea that common sense interventions more often than not do not work. (maybe even almost always don't work).
His proposed change at the end is that we must always guess and check (emphasis on check) for now, and we may eventually get into a "Psychohistory" level of understanding in the future, but we are far from it now.
> Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being?
Only if the value of whatever smidgen of knowledge you gain (assuming you do actually gain any) outweighs the much-more-than-smidgen cost of the intervention to all the people who weren't helped, and in a significant number of cases were harmed, by it.
As you can see, I don't think whatever benefits are gained actually do outweigh the costs. Which means that no, continuing to try such interventions is not a win.
The lesson we should be learning from all this is to stop trying to do so many things top down. That whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it. Or even concoct a small scale experiment to "test" the idea, with the intention of scaling it up if it seems to work. Still less should you sacrifice a poor goat who did nothing to deserve it, just to underscore the point that your intervention won't work.
The goal of a scientific experiment can't be to "help" people. What you're describing is a political experiment. The goal of science can only be to learn whether it helps people, who it helps, who it hurts, etc.
> whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it
Duplicating a top level comment I just posted, but this all assumes that people will generally respond the same way over time. I'd be surprised if culture, societal norms, life experiences, etc didn't make a meaningful difference and those absolutely change over time.
I think so. But a big part of the article's complaint is that we usually don't even bother checking.
The reason the article resonated with me is because policies and interventions are often sold as "this thing is self-evidently great, and if you oppose it you're an idiot/jerk", rather than "this seems promising, so let's give it a try and scale it up if it works well". When I look at proposed measures at my local government level (for example) almost all of the supporting literature focuses on why the problem is important, rather than why the proposed interventions are likely to succeed. Being "effective" as a legislator means enacting policy, so if you want to be effective you try to keep the conversation on the first part, where it's easier to find agreement.
I agree, but we should be more systematic about it. Instead of one intervention, we should be asking what are our ten best ideas for interventions and try all of them on a small scale in different places.
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
A lot of lottery players will say use a birthday or something. Kind of interesting to think about this as deciding on pure random search or setting a seed and sampling that seed over the space. The behavior becomes much less about juju and maybe more an actual functional one with some biological utility, this ability to sample a defined input across a function set. The lottery aspect is almost blinding the point of why it might be useful to think like that.
>Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
And the ignorant are always the loudest.
The fresh minted safety guy with his shiny new clipboard and vest is a fool running around the plant spewing rabid idiocy straight out of the internet comment section about how everything is non-optimal.
His boss who's been doing things for a decade is more measured, knows you can't tell a guy how to use a screwdriver if you also want to tell him how to use a ladder.
One very successful social intervention in the UK was the push to reduce drink-driving. Starting around the 80s I think, maybe earlier. There was a big media campaign with TV ads as well as implementing and enforcing stricter laws. What I find interesting is that many in the older generation still found it more acceptable to have a few drinks and drive, whereas those growing up during this period largely find it very socially unacceptable.
Generally, criminal enforcement is a very effective way to curb behavior. Not saying it’s a panacea or will reduce a behavior to 0. But look at prohibition and the rate of drinking in the US. It permanently decreased, even after the amendment was repealed.
This article makes an “ok” case for social science being in its infancy (I think it is in some ways, but is more mature than widely thought in others), and a very good case for most social interventions being ineffective due to their being undertaken in methodologically flawed ways.
Most interventions are embarked upon by working backwards from unproven (often political) theories and an extremely limited range of possible tools (“give money”, “take money”, “physically punish”). That leads to the mother of all searching-only-under-the-streetlight fallacies.
But that doesn’t necessarily indict the social science academy/frontiers of understanding (which, again, are plenty flawed—just perhaps not quite as stone-aged as the author suggests). Usually, it indicts the state/designers of the intervention in question. Blaming the sciences sometimes sounds like blaming car airbag designers for motorcyclist deaths.
I’m reminded of the conservation efforts for the spotted owl. Famous academic naturalists were asked to design a conservation plan that worked with logging. They did. The state replied “fine, but we need to clear-cut 90% of the forest you asked for, so make do with 10% of the habitat”. The program did poorly, but not because the academic understanding of how to address the problem was as flawed.
The article talks about measurement of impact as though it's a totally value-neutral idea, but it's absolutely not:
* Measurement requires substantial logistical overhead; both up-front in creating a plan for measurement, and during intervention due to the extra record keeping involved. This is a (probably frequently large) drain on program efficacy. Just listen to any schoolteacher's take on standardized testing, evaluation rubrics, etc.
* Some contexts, and some types of intervention are more amenable to measurement than others. Insisting that nothing which is not measured can be valuable biases our decision-making towards the most measurable actions, which are likely not the best. Strong correlation with simplistic actions, short timelines, and lack of responsiveness to individual context.
* Goodhart's law.
Author (and everyone) would benefit from the book Seeing Like a State.
> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.
The author is assuming good faith on behalf of the Arkansas state government that the purpose of the requirement was to increase employment, rather than merely cutting down on the number of citizens successfully receiving Medicaid.
tough article. 90% is a tongue in cheek ridicule, with lots of data points, and then the bait and switch at the end, with far less argumentation.
mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.
maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.
fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.
this reminds me of the thought that a country run by scientists and engineers should be better. but there's none, and i think this can easily be understood by Daniel Kahneman's fast/slow thinking.
in other words, government is time-constrained, so when a problem is sufficiently complex, there's just not enough time to really flesh out a good solution.
also, humans can be selfish and will occasionally try to game the system, which changes your problem into an active adversary. and when you have active adversaries, you're playing a completely different game (indeed, when designing the policy in the first place you may not have even considered it a game with opponents!).
That reminds me how people say china is a nation run by engineers and the US is a nation run by lawyers. I think both have its pros and cons but just regarding how much shit a nation gets done, my money is gonna be on engineers for sure.
That being said, I wonder if there is a case to be made that for there to be more social & policy interventions that actually are evidence-based and more rigorous, do we need to have technocrats that are not beholden to the short-termism of the democratic political system? But wouldn't they just accumulate too much power without enough accountability? Who will watch the watchers?
Chinese technocrats seem to achieve a lot when it comes to technical things, but not as much in societal context.
The "lying flat" movement seems to persist, the willingness of the young to accept 996 jobs is close to zero, and all attempts to raise the abysmal Chinese fertility rate have so far failed. The (long abandoned) One Child Policy on the other hand worked way too well and might have cost them 200 million young people.
> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
If you read the report instead of a blog post with an agenda you will find out that the plain packaging accelerated decline in smoking by about 25% (p<0.01)
But the paper cited by the blog post also covers the smoking prevalence data and finds it to not be statistically significant. Now you can look at their model and decide if you find that conclusion convincing, but I don't think just pulling a number from a different report (especially one from the Australian government, which has its own agenda in motivating its own policy) is really such a gotcha here.
To be clear, I don't want to play the game of paper vs. paper. I simply think there is more depth of detail here than can be nicely summarised in a comment.
Was about to comment the same thing, even if they smoked more, the program was meant to reduce the trend of new smokers, period. And it was successful about that
This seems like a super important corrective, but I don't see the agenda you're seeing. The article concludes by arguing we need to do lots more stuff, and suggests that people whose takeaway from the article is "stop doing social science interventions" should take the place of the goats in the sacrifices.
If true, it would somewhat undermine another point of the article, which is that since almost all of our interventions are ineffective we need to invest more in understanding which ones actually work.
I'm sure that many other examples could be found to replace this one, but if a lot of the "ineffective interventions" turned out to be effective under closer scrutiny then it would be a blow to the thesis for sure.
This really is not the message though. From the post:
“That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried.”
From reading the whole post, I gather that he’s truly not advocating some libertarian angle, he really is writing on the observation that policy is hard, and it’s nearly impossible to actually predict all the effects of societal intervention.
From my reading, I’m not sure even sure that he’s advocating caution or reluctance to social policy. Just goat sacrifices!
I think neither you nor the comment you are replying to are understanding what the article is trying to say.
It is saying, acts must be done to reaffirm the commitment to the goal, the goat is a litmus test as to whether the goal is worth it. Not to abandon the goal, some things are worth the sacrifice, but to be aware of it.
Wow. So we must not ask if any govt interventions work for fear that the wrong answer will empower political groups disfavoured by the govt and govt staff?
The entire post seems to be along the lines of ‘no laws ever work!’ and then you scratch the surface and they do work.
There was a post here about Mississippi miracle https://en.wikipedia.org/wiki/Mississippi_miracle recently which was due to one of the programs mentioned in the original post claiming it produced no improvements.
As in the original post is flat put lying to push an agenda. You can google for like 2mins on many of the points and see counter evidence.
The irony here of course is that most comments on HN - including this one and the one it responds to - are arm-chair commentary on generals' actions, even less informed, even more disconnected.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
Smoking cigarettes has an incredibly massive public health cost. Addiction to nicotine makes people wring their hands, but population wide doesn't have a significant cost and might even be a net benefit. Vaping likewise has an incomparably minimal public health cost, though at the margins there's probably more injury than nicotine use alone.
Yes, usually there are unintended consequences. The most important unintended consequence here was the anti-vaping sentiment engendered by the anti-smoking campaign choosing to conflate nicotine with the harms from inhaling combustion byproducts and tacitly embracing a cateforical, puritanical morality regarding substance use.
But this policy has been very successful ... Every policy has some unwanted side effects but smoking overall has seen large reductions since its introduction.
I think some "interventions" may also be the result of political compromises or "deals" where one side might want the intervention to be ineffective.
E.g. if the tobacco industry is forced to choose between either hard bans on smoking, or a highly visible but largely symbolic measure like changing the packaging, they will probably choose the latter.
Want to cut a social program because you don't want to pay for it, but don't want to look overtly evil? First, shoot out the kneecaps. You're making it more efficient by cutting weight! Second, notice that it can't walk anymore and cut it entirely. We need something better! Optional third step: give a high-minded speech about how attempts to solve problems are doomed so we shouldn't even try, or shouldn't try without an infinite amount of meetings first.
Maybe we should save the anger for the rich and powerful people selling addictive cancer rather than the ones who need to find some way of reducing their harm without getting sued or voted out.
Take the taxes off cigarettes. Attach them to a tax stamp which has to be purchased separately and redeemed upon price of cigarettes. Make it a hassle on top of more expensive.
>Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control.
I mean I think that's definitely true in some cases but it's a wide wide world out there, and the universe of things that count as "social programs" is impossibly vast, and some of them are quite well researched and simply work when rolled out. Congestion pricing in NYC for instance was the product of policy nerds and seems like it simply worked.
I completely agree there's unintended consequences and regulation can make things worse but not to the point of a universal anti government ethos.
Software engineering, for all its faults as a field, figured this out a long time ago. The essence of agile (the good parts) is: the people doing the work understand it best, and the plan is wrong (and making the plan is a waste of time).
Most implementations of Agile that I've seen do keep the plan, but the plan is allowed and even encouraged to change. High level planning still provides valuable insight, especially from a business perspective which is the whole point of the majority of systems.
"Findings were mixed amongst the four studies assessing consumption, with some studies finding no difference and some studies finding evidence of a decrease."
This also ignores that the intervention was intended to reduce the number of people taking up smoking and to encourage current smokers to quit, which appears to have happened according to that same review.
> In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.
And yes, even if that's true, it's separate from the question of how effective it was as stopping nonsmokers from starting.
Imagine if they had been able to take proper measures decades ago, bankrupting the tobacco companies and imprisoning their executives for covering up the damaging effects of their products and engaging in a massive campaign of false advertising. Maybe that would have moved the needle.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.
For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
I've seen this mentioned in the context of leadership. Someone said the best leaders get people to do things they wanted to do anyway. They just act as a kind of meeting point (or 'lightning rod', take your pick) that senses the general mood, and figures out a way forward that most of the group can agree with. Repeated (sometimes continuous) back-and-forth is important here.
In politics it's often the other way around: many people are unhappy, politician comes up with a plan to do something, and tries to push that through. Such action often fails as it -mostly- ignores feedback from the people the changes are imposed upon.
Acknowledging a problem is important. But more important is understanding the problem through the eyes of those affected by it, understanding how an action would change things from that point of view, and adjusting control knobs as you go. Few leaders are good at ALL those things.
The thing that is missing from most of these government initiatives (in the US at least) is a mechanism and willingness to revisit and iterate. That's how we do software these days, but social programs are developed and shipped in a full waterfall cycle, then politicians just spend decades arguing about whether it should be cancelled due to some flaws or kept exactly as-is. What's missing are real data-driven attempts to close the feedback loop and fix the issues to achieve the stated objective.
Measuring success is good, but we also need to exercise the ability to make small updates and continue measuring, instead of just using that data to argue on the side of "throw it all out!" or "we can't touch it, it's essential!"
Every government initiative of this sort either creates a new bureaucracy or bolsters an older one, which brings the Iron Law of Bureaucracy into play: whatever its original goal, every bureaucracy ultimately exists to perpetuate itself.
Also the more qualifications and requirements you add to your social programs, the more bureaucracy is required to process, document, and enforce those qualifications.
For example, if you implement a free lunch program for all public school students in a state, you actually get to subtract out the amount of money that each school district spends dealing with reduced and free lunch program qualifications for a subset of students, all of the paperwork involved in maintaining that list and re-verifying it annually, and the system that manages their account balances at the cafeteria register.
Not having a metric set out in advance may, in fact, be more rigging resistant! unfortunately.
But the real deal is that none of it matters because the program itself is the goat sacrifice. It's not there to work, it's there to signal the proponents values and ingratiate themselves with the goat suppliers. It's actually better for theses purposes if it DOESNT work, because then you can perpetually campaign on solving the same problem and conserve the considerable effort that went into convincing people that the problem exists and needs solving.
I think giving all schoolchildren free food is self-evidently a good thing and "verifiable measure of success" requirements are just a way to manufacture BS justifications for taking food away from children from famously unreliable policy outcome studies.
Lots of examples in this article, but none of them go deep enough to matter. Just like people bringing up the cigarette issue, I see other factors in sone of the other topics, such as medical school (residency slots weren't mentioned, default assumption was motivators were solely economic, etc).
I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.
Interestingly, we do have quite a lot of material and studies on how to change human behavior and a whole lot of data supporting it: it's called marketing. I think this goes to show that when the entire economy is working on a problem we actually can make an impact, it's just unfortunately all going towards projects that make people poorer and more depressed.
There are limits to that as well though. The USSR put an enormous amount of resources towards creating the “New Soviet Man”, including education, “re-education”, films, literature, psychology, and even biology (eugenics).
You have to make sure your end goal is realistic and attainable. The Soviet example is extreme, but sometimes even changes that look deceptively simple aren’t realistic due to deeply embedded social issues or plain human nature.
Generally, though, I agree that our social priorities seem to be misplaced.
> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
It bothers me that so much of modern life is opaque and assumes that no one has the time and energy to read the footnotes.
I went and looked at the first study on the School Improvement Grants,
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
At one end it says, "Although schools implementing SIG-funded models reported using more SIG promoted practices than other schools, we found no evidence that SIG caused those schools to implement more practices."
What are these practises? Well... it seems that as a pre-requisite to getting funding states were made to divide schools into four categories / intervention modes,
> ED required that each SIG-awarded school under Tier I or Tier II implement one of four school intervention models. These models required specific practices:
>
> • Transformation. This model required schools to replace the principal, adopt a teacher and principal evaluation system that accounted for student achievement growth as a significant factor, adopt a new governance structure, institute comprehensive instructional reforms, increase learning time, create community-oriented schools, and have operational flexibility.
> • Turnaround. This model required schools to replace the principal, replace at least 50 percent of the school staff, institute comprehensive instructional reforms, increase learning time, create community-oriented schools, and have operational flexibility.
> • Restart. This model required schools to convert to a charter school or close and reopen under the management of a charter management organization or education management organization.
> • School closure. This model required districts to close schools and enroll their students in higher-achieving schools within the district.
>
> These required practices can be grouped into the four main topic areas promoted by SIG. Table ES.1 lists these four broad areas and the objectives promoted by SIG within each. For example, replacing the principal falls under the “identifying and rewarding effective teachers and principals and removing ineffective ones” objective within the “developing and increasing teacher and principal effectiveness” topic area.
>
> The objectives listed in the table cover all practices promoted by SIG, which different models designated as either required or permissible.
In plain english, it seems that they're saying at one end that schools were "transformed," turned around, restarted, or closed in order to get funding. They also promited performance review plans and quarterly / yearly reviews. Fairly standard administrative procedure.
But then they start the summary by saying that there's "no evidence that SIG caused those schools to implement more practices" – the ones which were heavily promoted by the program and were a part of it?
I feel like I'm misunderstanding something as these seem to be two mutually and logically exclusive things. You are requiring something for funding, but then you're saying afterwards that you can't be sure that they did related things just because you required it?
What am I missing here?
Also some of these graphs look wrong. The error bars on Figure ES.3. go from ~ -0.2 to 0.25... With error bars that large isn't the graph meaningless?
I just looked at the study for one claim "might be a fluke, but it also showed up in a study where researchers wiped out a bunch of people’s medical debt and it maybe made them more depressed".
The paper's abstract says "First, we find no impact of debt relief on credit access, utilization, and financial distress on average. [...] Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis." which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)
Medical debt with a face value of $169 million was wiped out for 83,401 people, yet the study failed to find any benefit. I suspect this is representative of the quality of the reasoning in the rest of this article.
> which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)
On page 20 they describe using the Westfall and Young method to adjust for multiple hypothesis testing and report adjusted and unadjusted p-values.
They also pre registered their analysis with depression as a primary outcome and breaking down by medical debt quartile as pre registered subgroup analysis.
In the highest quartile of medical debt group, 45.9% percent in the no debt forgiveness group had at least moderate depression vs 58.3% in the debt forgiveness group, p=0.002
Dismissing this as a meaningless result seems unfair given that they preregistered their analysis and adjusted for multiple comparisons.
I’m not a stats expert and haven’t spent much time with the paper, but I’d suggest checking the methods section before making claims.
So the proper way to do public policy is actually run interventions like clinical trials before allocating more funds. But this runs counter to political reality where politicians need to signal to constituents over realtively short time scales (2-4 years). In effect, the policy is the experiment.
It's unrealistic to run pilots, data collection, and followup in 4 years for many political decisions.
The best policy recommendation here is simply to enforce data collection for every decision. This would also incentivize politicans to only propose policies they truely believe will work.
Money will still be burned, but at the very least over long term our understanding will grow, and bad policies will gradually be disincentivized.
This is a strange reaction to failed predictions. Very defeatist.
There's good evidence that we can improve accuracy on the predicted outcomes of policies, e.g. delphi method or prediction markets. See the "replication markets".
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
I read through this expecting some brilliant justification for the goat, and there wasn't one. All I saw was "HEY a lot of the time big experiments don't work."
This could have been much shorter.
"Generally, you should have skin-in-the-game. If you're going to propose something that has the potential for harmful waste, be sure that this is taken into account before doing it."
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
I had a related idea about vaccination back during COVID. Have Hillary Clinton be quoted saying nobody should be vaccinated and the "Trump vaccine" is bogus. Then have Trump be quoted saying nobody should be vaccinated and the "fake Democrat vaccine" is bogus. Result: everyone gets vaccinated.
A negative version of this, I think, is mandatory diversity education and DEI, which had the effect of making racism and fascism cool and countercultural among the youth. Deplatforming had the same effect. "It's being banned which means it's what THEY don't want you to know!" The big wave of deplatforming in 2019 and 2020 probably helped Trump win in 2024 by way of the Streisand effect.
Yes, I’m not disparaging the practice (though I personally think those who do it are deceived). I’m simply making the point that it’s not a real factor in societal change (which is along the lines of the point the article was trying to make).
ive never had a presentation go well if i didnt place a "sacrificial slide" in the deck, or plan an "oops here is a bug, let me recover, and we're back on track" during the demo
I think it's quite funny how the "unintended consequences" people love to highlight every unintended consequence or ineffective intervention, but also have zero interest in finding out why the interventions did not work out as planned. Wouldn't this be a great starting point for scientific research?
If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".
> Most of those are left-coded failures, so here are some right-coded failures:
"I'm neither left nor right, I'm just an apolitical neoliberal..."
Sacrificing animals was just a way for the powerful in your tribe to share some meat. So yes throw a party when you enact a new policy and invite the plebs.
> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.
> It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
No, the mere act of picking imbues their ticket with a bit of guarantee that you haven't maliciously selected numbers that don't win to cheat them. And I'd expect people's psychologies to pick up on that even if they can't explain why.
Even if that actually was the study that was conducted, it still wouldn't support belief in juju as an explaination.
Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being? Either way, we know more about what's likely to work for future interventions. It's the scientific method.
These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?
Edit: I wrote the above before reaching the end of the article. By the end, the article reaches a similar conclusion, albiet with less science and more mysticism.
Maybe that means it's a good article, leading me to reach its conclusion before taking the stance itself. On the other hand, the article's leading sections gave such a strong impression that the author thought something needs to be drastically changed about the current system — the opposite stance from its conclusion — on seemingly shakey grounds, that I almost didn't finish reading it. I'm not sure what to think, aside from please leave innocent goats alone.
I think most sociological interventions don't follow the scientific method because you need a control group. You can't just use random people you haven't helped elsewhere as a control, because you need to identify all of the external variables that you can so as to properly assess the effectiveness of your method (ruling out changes external to your intervention).
There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
Instead we have this reality where we throw money at problems and claim credit whenever something good happens, and claim we did good work when nothign changes.
That has to carry with it the assumption that the response to similar interventions is long lasting over time. I'm shooting from the hip here, but I'd expect culture, life experiences, etc to play a role in how interventions would play out.
If my hunch were to price right, anything we'd learn there would have a short half live and may not hold out past, say, a generation or two.
I can say anecdotally that my grandparents would respond differently to many situations and interventions than I would. They grew up during the Great Depression and he flew in the war, those two events alone very much impacted how they viewed things for the rest of their life.
Looking at those only a generation behind me and even they seem to respond very differently than I'd expect those in my generation would.
Repeating a successful intervention ~30 times before failure means you can dramatically boost the success to failure ratio.
I read the whole article, and in the beginning part the only thing I felt the author wanting to change was my own mind. He is drilling in the idea that common sense interventions more often than not do not work. (maybe even almost always don't work).
His proposed change at the end is that we must always guess and check (emphasis on check) for now, and we may eventually get into a "Psychohistory" level of understanding in the future, but we are far from it now.
> Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being?
Only if the value of whatever smidgen of knowledge you gain (assuming you do actually gain any) outweighs the much-more-than-smidgen cost of the intervention to all the people who weren't helped, and in a significant number of cases were harmed, by it.
As you can see, I don't think whatever benefits are gained actually do outweigh the costs. Which means that no, continuing to try such interventions is not a win.
The lesson we should be learning from all this is to stop trying to do so many things top down. That whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it. Or even concoct a small scale experiment to "test" the idea, with the intention of scaling it up if it seems to work. Still less should you sacrifice a poor goat who did nothing to deserve it, just to underscore the point that your intervention won't work.
The goal of a scientific experiment can't be to "help" people. What you're describing is a political experiment. The goal of science can only be to learn whether it helps people, who it helps, who it hurts, etc.
> whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it
aint that the truth.
Duplicating a top level comment I just posted, but this all assumes that people will generally respond the same way over time. I'd be surprised if culture, societal norms, life experiences, etc didn't make a meaningful difference and those absolutely change over time.
yea, social science researchers are well aware of longitudinal studies that track populations over time...
I think so. But a big part of the article's complaint is that we usually don't even bother checking.
The reason the article resonated with me is because policies and interventions are often sold as "this thing is self-evidently great, and if you oppose it you're an idiot/jerk", rather than "this seems promising, so let's give it a try and scale it up if it works well". When I look at proposed measures at my local government level (for example) almost all of the supporting literature focuses on why the problem is important, rather than why the proposed interventions are likely to succeed. Being "effective" as a legislator means enacting policy, so if you want to be effective you try to keep the conversation on the first part, where it's easier to find agreement.
By definition, a politician isn't trying be right; they are trying to be popular.
I agree, but we should be more systematic about it. Instead of one intervention, we should be asking what are our ten best ideas for interventions and try all of them on a small scale in different places.
> Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being
No, paying $3B for information of dubious value and which was pretty obvious ahead of time is not a win.
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
A lot of lottery players will say use a birthday or something. Kind of interesting to think about this as deciding on pure random search or setting a seed and sampling that seed over the space. The behavior becomes much less about juju and maybe more an actual functional one with some biological utility, this ability to sample a defined input across a function set. The lottery aspect is almost blinding the point of why it might be useful to think like that.
>Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
And the ignorant are always the loudest.
The fresh minted safety guy with his shiny new clipboard and vest is a fool running around the plant spewing rabid idiocy straight out of the internet comment section about how everything is non-optimal.
His boss who's been doing things for a decade is more measured, knows you can't tell a guy how to use a screwdriver if you also want to tell him how to use a ladder.
One very successful social intervention in the UK was the push to reduce drink-driving. Starting around the 80s I think, maybe earlier. There was a big media campaign with TV ads as well as implementing and enforcing stricter laws. What I find interesting is that many in the older generation still found it more acceptable to have a few drinks and drive, whereas those growing up during this period largely find it very socially unacceptable.
Generally, criminal enforcement is a very effective way to curb behavior. Not saying it’s a panacea or will reduce a behavior to 0. But look at prohibition and the rate of drinking in the US. It permanently decreased, even after the amendment was repealed.
Criminalizing a thing and then arresting people definitely does something.
As you say, it reduced the rate of drinking. It had a few other side effects.
I see there is bunch of failure of understanding in the comments.
Lots of social programs are not working to improve lives of adults that are already messed up.
Aiming at kids is the way to go.
That’s how the YouTube influencers got big, they basically groomed generations of followers.
This article makes an “ok” case for social science being in its infancy (I think it is in some ways, but is more mature than widely thought in others), and a very good case for most social interventions being ineffective due to their being undertaken in methodologically flawed ways.
Most interventions are embarked upon by working backwards from unproven (often political) theories and an extremely limited range of possible tools (“give money”, “take money”, “physically punish”). That leads to the mother of all searching-only-under-the-streetlight fallacies.
But that doesn’t necessarily indict the social science academy/frontiers of understanding (which, again, are plenty flawed—just perhaps not quite as stone-aged as the author suggests). Usually, it indicts the state/designers of the intervention in question. Blaming the sciences sometimes sounds like blaming car airbag designers for motorcyclist deaths.
I’m reminded of the conservation efforts for the spotted owl. Famous academic naturalists were asked to design a conservation plan that worked with logging. They did. The state replied “fine, but we need to clear-cut 90% of the forest you asked for, so make do with 10% of the habitat”. The program did poorly, but not because the academic understanding of how to address the problem was as flawed.
The article talks about measurement of impact as though it's a totally value-neutral idea, but it's absolutely not:
Author (and everyone) would benefit from the book Seeing Like a State.Great book, funny enough I ended up reading that book thanks to a footnote in one of his prior writings where he borrowed an example from it.
https://www.experimental-history.com/p/the-one-science-refor...
> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.
The author is assuming good faith on behalf of the Arkansas state government that the purpose of the requirement was to increase employment, rather than merely cutting down on the number of citizens successfully receiving Medicaid.
tough article. 90% is a tongue in cheek ridicule, with lots of data points, and then the bait and switch at the end, with far less argumentation.
mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.
maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.
fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.
this reminds me of the thought that a country run by scientists and engineers should be better. but there's none, and i think this can easily be understood by Daniel Kahneman's fast/slow thinking.
in other words, government is time-constrained, so when a problem is sufficiently complex, there's just not enough time to really flesh out a good solution.
also, humans can be selfish and will occasionally try to game the system, which changes your problem into an active adversary. and when you have active adversaries, you're playing a completely different game (indeed, when designing the policy in the first place you may not have even considered it a game with opponents!).
China is basically run by technocrats and their achievements are vast. We will also see how Serbia does after this recent election.
That reminds me how people say china is a nation run by engineers and the US is a nation run by lawyers. I think both have its pros and cons but just regarding how much shit a nation gets done, my money is gonna be on engineers for sure.
That being said, I wonder if there is a case to be made that for there to be more social & policy interventions that actually are evidence-based and more rigorous, do we need to have technocrats that are not beholden to the short-termism of the democratic political system? But wouldn't they just accumulate too much power without enough accountability? Who will watch the watchers?
Chinese technocrats seem to achieve a lot when it comes to technical things, but not as much in societal context.
The "lying flat" movement seems to persist, the willingness of the young to accept 996 jobs is close to zero, and all attempts to raise the abysmal Chinese fertility rate have so far failed. The (long abandoned) One Child Policy on the other hand worked way too well and might have cost them 200 million young people.
Social engineering is hard.
> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
If you read the report instead of a blog post with an agenda you will find out that the plain packaging accelerated decline in smoking by about 25% (p<0.01)
https://www.health.gov.au/sites/default/files/study-of-the-i... (page 12)
They also tried quitting more often (p=0.013)
https://pmc.ncbi.nlm.nih.gov/articles/PMC3710988/
But the paper cited by the blog post also covers the smoking prevalence data and finds it to not be statistically significant. Now you can look at their model and decide if you find that conclusion convincing, but I don't think just pulling a number from a different report (especially one from the Australian government, which has its own agenda in motivating its own policy) is really such a gotcha here.
To be clear, I don't want to play the game of paper vs. paper. I simply think there is more depth of detail here than can be nicely summarised in a comment.
Was about to comment the same thing, even if they smoked more, the program was meant to reduce the trend of new smokers, period. And it was successful about that
This seems like a super important corrective, but I don't see the agenda you're seeing. The article concludes by arguing we need to do lots more stuff, and suggests that people whose takeaway from the article is "stop doing social science interventions" should take the place of the goats in the sacrifices.
If true, it would somewhat undermine another point of the article, which is that since almost all of our interventions are ineffective we need to invest more in understanding which ones actually work.
I'm sure that many other examples could be found to replace this one, but if a lot of the "ineffective interventions" turned out to be effective under closer scrutiny then it would be a blow to the thesis for sure.
What agenda? Like a hidden motive?
The not-so-hidden message is, "Interventions don't do anything, so make the government smaller, and let corporations do whatever they want."
This really is not the message though. From the post:
“That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried.”
From reading the whole post, I gather that he’s truly not advocating some libertarian angle, he really is writing on the observation that policy is hard, and it’s nearly impossible to actually predict all the effects of societal intervention.
From my reading, I’m not sure even sure that he’s advocating caution or reluctance to social policy. Just goat sacrifices!
Well, my assumption was he doesn't really want more goat sacrifices but rather less interventions. Otherwise, what does he have against goats?
I think neither you nor the comment you are replying to are understanding what the article is trying to say.
It is saying, acts must be done to reaffirm the commitment to the goal, the goat is a litmus test as to whether the goal is worth it. Not to abandon the goal, some things are worth the sacrifice, but to be aware of it.
Wow. So we must not ask if any govt interventions work for fear that the wrong answer will empower political groups disfavoured by the govt and govt staff?
"Because there are sometimes unintended consequences to doing stuff, we should never do anything" is a great motto to live by. (/S)
The entire post seems to be along the lines of ‘no laws ever work!’ and then you scratch the surface and they do work.
There was a post here about Mississippi miracle https://en.wikipedia.org/wiki/Mississippi_miracle recently which was due to one of the programs mentioned in the original post claiming it produced no improvements.
As in the original post is flat put lying to push an agenda. You can google for like 2mins on many of the points and see counter evidence.
The irony here of course is that most comments on HN - including this one and the one it responds to - are arm-chair commentary on generals' actions, even less informed, even more disconnected.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
Smoking cigarettes has an incredibly massive public health cost. Addiction to nicotine makes people wring their hands, but population wide doesn't have a significant cost and might even be a net benefit. Vaping likewise has an incomparably minimal public health cost, though at the margins there's probably more injury than nicotine use alone.
Yes, usually there are unintended consequences. The most important unintended consequence here was the anti-vaping sentiment engendered by the anti-smoking campaign choosing to conflate nicotine with the harms from inhaling combustion byproducts and tacitly embracing a cateforical, puritanical morality regarding substance use.
But this policy has been very successful ... Every policy has some unwanted side effects but smoking overall has seen large reductions since its introduction.
based on the policy or the humongous multi-decade campaign to demonise smoking?
I think some "interventions" may also be the result of political compromises or "deals" where one side might want the intervention to be ineffective.
E.g. if the tobacco industry is forced to choose between either hard bans on smoking, or a highly visible but largely symbolic measure like changing the packaging, they will probably choose the latter.
Yep, see also: the one-two punch.
Want to cut a social program because you don't want to pay for it, but don't want to look overtly evil? First, shoot out the kneecaps. You're making it more efficient by cutting weight! Second, notice that it can't walk anymore and cut it entirely. We need something better! Optional third step: give a high-minded speech about how attempts to solve problems are doomed so we shouldn't even try, or shouldn't try without an infinite amount of meetings first.
This happens a lot.
Maybe we should save the anger for the rich and powerful people selling addictive cancer rather than the ones who need to find some way of reducing their harm without getting sued or voted out.
Take the taxes off cigarettes. Attach them to a tax stamp which has to be purchased separately and redeemed upon price of cigarettes. Make it a hassle on top of more expensive.
Those are inherent to complex systems.
You can still try and revert back if it fails.
>Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control.
I mean I think that's definitely true in some cases but it's a wide wide world out there, and the universe of things that count as "social programs" is impossibly vast, and some of them are quite well researched and simply work when rolled out. Congestion pricing in NYC for instance was the product of policy nerds and seems like it simply worked.
I completely agree there's unintended consequences and regulation can make things worse but not to the point of a universal anti government ethos.
Software engineering, for all its faults as a field, figured this out a long time ago. The essence of agile (the good parts) is: the people doing the work understand it best, and the plan is wrong (and making the plan is a waste of time).
Most implementations of Agile that I've seen do keep the plan, but the plan is allowed and even encouraged to change. High level planning still provides valuable insight, especially from a business perspective which is the whole point of the majority of systems.
> it looks like the policy caused smokers to switch to cheaper brands
This is documented, but it is not clear how much this is due to packaging vs. Australia having the highest tobacco tax in the world.
> and to smoke more
{{Citation needed}}
This claim is directly contradicted by this Cochrane review. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD...
"Findings were mixed amongst the four studies assessing consumption, with some studies finding no difference and some studies finding evidence of a decrease."
This also ignores that the intervention was intended to reduce the number of people taking up smoking and to encourage current smokers to quit, which appears to have happened according to that same review.
From the linked abstract (https://pubmed.ncbi.nlm.nih.gov/32958901/):
> In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.
And yes, even if that's true, it's separate from the question of how effective it was as stopping nonsmokers from starting.
Imagine if they had been able to take proper measures decades ago, bankrupting the tobacco companies and imprisoning their executives for covering up the damaging effects of their products and engaging in a massive campaign of false advertising. Maybe that would have moved the needle.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.
For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
"to be seen to be doing something"
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
"Effortless action": https://en.wikipedia.org/wiki/Wu_wei
I've seen this mentioned in the context of leadership. Someone said the best leaders get people to do things they wanted to do anyway. They just act as a kind of meeting point (or 'lightning rod', take your pick) that senses the general mood, and figures out a way forward that most of the group can agree with. Repeated (sometimes continuous) back-and-forth is important here.
In politics it's often the other way around: many people are unhappy, politician comes up with a plan to do something, and tries to push that through. Such action often fails as it -mostly- ignores feedback from the people the changes are imposed upon.
Acknowledging a problem is important. But more important is understanding the problem through the eyes of those affected by it, understanding how an action would change things from that point of view, and adjusting control knobs as you go. Few leaders are good at ALL those things.
> aim is to be seen to be doing something.
A signal of an act. Not the act itself. Post l-modernisms ugly head rears itself once again.
Seems like every social program should have a verifiable measure of success and a termination clause if the measure indicates that it isn't working.
But as software people we all know that turning a measure into a target yields problems.
The thing that is missing from most of these government initiatives (in the US at least) is a mechanism and willingness to revisit and iterate. That's how we do software these days, but social programs are developed and shipped in a full waterfall cycle, then politicians just spend decades arguing about whether it should be cancelled due to some flaws or kept exactly as-is. What's missing are real data-driven attempts to close the feedback loop and fix the issues to achieve the stated objective.
Measuring success is good, but we also need to exercise the ability to make small updates and continue measuring, instead of just using that data to argue on the side of "throw it all out!" or "we can't touch it, it's essential!"
Every government initiative of this sort either creates a new bureaucracy or bolsters an older one, which brings the Iron Law of Bureaucracy into play: whatever its original goal, every bureaucracy ultimately exists to perpetuate itself.
Also the more qualifications and requirements you add to your social programs, the more bureaucracy is required to process, document, and enforce those qualifications.
For example, if you implement a free lunch program for all public school students in a state, you actually get to subtract out the amount of money that each school district spends dealing with reduced and free lunch program qualifications for a subset of students, all of the paperwork involved in maintaining that list and re-verifying it annually, and the system that manages their account balances at the cafeteria register.
whomever is profiting will just rig the metric.
Not having a metric set out in advance may, in fact, be more rigging resistant! unfortunately.
But the real deal is that none of it matters because the program itself is the goat sacrifice. It's not there to work, it's there to signal the proponents values and ingratiate themselves with the goat suppliers. It's actually better for theses purposes if it DOESNT work, because then you can perpetually campaign on solving the same problem and conserve the considerable effort that went into convincing people that the problem exists and needs solving.
I think giving all schoolchildren free food is self-evidently a good thing and "verifiable measure of success" requirements are just a way to manufacture BS justifications for taking food away from children from famously unreliable policy outcome studies.
These programs are pitched along with statistics, so maybe it’s on us to demand the methods of those statistics continue with the program.
Lots of examples in this article, but none of them go deep enough to matter. Just like people bringing up the cigarette issue, I see other factors in sone of the other topics, such as medical school (residency slots weren't mentioned, default assumption was motivators were solely economic, etc).
I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.
Interestingly, we do have quite a lot of material and studies on how to change human behavior and a whole lot of data supporting it: it's called marketing. I think this goes to show that when the entire economy is working on a problem we actually can make an impact, it's just unfortunately all going towards projects that make people poorer and more depressed.
There are limits to that as well though. The USSR put an enormous amount of resources towards creating the “New Soviet Man”, including education, “re-education”, films, literature, psychology, and even biology (eugenics).
You have to make sure your end goal is realistic and attainable. The Soviet example is extreme, but sometimes even changes that look deceptively simple aren’t realistic due to deeply embedded social issues or plain human nature.
Generally, though, I agree that our social priorities seem to be misplaced.
This is so inspiring I'll start to do the same for my real life.
> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
It bothers me that so much of modern life is opaque and assumes that no one has the time and energy to read the footnotes.
I went and looked at the first study on the School Improvement Grants,
Is this true? I looked at the linked report and it's strange, https://ies.ed.gov/sites/default/files/migrated/nces_pubs/nc...At one end it says, "Although schools implementing SIG-funded models reported using more SIG promoted practices than other schools, we found no evidence that SIG caused those schools to implement more practices."
What are these practises? Well... it seems that as a pre-requisite to getting funding states were made to divide schools into four categories / intervention modes,
In plain english, it seems that they're saying at one end that schools were "transformed," turned around, restarted, or closed in order to get funding. They also promited performance review plans and quarterly / yearly reviews. Fairly standard administrative procedure.But then they start the summary by saying that there's "no evidence that SIG caused those schools to implement more practices" – the ones which were heavily promoted by the program and were a part of it?
I feel like I'm misunderstanding something as these seem to be two mutually and logically exclusive things. You are requiring something for funding, but then you're saying afterwards that you can't be sure that they did related things just because you required it?
What am I missing here?
Also some of these graphs look wrong. The error bars on Figure ES.3. go from ~ -0.2 to 0.25... With error bars that large isn't the graph meaningless?
I just looked at the study for one claim "might be a fluke, but it also showed up in a study where researchers wiped out a bunch of people’s medical debt and it maybe made them more depressed".
The paper's abstract says "First, we find no impact of debt relief on credit access, utilization, and financial distress on average. [...] Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis." which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)
Medical debt with a face value of $169 million was wiped out for 83,401 people, yet the study failed to find any benefit. I suspect this is representative of the quality of the reasoning in the rest of this article.
> which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)
On page 20 they describe using the Westfall and Young method to adjust for multiple hypothesis testing and report adjusted and unadjusted p-values.
They also pre registered their analysis with depression as a primary outcome and breaking down by medical debt quartile as pre registered subgroup analysis.
In the highest quartile of medical debt group, 45.9% percent in the no debt forgiveness group had at least moderate depression vs 58.3% in the debt forgiveness group, p=0.002
Dismissing this as a meaningless result seems unfair given that they preregistered their analysis and adjusted for multiple comparisons.
I’m not a stats expert and haven’t spent much time with the paper, but I’d suggest checking the methods section before making claims.
Which do you think is more likely: That medical debt forgiveness causes depression, or there is something else going on?
So the proper way to do public policy is actually run interventions like clinical trials before allocating more funds. But this runs counter to political reality where politicians need to signal to constituents over realtively short time scales (2-4 years). In effect, the policy is the experiment.
It's unrealistic to run pilots, data collection, and followup in 4 years for many political decisions.
The best policy recommendation here is simply to enforce data collection for every decision. This would also incentivize politicans to only propose policies they truely believe will work.
Money will still be burned, but at the very least over long term our understanding will grow, and bad policies will gradually be disincentivized.
This is a strange reaction to failed predictions. Very defeatist.
There's good evidence that we can improve accuracy on the predicted outcomes of policies, e.g. delphi method or prediction markets. See the "replication markets".
This article reminds me of a paper from Michael Huemer, In Praise Of Passivity. If you enjoyed the op, you'll probably like this.
https://spot.colorado.edu/~huemer/papers/passivity.htm
5 years later...
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
I read through this expecting some brilliant justification for the goat, and there wasn't one. All I saw was "HEY a lot of the time big experiments don't work."
This could have been much shorter.
"Generally, you should have skin-in-the-game. If you're going to propose something that has the potential for harmful waste, be sure that this is taken into account before doing it."
Problem: Graffiti
Solution:
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
Graffiti will be dead within ten years. :-)
I had a related idea about vaccination back during COVID. Have Hillary Clinton be quoted saying nobody should be vaccinated and the "Trump vaccine" is bogus. Then have Trump be quoted saying nobody should be vaccinated and the "fake Democrat vaccine" is bogus. Result: everyone gets vaccinated.
A negative version of this, I think, is mandatory diversity education and DEI, which had the effect of making racism and fascism cool and countercultural among the youth. Deplatforming had the same effect. "It's being banned which means it's what THEY don't want you to know!" The big wave of deplatforming in 2019 and 2020 probably helped Trump win in 2024 by way of the Streisand effect.
> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.
That... that's not a failure, that's exactly the point of this requirement
Unless you’re being ironic, the claimed intent was that coverage would stay stable or increase and employment would increase.
that's the problem with right-wing lawmakers, you never know whether they lie or are ill-informed
I realize the title is clickbait. Still, my immediate reaction is, “Yes, because that worked out so well for Haiti.”
It works for the Gurkhas, who fought in both World Wars. They still “sacrifice” goats for Dashain.
Yes, I’m not disparaging the practice (though I personally think those who do it are deceived). I’m simply making the point that it’s not a real factor in societal change (which is along the lines of the point the article was trying to make).
Is there not also temple in Guwahati, Assam that specializes in goat sacrifice or one in Kathmandu Pashupatinath?
ive never had a presentation go well if i didnt place a "sacrificial slide" in the deck, or plan an "oops here is a bug, let me recover, and we're back on track" during the demo
it's the surprising surprises that get you
I have no comment on the content, but man is it refreshing to read such a well-written article
I'm getting Malcolm Gladwell vibes.
Cherry picking failed programmes does not mean all programmes fail
Love the title, made me laugh :)
Viktor Pelevin has a book where Russian politicians sacrifice animals at oil wells to ensure Russia's continued prosperity.
I think it's quite funny how the "unintended consequences" people love to highlight every unintended consequence or ineffective intervention, but also have zero interest in finding out why the interventions did not work out as planned. Wouldn't this be a great starting point for scientific research?
If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".
> Most of those are left-coded failures, so here are some right-coded failures:
"I'm neither left nor right, I'm just an apolitical neoliberal..."
Sacrificing animals was just a way for the powerful in your tribe to share some meat. So yes throw a party when you enact a new policy and invite the plebs.
> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.
My Hindu friend sacrificed a gourd instead of a goat for the last full moon.