This is a great list for people who want to smugly say "Um, actually" a lot in conversation.
Based on my brief stint doing data work in psychology research, amongst many other problems they are AWFUL at stats. And it isn't a skill issue as much as a cultural one. They teach it wrong and have a "well, everybody else does it" attitude towards p-hacking and other statistical malpractice.
SF author Michael Flynn was a process control engineer as his day job; he wrote about how designing statistically valid experiments is incredibly difficult, and the potential for fooling yourself is high, even when you really do know what you are doing and you have nearly perfect control over the measurement setup.
And on top of it you're trying to measure the behavior of people not widgets; and people change their behavior based on the context and what they think you're measuring.
There was a lab set up to do "experimental economics" at Caltech back in the late 80's/early 90's. Trouble is, people make different economic decisions when they are working with play money rather than real money.
Expermential Design is one of the big four adacemic subjects within Statistics. The math is complex even before the issues of the effects of the expermential situation.
I was very surprised at how many statistical methods are taught in undergraduate psychology. Far more statistics than I ever touched in engineering for sure. Yet the undergrads really treated statistics as a cookbook, where they just wanted to be told the recipe and they'd follow it. Honestly they'd have been better off just eyeballing data and collaborating with statisticians for the analysis.
>meta-analyses and systematic reviews have shown significant evidence for the effects of stereotype threat, though the phenomenon defies over-simplistic characterization.[22][23][24][25][26][27][28][9]
Failing to reproduce an effect doesn't prove it isn't real. Mythbusters would do this all the time.
On the other hand, some empires are built on publication malpractice.
One of the worst that I know is John Gottman. Marriage counselling based on 'thin slicing'/microexpressions/'Horsemen of the Apocalypse'. His studies had been exposed as fundamentally flawed, and training based on his principles performed worse than prior offerings, before he was further popularized by Malcolm Gladwell in Blink.
This type of intellectual dishonesty underlies both of their careers.
Um, actually I’d say it is the responsibility of all scientists, both professional and amateur, to point out falsehoods when they’re uttered, and not an act of smugness.
[um], has contexts but is usually a cue, that an unexpected, off the average, something is about to be said.
[actually], is a neutral declaration that some cognitive structure was presented, but is at odds with physically observable fact that will now be laid out to you.
As someone who's part of a startup (hrpotentials.com) trying to bring truly scientifically valid psychological testing into HR processes .... yeah. We've been at it for almost 7 years, and we're finally at a point where we can say we have something that actually makes scientific sense - and we're not inventing anything new, just commercializing the science! It only took an electrical engineer (not me) with a strong grasp of statistics working for years with a competent professor of psychology to separate the wheat from the chaff. There's some good science there it's just ... not used much.
How are you going to get around Griggs v. Duke Power Co.? AFAIK, personality tests have not (yet) been given the regulatory eye, but testing cognitive ability has.
Nonreplicable publications are cited more than replicable ones (2021)
> We use publicly available data to show that published papers in top psychology, economics, and general interest journals that fail to replicate are cited more than those that replicate. This difference in citation does not change after the publication of the failure to replicate. Only 12% of postreplication citations of nonreplicable findings acknowledge the replication failure.
This is at least partially a failure in publication. Once a paper is published, it's usually left up in the same state forever. If it fails to replicate, that data is published somewhere else. So when someone references the paper, and the diligent reader follows up and reads the reference, it looks convincing, just as it did when first published. It's not reasonable to expect the reader, or even the writer, to be so well versed in all the thousands and thousands of papers published that they know when something has failed to be replicated.
What we need is for every paper to be published alongside a stats card that is kept up to date. How many times it's been cited, how many times people tried to replicate it, and how many times they failed.
This feels like some sort of truth telling paradox, where if you assume the study is true, then seeing a citation like this means it's likely not true.
There may be minute details like having a confident frame of reference for the confidence tests. Cultures, even psychologies might swing certain ideas and their compulsions.
Thanks for providing the reference, that's useful context. Those are awful replication rates, worse than a coin flip. Sounds like the OP can add their own introduction to their list. From the introduction:
> Most results in the field do actually replicate and are robust[citation needed]
The incentive of all psychology researchers is to do new work rather than replications. Because of this, publicly-funded psychology PhDs should be required to perform study replication as part of their training. Protocol + results should be put in a database.
Sure, dump it on the lowest level employee, who has the least training and the most to lose. Punish them for someone else's bad research. Grad school already takes too long, pays too little, and involves too much risk of not finishing. And it doesn't solve the problem of people having to generate copious quantities of research in order to sustain their careers.
> Claimed result: Holding a pen in your teeth (forcing a smile-like expression) makes you rate cartoons as funnier compared to holding a pen with your lips (preventing smiling). More broadly, facial expressions can influence emotional experiences: "fake it till you make it."
I read this about a decade ago, and started, when going into a situation where I wanted to have a natural smile, grimacing maniacally like I had a pencil in my teeth. The thing is, it's just so silly, it always makes me laugh at myself, at which point I have a genuine smile. I always doubted whether the claimed connection was real, but it's been a useful tool anyway.
If you were to meet a "normal" person, would you interpret that as meaning "perpendicular" or as meaning "the kind of person that doesn't look at everything like it's a mathematical expression"?
Normal: "The kind of person who doesn't go out of their way to put-down other people on HN for being nerdy when a linked article contains a weird editorial interjection that bears an unusual resemblance to a math expression."
Disturbing fact: The Stanford prison experiment, run by Philip Zimbardo, wasn't reproducible but that didn't stop Zimbardo from using it to promote his ideologies about the impossibility of rehabilitating criminals, or from becoming the president of the American Psychological Association.
The APA has a really good style guide, but I don't trust them for actual psychology.
A reality TV show tried, because they have more practical ethics requirements than academic institutions, and were able to run the study as long as everyone agreed to be there and there was no sign of abuse: https://en.wikipedia.org/wiki/The_Experiment
I thought you were pointing out some bias by comparing the research to previous research from the same authors. It took me far too long to realize that the experiment was from 100 years ago, and you were pointing out that the names were coincidentally the same.
IIRC the 2013 "racism predicted by telling leading questions" one and ita predecessor, which is listed here but also says there is slight trend toward replication, is just based on implicit association tasks.
So you have a green and red button for good and bad, and then a word pops up and you have less a second to choose which button to press. Oversimplifying complex thought processes in my opinion is junk psychology.
It’s also self-referential because there is no objective measure of “racism,” so how can you even measure whether someone is “more racist” based on reaction time to stereotypical stimuli?
“No objective measure” pretty much sums up the whole field, to be honest. I started on a CS & Psych double major, did about eight psych courses, and then decided it was mostly a joke once I got to the quantitative portions. But those courses were very useful for general life skills. Developmental psychology in particular was packed with dense lessons about how we learn as children… social psych was a good overview of all the “well-known” experiments… etc.
> Claimed result: Adopting expansive body postures for 2 minutes (like standing with hands on hips or arms raised) increases testosterone, decreases cortisol, and makes people feel more powerful and take more risks.
A heuristic I use that is unreasonably good at identifying grifters and charlatans: Unnecessarily invoking cortisol or other hormones when discussing behavioral topics. Influencers, podcasters, and pseudoscience practitioners love to invoke cortisol, testosterone, inflammation, and other generic concepts to make their ideas sound more scientific. Instead of saying "stress levels" they say "cortisol". They also try to suggest that cortisol is bad and you always want it lower, which isn't true.
Dopamine is another favorite of the grifters. Whenever someone starts talking about raising dopamine or doing something to increase dopamine, they're almost always being misleading or just outright lying. Health and fitness podcasters are the worst at this right now.
I have a draft of a blog post on this.
Originally I was going to write about how cortisol isn't always bad, or good, it's just a chemical in us.
But then I started noticing the pattern you point out here where I'm not sure anyone uses the cortisol argument in good faith. Everyone who brings up cortisol is usually trying to sell you something
The most obvious one is the breakdown of trust in scientific research. A frequent discussion I would have with another statistics friend of mine was that that anti-vax crowd really isn't as off base as they are more popularly portrayed and if anything, the "trust the science!" rhetoric is more clearly incorrect.
Science should never be taught as dogmatic, but the reproducibility crisis has ultimately fostered a culture where one should not question "established" results (Kahneman famously proclaimed that one "must" accept the results of the unbelievable priming results in his famous book), especially if that one is interested in a long academic career.
The trouble is that some trust is necessary in communicating scientific observations and hypothesis to the general public. It's easy to blame the failure of the public to unify around Covid as based around cultural divides, but the truth is that skepticism around high stakes, hastily done science is well warranted. The trouble is that even when you can step through the research and see the conclusions are sound, the skepticism remains.
However, as someone that has spent a long career using data to understand the world, I suspect the harm directly caused by the wrong conclusions being reached is more minimal than one would think. This is largely because, despite lip service to "data driven decision making", science and statistics very rarely are the prime driver of any policy decision.
I agree wholeheartedly with your conclusion. Science is relevant for those who care about finding the truth, just because they want to know for sure.
But for most people science doesn't really make much difference in how they choose and operate.
Knowing the truth doesn't mean you are ready to adapt your behavior.
From a political point of view, it may actually be beneficial for a policy to have no scientific basis. What happens when the science gets updated?
You either have to change the policy and admit you were "wrong" to an electorate who can't understand nuance, or continue with the policy and accept a few bad news days before the media cycle resets to something else.
We rack up quite a lot of awfulness with eugenics, phrenology, the "science" that influenced Stalin's disastrous agriculture policies in the early USSR, overpopulation scares leading to China's one-child policy, etc. Although one could argue these were back-justifications for the awfulness that people wanted to do anyway.
Those things were not done by awful people though - they all thought they were serving the public good. We only judge it as awful now because of the results. Nearly of these ideas (Lysenkoism I think was always fringe) were embraced by the educated elites of the time.
Lysenkoism! That's the one. Thank you for reminding me of the name (and for knowing what I was grasping at).
I think some "bad people" used eugenics and phrenology to justify prior hate, but they were also effective tools at convincing otherwise "good people" to join them.
Public policies were made (or justified) based on some of this research. People used this "settled science" to make consequential decisions.
Stereotype threat for example was widely used to explain test score gaps as purely environmental, which contributed to the public seeing gaps as a moral emergency that needed to be fixed, leading to affirmative action policies.
To be honest, whether they had a "study" proving it or not I think those things would have happened anyway.
It's just a question of power in the end. And even if you could question the legitimacy of "studies" the people in power use to justify their ruling, they would produce a dozen more flawed justifications before you could even produce one serious debunking. And they wouldn't even have to give much light to your production so you would need large cultural and political support.
Psychology exists mostly as a new religion; it serves as a tool for justification for people in power, it is used just in the same way as the bible.
It should not be surprising to anyone that much of it isn't replicable (nor falsifiable in the first place) and when it is, the effects are so close to randomness that you can't even be sure of what it means.
This is all by design, you need to keep people confused to rule over them. If they start asking questions you can't answer, you lose authority and legitimacy. Psychology is the tool that serves the dominant ideology that is used to "answer" those questions.
I once did a corporate internal management course that was filled with pseudoscience bullshit. I imagine the impact of that course on the company's productivity was net negative. I'm sure lots of orgs have similar courses.
Learning styles have also been debunked for decades though they continue to be used in education. I saw an amusing line in an article that said 90% of teachers were happy to continue using them even after accepting they're nonsense.
And that's just theories that have been debunked (i.e. proven wrong).
> in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.
The normal distribution predates the general factor model of IQ by hundreds of years.[0]
You can try other distributions yourself, it's going to be hard to find one that better fits the existing IQ data than the normal (bell curve) distribution.
Darwin's cousin, Francis Galton, for whom the log-normal distribution is often called the Galton distribution, was among the first to investigate psychometrics.
not realizing he was hundreds of years late to the game, he still went ahead and coined the term "median"
I took an IQ test as a high school student, and one of the subtests involved placing a stack of shuffled pictures in chronological order. I had one series in the incorrect order, because I had no understanding of the typical behavior of snowfall. The test proctor said almost everyone she tested mixed that one up, because it doesn't snow in the area where I live.
I have no doubt that IQ tests reproducibly measure the test takers ability to pass tests, as well as to perform in a society that the tests are based on.
I think it's disingenuous to attribute IQ to intelligence as a whole though, and it is better understood as an indicator of cultural intelligence.
I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.
> I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.
Raven's Progressive Matrices is often administered. Is that test culturally biased? Does that test measure only ability to take that test and nothing else?
Puzzle tests have their own problems. They're only effective at measuring puzzles solving abilities when they are novel, so retaking the test would lead to higher scores, and practicing even more so. They also only measure puzzle solving abilities which are necessary in some but not all applied intelligence tasks.
A fun irony (every part of this scientific question is gnarly as fuck, which can make it interesting to follow) is that the more culturally biased an IQ test is, the more g-loaded it will turn out to be.
I think humanity majorly underplays how much success is based on culture. I have a long-held theory that offices don't exist to accomplish work, but to establish social relationships, and that work itself is a secondary product of the office community.
My belief was reinforced when companies switched to remote work, and management at many companies complained that it was difficult to tell who was and wasn't working, when the managers didn't get to watch the workers. Abstracting the social relationship from the results of work will make it easier to judge the work itself, but more difficult to enforce the social relationship. When the abstraction occurred, those who were basing the status of their employees on the social relationship, and not the work output, were especially disadvantaged.
I can't quite find the study but there was one mentioned to me about showing the Ravens progressive matrices test to hunter / gatherer tribes, and they did horribly. But those tribes do geometric pattern recognition on the daily basis during hunting, so the tester tried to modify the base shapes to mimic more "realistic" shapes for hunter gatherers (rather than unusual shapes such as perfect triangle, circles and rectangles, hard to find in nature) and the score normalized to median.
I was told this in context of "cultural psychology" how many tests or psychological observations and metrics poorly translate over culture. (especially when you try to pin it on some success metric)
What exactly are they meant to replicate other than other IQ tests? They don't make a statement about anything that is falsifiable, other than that if you give somebody who scores high on a test carefully designed and tested to match the results of previously given IQ tests when given to the same people, they'll tend to match the results that those people will get on other tests that were calibrated in the exact same way.
If you're trying to say they replicate over the lifetime of the same person, I've had a 15 point swing between tests, out of the few I've taken. What did stay constant for me from age 10 to age 40 was my Myers-Briggs test (my dad was a metrics obsessive), and that's obvious horseshit. Consistency doesn't mean you're measuring what you claim to be measuring.
edit: if it matters, scores were between 137 and 152, so exactly an entire standard deviation. That's like the difference between sub-Saharan and European that racists are always crowing about, but in the same person. IQ doesn't even personally replicate for me.
You can prepare for IQ tests, just as you can for any other test, and you can get better at some of the problems in these tests the more you practice them, just as you get better at Sudoku puzzles the more you do them.
Given how long the whole field has been malicious/incompetent failing at basic statistics and sweeping it under the rug, I think that rather then discarding the experiments that don't replicate the better baseline is to discard everything and wait for the future generation of better cognitive psychologists that come up with any good discovery that is widely replicated?
I recently read the lifework book of a nobel prize winning psychologist and it was full of these disproven experiments. As a non-psychologist my trust in the experts is extremely low.
All the "hypothesis" or supposed "results" are so bonkers than it's an insult to intelligence itself that such things can be "proved" with psych "experiment".
Not that it matters, most of the psychology field is inherently bullshit, those are just the example of cases they went so far in the insult to intelligence, no amount of "studies" and rhetoric can save them.
You'd think it's so far in the past that it isn't even considered, but Zimbardo was elected president of the American Psychological Association in 2002, which wasn't all that long ago.
The economics of pyschology are the psychology of economics.
If you won't trust the process, you will gain no real outcome.
What we recieve from the process is not necessarily tangible, but instead a fresh perspective on what may be possible. Thus, the inversion is complete, and we may then move forward.
Papers should not be accepted until an independent lab has replicated the results. It’s pretty simple but people are incentivized to not care if it’s replicable because they need the paper to publish to advance their career
From working in industry and rubbing shoulders with CS people who prioritize writing papers over writing working software I’m sure that in a high fraction of papers people didn’t implement the algorithm they thought they implemented.
Don't get me started, I have seem repos that I'm fairly sure never ran in their presented form.
A guy in our lab thinks authors purposefully mess up their code when publishing on GitHub to make it harder to replicate. I'm starting to come around on his theory.
Well, at least the growth mindset study is not fully debunked yet. It's basically a modern interpretation of what we've known to be true about self-fulfilling prophecies. If you tell children they are can be smart and competent if they work hard, then they will work hard and become smart and competent. This should be a given.
If the "failed replication" was a single study, as in many cases listed here, there is still an open question as to whether the 1) replication study was underpowered (the ones I looked at had pretty small n's), or 2) the re-implementation of the original study was flawed. So I'm not so sure we can quickly label the original studies as "debunked", no more than we can express a high level of confidence in the original studies.
(This isn't a comment on any of the individual studies listed.)
> Most results in the field do actually replicate and are robust [citation needed], so it would be a pity to lose confidence in the whole field just because of a few bad apples.
Is there a good list of results that do consistently replicate?
One thing that confuses me is that some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.
The ego depletion effect seems intuitively surprising to me. Science is often unintuitive. I do know that it is easier to make forward-thinking decisions when I am not tired so I dont know.
>some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.
I don't like Giancotti's claims. He wrote:
>This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false.
I don't agree with Giancotti's epistemological claims but today I will not bloviate at length about the epistemology of science. I will try to be brief.
If I understand Marco Giancotti correctly, one particular point is that Giancotti seems to be saying that Hagger et al. have impressively debunked Baumeister et al.
The ego depletion "debunking" is not really what I would call a refutation. It says, "Results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero. ... Although the current analysis provides robust evidence that questions the strength of the ego-depletion effect and its replicability, it may be premature to reject the ego-depletion effect altogether based on these data alone."
Maybe Baumeister's protocol was fundamentally flawed, but the counter-argument from Hagger et al. does not convince me. I wasn't thrilled with Baumeister's claims when they came out, but now I am somehow even less thrilled with the claims of Hagger et al., and I absolutely don't trust Giancotti's assessment.
I could believe that Hagger executed Baumeister's protocol correctly, but I can't believe Giancotti has a grasp of what scientific claims "should" be "believed."
You make some good points based on your deeper read. I am a bit saddened that the rest of the comment section (the top 6 comments as of right now) devolved into "look at how silly psychology is with all its p-hacking"
That might be true, but this article's comment section isn't a good place for it because it doesn't seem like the article is entirely fair. I would not call it dishonest, but there is a lack of certainty and finality in being able to conclude that these papers have been successfully proven to not be replicable.
The idea isn't that it is easier to do things when not tired. It is that you specifically get tired exercising self control.
I think that can be subtly confused by people thinking you can't get better at self control with practice? That is, I would think a deliberate practice of doing more and more self control every day should build up your ability to do more self control. And it would be easy to think that that means you have a stamina for self control that depletes in the same way that aerobic fitness can work. But, those don't necessarily follow each other.
I thought we knew that these were vehicles by wannabe self-help authors to puff up their status for money. See for example “Grit” and “Deep Work” and other bullshit entries in a breathlessly hyped up genre of pseudoscience.
Note that nearly non of these studies are pure cognitive psychology. Most have intersections with social psychology (and I would deem primarily social psychology) or developmental psychology. For example the debunked study on social priming was published in Journal of Personality and Social Psychology.
This title would be much more accurate if the author omitted “cognitive” from the title.
A key factor behind psychology's low replication rate is the absence of theories that define the field. In most science fields, an initial finding can be compared to theory before publication, which may weed out unlikely results in advance. But psychology doesn't have this option -- no theories, so no Litmus test.
It's important to say that a psychology study can be scientific in one sense -- say, rigorous and disciplined, but at the same time be unscientific, in the sense that it doesn't test a falsifiable, defining psychological theory -- because there aren't any of those.
Or, to put it more simply, scientific fields require falsifiable theories about some aspect of nature, and the mind is not part of nature.
Future neuroscience might fix this, but don't hold your breath for that outcome. I suspect we'll have AGI in artificial brains before we have testable, falsifiable neuroscience theories about our natural brains.
> Claimed result: Listening to Mozart temporarily makes you smarter.
This belongs in a dungeon crawl game. You find an artifact that plays music to you. Depending on the music played (depends on the artifact's enchantment and blessed status), it can buff or debuff your intelligence by several points temporarily.
<On average, african americans are taller than white men and have a higher muscular density.>
Are you comparing direct descendants of Yoruba versus descendants of Celts in America ? or mixed descendants of Bantu and Cherokee versus mixed descendants of Anglo-Saxons and Slavs ? In your study would Barack Obama be a person of color or a person of pallor ?
Or is this data you have gathered observing people at Costco. Just checking on your scientific methodology.
Differences are hidden because (1) differences, even small ones, are used to justify discrimination (2) some feel the need to correct for stereotypes (3) these differences often don't really exist or amount to a small effect size.[0]
In the end, we're talking about distributions of people, and staring at these differences mischaracterizes all but those at the mean.
All that matters is who can pass the test.
[0]: I also encourage you to ask ChatGPT/Grok/Claude "men vs women math performance studies." You'll be shocked to find most studies point to no or small differences.
Quite often those differences exist because of systemic or cultural bias that affects the test design. Tests are often validated based off of other tests that showed a difference, but those tests often had a severe sampling bias that showed a group difference where non-existed. It then became an established theory that if you design a test that measures e.g. “emotional intelligence” (whatever that means) and it didn’t show a group difference, it was invalid and had to be adjusted until it did.
> We know for a fact that sex or ethnicity impacts body yet we seem unable to cope with the idea that there are also differences in how brains work.
Here is your error. You’re assuming that a physical difference in morphology is linked to behavioral or neural correlates. That’s not the case, since observed statistical- or group-level differences need not be driven by biology. You’re assuming biological determinism, and the evidence for direct genetic effects on behavior isn’t there.
> and the evidence for direct genetic effects on behavior isn’t there.
Yes it is. There's an entire field for studying this called Behavioral Genetics.
The easiest evidence comes from comparing monozygotic and dizygotic twins (maternal vs fraternal twins). The variance in behavior is higher among the dizygotic twins who have different genomes.
Twin studies are inherently biased. We have also learned since the 1950s that your genes are not nearly as static as previously assumed. The field of behavior genetics is very fraught indeed, full of disproven assumptions, flawed statistics, and racist pseudo-science. For the longest time behavioral genetics served to justify discrimination through eugenics.
Your parent is correct, the evidence for genetic effects only exist in pseudo-scientific fields using long debunked and flawed methodology. In other words, the “evidence” for behavioral genetics has failed to replicate.
How? And why do you think this completely invalidates their observations?
> We have also learned since the 1950s that your genes are not nearly as static as previously assumed
Aside from random mutations, your genes are essentially static for your lifetime. Genetic expression can change, but you don’t suddenly flip from being blue eyed to brown eyed because your genes change.
We already know that genetics predispose people to certain conditions like schizophrenia. We have ample evidence that behavioral traits are passed down via genes from centuries of animal breeding. How would anyone possibly conclude that genes have no impact on behavior?
I don't know how people like him can come up with so much ideological bullshit that is very obviously proved wrong just by observing other species or consulting history.
If any of it was wrong not only, we would just not breed and select animals for specific traits but pretty much most of our civilisation wouldn't even exist as it does.
We got there precisely by selecting and using animals as tools and food security. Our farm animals are quite passive, precisely because we selected that trait.
There are some people who pay over 30K€ for the breeding of a specific horse in an attempt to create a race winner; and then we have guys like this, supposedly smart but who keep spiting nonsense and even pretend to have the authority of science behind him.
The evidence is right under everyone's nose. It is extremely hard to "prove" in a "scientific" (at some points statistics have too much interpretation behind them to be meaningful) way but anyone who is completely blinded by ideologies.
People who wouldn't be considered "smart" here have an old saying: "the apple doesn't fall far from the tree". They may not be smart but have infinitely more wisdom and what they say might be true more often than the "smart" peoples.
I would argue that behavioral genetics is extremely ideologically driven. This whole sub-field was started by a white supremacist (Francis Galton) with the aim of “proving” the superiority of the white race. The early days were wrought with pseudoscientific bullshit and unlike me complaining about it on a tech forum, the ideology of behavioral geneticist resulted in an actual policy and the horrors of eugenics.
If you want to find more about what makes Behavioral Genetics such a terrible “scientific” endeavor and how the whole field is driven by ideology instead of science, there is a whole book dedicated to the subject: https://en.wikipedia.org/wiki/Misbehaving_Science
It is not about whether or not genes can encode behavior, they obviously can. But explaining behavior with genetics has never actually been done, and most attempts of doing so have been pseudo-scientific race science with the goal of showing the superiority of the white race.
Yes we can select a behavioral trade and breed animals that are more likely to behave in a certain way in a given situation. Those animals are usually also trained from birth to behave in that matter, demonstrating how important environmental interaction is to explaining behavior. Selective breeding is a proof that some behavior is heritable. But heritability does not mean that you can explain behavior with genetics, it just means that some behavior is more likely within some population, regardless of whether or not genes are the reason for the variety. Twin studies, even if they were valid and unbiased, fail to account for that, and are therefor not evidence for the wild claims of behavior genetics.
But it gets worse. Like I said before, twin studies are inherently biased and they don‘t actually show an accurate estimate of heritability as behavior geneticist claim. Twins are not a random sample of the population, twins share the same environment at least until the first minutes after birth (and quite often months or even years after birth), they often interact with each other, and even if they are separated soon after birth, they are more often adopted into similar (often high income) families. In short twin studies suffer from bad statistics resulting from junk-in junk-out.
And it gets even worse, because even looking past the fact that heritability does not offer any evidence for behavior genetics, and the fact that twin studies are fraught with bad statistics, the fundamental assumptions of twin studies are wrong. The human genome is not static throughout the live of the individual, our own genes only account for less then half of our genetic mass (the rest are from microorganisms some are there from birth, others leave and enter our bodies frequently, some we might even trade genes with, and a lot of them affect our behavior).
> From the impressing 80% heritability stated by Sullivan, very little has been pinpointed and confirmed in live models or come even close to the ultimate goal of targeted therapy. [...] Many confounding variables still plague all levels of testing starting from sample sizes, absence of negative studies of insufficient follow-up for high credibility.
It's not an error unless you're able to demonstrate the opposite.
I have yet to see studies that demonstrate that different sexes, hormones or even ethnicities do not impact cognitive abilities or higher proficiency in different fields.
Whereas I've seen plenty that show that women, on average, demonstrate higher cognitive abilities linked to verbal proficiency, text comprehension or executive tasks. Women also tend to have better memory than men.
Facts are that there are genetic differences in how our brains work. And let's not ignore the huge importance of hormones, extremely potent regulators of how we function.
To ignore that we have differences that, at large, help explain statistics is asinine.
And how are you able to rule out that societal or environmental effects are the primary driver? How is your argument not circular, that observed differences are therefore the result of biology?
I see you edited your response after my reply. I’m not denying that you’ve read about those observed differences. I’m trying to say that those differences don’t need to be driven by biology, and evidence suggests otherwise. Behavior can’t be reduced to genetics, and the mechanistic link isn’t there. You are claiming that morphological differences explain the variation. Besides, by your reasoning, you could look at the NBA before Bill Russell and make very different claims.
"on average, women tend to learn languages easier than men": I'm a linguist (although not an expert on second language learning), and I've never heard that. Citation?
I am not the person you're responding to, but is this a surprising and counterintuitive claim? It holds true in my observation (n significantly larger than 32)
> Women have, on average, a higher emotional intelligence which is e.g. tied to higher linguistic proficiency. That helps in many different fields and, on average, women tend to learn languages easier than men.
Has this been experimentally shown to be the case with studies that don‘t fail to replicate?
Between studies that fail to replicate and pure conjecture and pseudo-science I certainly favor the former, at least actual studies that fail to replicate can be disproven, your conjectures are just race/sex science and nothing but pseudo-science. I can either take you at your word, or choose not to believe you. I pick the latter.
"Are Smart People Less Racist? Verbal Ability, Anti-Black Prejudice, and the Principle-Policy Paradox"
Simple study that implies that you should expect white people who think black people are less intelligent than white people to have about 8-11 fewer IQ points than other white people. Just a survey about racial attitudes, and a test of verbal ability that correlates well with IQ tests.
This is a great list for people who want to smugly say "Um, actually" a lot in conversation.
Based on my brief stint doing data work in psychology research, amongst many other problems they are AWFUL at stats. And it isn't a skill issue as much as a cultural one. They teach it wrong and have a "well, everybody else does it" attitude towards p-hacking and other statistical malpractice.
"they are AWFUL at stats."
SF author Michael Flynn was a process control engineer as his day job; he wrote about how designing statistically valid experiments is incredibly difficult, and the potential for fooling yourself is high, even when you really do know what you are doing and you have nearly perfect control over the measurement setup.
And on top of it you're trying to measure the behavior of people not widgets; and people change their behavior based on the context and what they think you're measuring.
There was a lab set up to do "experimental economics" at Caltech back in the late 80's/early 90's. Trouble is, people make different economic decisions when they are working with play money rather than real money.
Expermential Design is one of the big four adacemic subjects within Statistics. The math is complex even before the issues of the effects of the expermential situation.
Oblig link to Norvig's "Warning Signs in Experimental Design": https://www.norvig.com/experiment-design.html
> Trouble is, people make different economic decisions when they are working with play money rather than real money.
Understated even. Ever play poker with just chips and no money behind them? Nobody cares, there is no value to the plastic coins.
I was very surprised at how many statistical methods are taught in undergraduate psychology. Far more statistics than I ever touched in engineering for sure. Yet the undergrads really treated statistics as a cookbook, where they just wanted to be told the recipe and they'd follow it. Honestly they'd have been better off just eyeballing data and collaborating with statisticians for the analysis.
https://en.wikipedia.org/wiki/Stereotype_threat shows up in this list as not replicated, however, it is one of the most studied phenomena in psychology.
>meta-analyses and systematic reviews have shown significant evidence for the effects of stereotype threat, though the phenomenon defies over-simplistic characterization.[22][23][24][25][26][27][28][9]
Failing to reproduce an effect doesn't prove it isn't real. Mythbusters would do this all the time.
On the other hand, some empires are built on publication malpractice.
One of the worst that I know is John Gottman. Marriage counselling based on 'thin slicing'/microexpressions/'Horsemen of the Apocalypse'. His studies had been exposed as fundamentally flawed, and training based on his principles performed worse than prior offerings, before he was further popularized by Malcolm Gladwell in Blink.
This type of intellectual dishonesty underlies both of their careers.
https://en.wikipedia.org/wiki/Cascade_Model_of_Relational_Di...
https://en.wikipedia.org/wiki/The_Seven_Principles_for_Makin...
https://www.gottman.com/blog/this-one-thing-is-the-biggest-p...
The problem with a lot of the “higher free variable” sciences like psychology, ecology and sociology etc
Is they are the ones who need to be at the bleeding edge of statistics but often aren’t
They absolutely need Bayesian competitive hypothesis testing but are often the least likely to use it
> And it isn't a skill issue as much as a cultural one. They teach it wrong
It's definitely a skill issue then
Um, actually I’d say it is the responsibility of all scientists, both professional and amateur, to point out falsehoods when they’re uttered, and not an act of smugness.
[um], has contexts but is usually a cue, that an unexpected, off the average, something is about to be said.
[actually], is a neutral declaration that some cognitive structure was presented, but is at odds with physically observable fact that will now be laid out to you.
As someone who's part of a startup (hrpotentials.com) trying to bring truly scientifically valid psychological testing into HR processes .... yeah. We've been at it for almost 7 years, and we're finally at a point where we can say we have something that actually makes scientific sense - and we're not inventing anything new, just commercializing the science! It only took an electrical engineer (not me) with a strong grasp of statistics working for years with a competent professor of psychology to separate the wheat from the chaff. There's some good science there it's just ... not used much.
How are you going to get around Griggs v. Duke Power Co.? AFAIK, personality tests have not (yet) been given the regulatory eye, but testing cognitive ability has.
Yeah, this is an era which is notorious for pseudoscience.
There’s surely irony here
Reminds me of Feynman’s Cargo Cult Science speech:
https://people.cs.uchicago.edu/~ravenben/cargocult.html
I read in a study that it takes 10,000 hours to become proficient in statistics /s
Approximate replication rates in psychology:
So a list of famous psychology experiments that do replicate may be shorter.https://www.nature.com/articles/nature.2015.18248
I think one would wish the famous ones to be more often replicable.
Nonreplicable publications are cited more than replicable ones (2021)
> We use publicly available data to show that published papers in top psychology, economics, and general interest journals that fail to replicate are cited more than those that replicate. This difference in citation does not change after the publication of the failure to replicate. Only 12% of postreplication citations of nonreplicable findings acknowledge the replication failure.
https://www.science.org/doi/10.1126/sciadv.abd1705
Press release: https://rady.ucsd.edu/why/news/2021/05-21-a-new-replication-...
This is at least partially a failure in publication. Once a paper is published, it's usually left up in the same state forever. If it fails to replicate, that data is published somewhere else. So when someone references the paper, and the diligent reader follows up and reads the reference, it looks convincing, just as it did when first published. It's not reasonable to expect the reader, or even the writer, to be so well versed in all the thousands and thousands of papers published that they know when something has failed to be replicated.
What we need is for every paper to be published alongside a stats card that is kept up to date. How many times it's been cited, how many times people tried to replicate it, and how many times they failed.
This feels like some sort of truth telling paradox, where if you assume the study is true, then seeing a citation like this means it's likely not true.
Isn't the unexpected more famous than the expected?
There may be minute details like having a confident frame of reference for the confidence tests. Cultures, even psychologies might swing certain ideas and their compulsions.
Thanks for providing the reference, that's useful context. Those are awful replication rates, worse than a coin flip. Sounds like the OP can add their own introduction to their list. From the introduction:
> Most results in the field do actually replicate and are robust[citation needed]
The incentive of all psychology researchers is to do new work rather than replications. Because of this, publicly-funded psychology PhDs should be required to perform study replication as part of their training. Protocol + results should be put in a database.
Sure, dump it on the lowest level employee, who has the least training and the most to lose. Punish them for someone else's bad research. Grad school already takes too long, pays too little, and involves too much risk of not finishing. And it doesn't solve the problem of people having to generate copious quantities of research in order to sustain their careers.
Disclosure: Physics PhD.
How interesting would it be if every PhD thesis had to have a "replication" section, where they tried to replicate some famous paper's results.
Little of this is considered cognitive psychology. The vast majority would be viewed as "social psychology"
Setting that aside, among any scientific field I'm aware of, psychology has taken the replication crisis most seriously. Rigor across all areas of psychology is steadily increasing: https://journals.sagepub.com/doi/full/10.1177/25152459251323...
> Smile to Feel Better Effect
> Claimed result: Holding a pen in your teeth (forcing a smile-like expression) makes you rate cartoons as funnier compared to holding a pen with your lips (preventing smiling). More broadly, facial expressions can influence emotional experiences: "fake it till you make it."
I read this about a decade ago, and started, when going into a situation where I wanted to have a natural smile, grimacing maniacally like I had a pencil in my teeth. The thing is, it's just so silly, it always makes me laugh at myself, at which point I have a genuine smile. I always doubted whether the claimed connection was real, but it's been a useful tool anyway.
Yeah, the marshmallow one taught me to have patience and look for the long returns on investments of personal effort.
I think there may be something to a few of these, and more may need considering regarding how these are conducted.
Let’s leave open our credulities for the inquest of time.
> Source: Hagger et (63!) al. 2016
I can't help chuckling at the idea that over 1.98 * 10^87 people were involved in the paper.
If you were to meet a "normal" person, would you interpret that as meaning "perpendicular" or as meaning "the kind of person that doesn't look at everything like it's a mathematical expression"?
Normal: "The kind of person who doesn't go out of their way to put-down other people on HN for being nerdy when a linked article contains a weird editorial interjection that bears an unusual resemblance to a math expression."
Um, actually, the interpretation here is "factorial", not "perpendicular".
Disturbing fact: The Stanford prison experiment, run by Philip Zimbardo, wasn't reproducible but that didn't stop Zimbardo from using it to promote his ideologies about the impossibility of rehabilitating criminals, or from becoming the president of the American Psychological Association.
The APA has a really good style guide, but I don't trust them for actual psychology.
Has anyone tried to reproduce it? Good luck convincing an ethics review board to let you try that again.
Meanwhile, it’s been reproduced “in vitro” in numerous episodes of atrocity, e.g. Abu Ghraib…
A reality TV show tried, because they have more practical ethics requirements than academic institutions, and were able to run the study as long as everyone agreed to be there and there was no sign of abuse: https://en.wikipedia.org/wiki/The_Experiment
Also, how do you italicize text in your comment?
> Also, how do you italicize text in your comment?
HN will italicize any string between a pair of asterisks. [0]
> practical ethics requirements
Practical ethics requirements :)
[0] https://news.ycombinator.com/formatdoc
Yes, the APA certainly has a lot to answer for in their history. The Guantanamo Prison torture scandal is still fresh in my memory.
https://www.democracynow.org/2007/8/20/apa_members_hold_fier...
>Source: Stern, Gerlach, & Penke (2020)
Wow, what are the odds?
https://en.wikipedia.org/wiki/Stern%E2%80%93Gerlach_experime...
I thought you were pointing out some bias by comparing the research to previous research from the same authors. It took me far too long to realize that the experiment was from 100 years ago, and you were pointing out that the names were coincidentally the same.
I'm still amazed that wikipedia doesn't have redirect away from its mobile site
(It's on my list to rewrite those URLs in HN comments at least)
Please, please, please... can you rewrite reddit links to old.reddit.com too? Not that there's much reason to link there, but it makes my eyes bleed.
It depends if the bulk of the HN community supports it. As you probably know, we already do that for submission URLs.
IIRC the 2013 "racism predicted by telling leading questions" one and ita predecessor, which is listed here but also says there is slight trend toward replication, is just based on implicit association tasks. So you have a green and red button for good and bad, and then a word pops up and you have less a second to choose which button to press. Oversimplifying complex thought processes in my opinion is junk psychology.
It’s also self-referential because there is no objective measure of “racism,” so how can you even measure whether someone is “more racist” based on reaction time to stereotypical stimuli?
“No objective measure” pretty much sums up the whole field, to be honest. I started on a CS & Psych double major, did about eight psych courses, and then decided it was mostly a joke once I got to the quantitative portions. But those courses were very useful for general life skills. Developmental psychology in particular was packed with dense lessons about how we learn as children… social psych was a good overview of all the “well-known” experiments… etc.
> Claimed result: Adopting expansive body postures for 2 minutes (like standing with hands on hips or arms raised) increases testosterone, decreases cortisol, and makes people feel more powerful and take more risks.
A heuristic I use that is unreasonably good at identifying grifters and charlatans: Unnecessarily invoking cortisol or other hormones when discussing behavioral topics. Influencers, podcasters, and pseudoscience practitioners love to invoke cortisol, testosterone, inflammation, and other generic concepts to make their ideas sound more scientific. Instead of saying "stress levels" they say "cortisol". They also try to suggest that cortisol is bad and you always want it lower, which isn't true.
Dopamine is another favorite of the grifters. Whenever someone starts talking about raising dopamine or doing something to increase dopamine, they're almost always being misleading or just outright lying. Health and fitness podcasters are the worst at this right now.
I have a draft of a blog post on this. Originally I was going to write about how cortisol isn't always bad, or good, it's just a chemical in us. But then I started noticing the pattern you point out here where I'm not sure anyone uses the cortisol argument in good faith. Everyone who brings up cortisol is usually trying to sell you something
Is anyone tracking how much damage to society bad social science has done? I imagine it's quite a bit.
The most obvious one is the breakdown of trust in scientific research. A frequent discussion I would have with another statistics friend of mine was that that anti-vax crowd really isn't as off base as they are more popularly portrayed and if anything, the "trust the science!" rhetoric is more clearly incorrect.
Science should never be taught as dogmatic, but the reproducibility crisis has ultimately fostered a culture where one should not question "established" results (Kahneman famously proclaimed that one "must" accept the results of the unbelievable priming results in his famous book), especially if that one is interested in a long academic career.
The trouble is that some trust is necessary in communicating scientific observations and hypothesis to the general public. It's easy to blame the failure of the public to unify around Covid as based around cultural divides, but the truth is that skepticism around high stakes, hastily done science is well warranted. The trouble is that even when you can step through the research and see the conclusions are sound, the skepticism remains.
However, as someone that has spent a long career using data to understand the world, I suspect the harm directly caused by the wrong conclusions being reached is more minimal than one would think. This is largely because, despite lip service to "data driven decision making", science and statistics very rarely are the prime driver of any policy decision.
I agree wholeheartedly with your conclusion. Science is relevant for those who care about finding the truth, just because they want to know for sure.
But for most people science doesn't really make much difference in how they choose and operate. Knowing the truth doesn't mean you are ready to adapt your behavior.
I imagine it's comparable to the damage done when policies are set that are not based on studies.
Let's be candid: Most policies have no backing in science whatsoever. The fact that some were backed by poor science is not an indictment of much.
From a political point of view, it may actually be beneficial for a policy to have no scientific basis. What happens when the science gets updated?
You either have to change the policy and admit you were "wrong" to an electorate who can't understand nuance, or continue with the policy and accept a few bad news days before the media cycle resets to something else.
We rack up quite a lot of awfulness with eugenics, phrenology, the "science" that influenced Stalin's disastrous agriculture policies in the early USSR, overpopulation scares leading to China's one-child policy, etc. Although one could argue these were back-justifications for the awfulness that people wanted to do anyway.
Those things were not done by awful people though - they all thought they were serving the public good. We only judge it as awful now because of the results. Nearly of these ideas (Lysenkoism I think was always fringe) were embraced by the educated elites of the time.
You are absolutely right. Another interesting example: The man who invented the lobotomy won a Nobel Prize for it.
Lysenkoism! That's the one. Thank you for reminding me of the name (and for knowing what I was grasping at).
I think some "bad people" used eugenics and phrenology to justify prior hate, but they were also effective tools at convincing otherwise "good people" to join them.
i'm struggling to imagine many negative effects on society caused by the specific papers in this list
Public policies were made (or justified) based on some of this research. People used this "settled science" to make consequential decisions.
Stereotype threat for example was widely used to explain test score gaps as purely environmental, which contributed to the public seeing gaps as a moral emergency that needed to be fixed, leading to affirmative action policies.
To be honest, whether they had a "study" proving it or not I think those things would have happened anyway.
It's just a question of power in the end. And even if you could question the legitimacy of "studies" the people in power use to justify their ruling, they would produce a dozen more flawed justifications before you could even produce one serious debunking. And they wouldn't even have to give much light to your production so you would need large cultural and political support.
Psychology exists mostly as a new religion; it serves as a tool for justification for people in power, it is used just in the same way as the bible.
It should not be surprising to anyone that much of it isn't replicable (nor falsifiable in the first place) and when it is, the effects are so close to randomness that you can't even be sure of what it means. This is all by design, you need to keep people confused to rule over them. If they start asking questions you can't answer, you lose authority and legitimacy. Psychology is the tool that serves the dominant ideology that is used to "answer" those questions.
I once did a corporate internal management course that was filled with pseudoscience bullshit. I imagine the impact of that course on the company's productivity was net negative. I'm sure lots of orgs have similar courses.
Learning styles have also been debunked for decades though they continue to be used in education. I saw an amusing line in an article that said 90% of teachers were happy to continue using them even after accepting they're nonsense.
And that's just theories that have been debunked (i.e. proven wrong).
famous cognitive psychology experiments that do replicate: IQ tests
http://www.psychpage.com/learning/library/intell/mainstream....
in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.
> in fact, the foundational statistical models considered the gold standard for statistics today were developed for this testing.
The normal distribution predates the general factor model of IQ by hundreds of years.[0]
You can try other distributions yourself, it's going to be hard to find one that better fits the existing IQ data than the normal (bell curve) distribution.
[0] https://en.wikipedia.org/wiki/Normal_distribution#History
Darwin's cousin, Francis Galton, for whom the log-normal distribution is often called the Galton distribution, was among the first to investigate psychometrics.
not realizing he was hundreds of years late to the game, he still went ahead and coined the term "median"
more tidbits here https://en.wikipedia.org/wiki/Francis_Galton#Statistical_inn...
I took an IQ test as a high school student, and one of the subtests involved placing a stack of shuffled pictures in chronological order. I had one series in the incorrect order, because I had no understanding of the typical behavior of snowfall. The test proctor said almost everyone she tested mixed that one up, because it doesn't snow in the area where I live.
I have no doubt that IQ tests reproducibly measure the test takers ability to pass tests, as well as to perform in a society that the tests are based on.
I think it's disingenuous to attribute IQ to intelligence as a whole though, and it is better understood as an indicator of cultural intelligence.
I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.
> I would expect that, for cultures who's members score below average on IQ tests from the US, an equivalent IQ test created within that culture would show average members of that culture scoring higher than average members of US culture.
A moment from the show "Good Times" in 1974. https://m.youtube.com/watch?v=DhbsDdMoHC0 at 1:25
Apparently it's referencing a real test, called the BITCH test: https://en.wikipedia.org/wiki/Black_Intelligence_Test_of_Cul...
Also, I forgot how annoying comic relief characters were in sitcoms. They are the opposite of relieving.
Raven's Progressive Matrices is often administered. Is that test culturally biased? Does that test measure only ability to take that test and nothing else?
Puzzle tests have their own problems. They're only effective at measuring puzzles solving abilities when they are novel, so retaking the test would lead to higher scores, and practicing even more so. They also only measure puzzle solving abilities which are necessary in some but not all applied intelligence tasks.
A fun irony (every part of this scientific question is gnarly as fuck, which can make it interesting to follow) is that the more culturally biased an IQ test is, the more g-loaded it will turn out to be.
https://pubmed.ncbi.nlm.nih.gov/24104504/
I think humanity majorly underplays how much success is based on culture. I have a long-held theory that offices don't exist to accomplish work, but to establish social relationships, and that work itself is a secondary product of the office community.
My belief was reinforced when companies switched to remote work, and management at many companies complained that it was difficult to tell who was and wasn't working, when the managers didn't get to watch the workers. Abstracting the social relationship from the results of work will make it easier to judge the work itself, but more difficult to enforce the social relationship. When the abstraction occurred, those who were basing the status of their employees on the social relationship, and not the work output, were especially disadvantaged.
I can't quite find the study but there was one mentioned to me about showing the Ravens progressive matrices test to hunter / gatherer tribes, and they did horribly. But those tribes do geometric pattern recognition on the daily basis during hunting, so the tester tried to modify the base shapes to mimic more "realistic" shapes for hunter gatherers (rather than unusual shapes such as perfect triangle, circles and rectangles, hard to find in nature) and the score normalized to median.
I was told this in context of "cultural psychology" how many tests or psychological observations and metrics poorly translate over culture. (especially when you try to pin it on some success metric)
Yes, it’s almost certainly linked to quality of schooling and exposure to those types of problems, amongst other things, see the Flynn Effect.
https://en.wikipedia.org/wiki/Flynn_effect
What exactly are they meant to replicate other than other IQ tests? They don't make a statement about anything that is falsifiable, other than that if you give somebody who scores high on a test carefully designed and tested to match the results of previously given IQ tests when given to the same people, they'll tend to match the results that those people will get on other tests that were calibrated in the exact same way.
If you're trying to say they replicate over the lifetime of the same person, I've had a 15 point swing between tests, out of the few I've taken. What did stay constant for me from age 10 to age 40 was my Myers-Briggs test (my dad was a metrics obsessive), and that's obvious horseshit. Consistency doesn't mean you're measuring what you claim to be measuring.
edit: if it matters, scores were between 137 and 152, so exactly an entire standard deviation. That's like the difference between sub-Saharan and European that racists are always crowing about, but in the same person. IQ doesn't even personally replicate for me.
You can prepare for IQ tests, just as you can for any other test, and you can get better at some of the problems in these tests the more you practice them, just as you get better at Sudoku puzzles the more you do them.
Related: that brain is plastic and can adapt to challenges in different ways. https://www.scientificamerican.com/article/london-taxi-memor...
Survivorship bias. You can easily make someone's IQ test not replicate. (Hit them on the head really hard.)
Before dunking on psychology for not replicating, remember this is a cross-discipline problem.
In biomedicine, Amgen could reproduce only 6/53 “landmark” preclinical cancer papers and Bayer reported widespread failures.
Given how long the whole field has been malicious/incompetent failing at basic statistics and sweeping it under the rug, I think that rather then discarding the experiments that don't replicate the better baseline is to discard everything and wait for the future generation of better cognitive psychologists that come up with any good discovery that is widely replicated?
I recently read the lifework book of a nobel prize winning psychologist and it was full of these disproven experiments. As a non-psychologist my trust in the experts is extremely low.
All the "hypothesis" or supposed "results" are so bonkers than it's an insult to intelligence itself that such things can be "proved" with psych "experiment".
Not that it matters, most of the psychology field is inherently bullshit, those are just the example of cases they went so far in the insult to intelligence, no amount of "studies" and rhetoric can save them.
No mention of the Stanford Prison Experiment I notice.
You'd think it's so far in the past that it isn't even considered, but Zimbardo was elected president of the American Psychological Association in 2002, which wasn't all that long ago.
And during which time APA was complicit in and participated in torturing prisoners at Guantanamo Bay.
https://www.apa.org/about/policy/chapter-4b
[dead]
The economics of pyschology are the psychology of economics.
If you won't trust the process, you will gain no real outcome.
What we recieve from the process is not necessarily tangible, but instead a fresh perspective on what may be possible. Thus, the inversion is complete, and we may then move forward.
Papers should not be accepted until an independent lab has replicated the results. It’s pretty simple but people are incentivized to not care if it’s replicable because they need the paper to publish to advance their career
In many cases--longitudinal studies are an example, but not the only one--that's not feasible. And it's often expensive--who would pay for it?
Agreed, and the independent lab should be chosen by the publisher, and be kept secret until results are in.
i wonder the replication rate is for ML papers
From working in industry and rubbing shoulders with CS people who prioritize writing papers over writing working software I’m sure that in a high fraction of papers people didn’t implement the algorithm they thought they implemented.
Don't get me started, I have seem repos that I'm fairly sure never ran in their presented form. A guy in our lab thinks authors purposefully mess up their code when publishing on GitHub to make it harder to replicate. I'm starting to come around on his theory.
And most medical studies. It's just as bad as social psych, if not worse, because there's real money at stake in churning out new drugs.
Nowadays everyone publishes their code. There’s typically a project page on github.io, a paper on arxiv.org, and a public repo.
Well, at least the growth mindset study is not fully debunked yet. It's basically a modern interpretation of what we've known to be true about self-fulfilling prophecies. If you tell children they are can be smart and competent if they work hard, then they will work hard and become smart and competent. This should be a given.
Dear lazyweb: is there the opposite of this list anywhere?
If the "failed replication" was a single study, as in many cases listed here, there is still an open question as to whether the 1) replication study was underpowered (the ones I looked at had pretty small n's), or 2) the re-implementation of the original study was flawed. So I'm not so sure we can quickly label the original studies as "debunked", no more than we can express a high level of confidence in the original studies.
(This isn't a comment on any of the individual studies listed.)
> Most results in the field do actually replicate and are robust [citation needed], so it would be a pity to lose confidence in the whole field just because of a few bad apples.
Is there a good list of results that do consistently replicate?
One thing that confuses me is that some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.
The ego depletion effect seems intuitively surprising to me. Science is often unintuitive. I do know that it is easier to make forward-thinking decisions when I am not tired so I dont know.
>some of these papers were successfully replicated, so juxtaposing them to the ones that have not been replicated at all given the title of the page feels a bit off. Not sure if fair.
I don't like Giancotti's claims. He wrote: >This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false.
I don't agree with Giancotti's epistemological claims but today I will not bloviate at length about the epistemology of science. I will try to be brief.
If I understand Marco Giancotti correctly, one particular point is that Giancotti seems to be saying that Hagger et al. have impressively debunked Baumeister et al.
The ego depletion "debunking" is not really what I would call a refutation. It says, "Results from the current multilab registered replication of the ego-depletion effect provide evidence that, if there is any effect, it is close to zero. ... Although the current analysis provides robust evidence that questions the strength of the ego-depletion effect and its replicability, it may be premature to reject the ego-depletion effect altogether based on these data alone."
Maybe Baumeister's protocol was fundamentally flawed, but the counter-argument from Hagger et al. does not convince me. I wasn't thrilled with Baumeister's claims when they came out, but now I am somehow even less thrilled with the claims of Hagger et al., and I absolutely don't trust Giancotti's assessment. I could believe that Hagger executed Baumeister's protocol correctly, but I can't believe Giancotti has a grasp of what scientific claims "should" be "believed."
You make some good points based on your deeper read. I am a bit saddened that the rest of the comment section (the top 6 comments as of right now) devolved into "look at how silly psychology is with all its p-hacking"
That might be true, but this article's comment section isn't a good place for it because it doesn't seem like the article is entirely fair. I would not call it dishonest, but there is a lack of certainty and finality in being able to conclude that these papers have been successfully proven to not be replicable.
The idea isn't that it is easier to do things when not tired. It is that you specifically get tired exercising self control.
I think that can be subtly confused by people thinking you can't get better at self control with practice? That is, I would think a deliberate practice of doing more and more self control every day should build up your ability to do more self control. And it would be easy to think that that means you have a stamina for self control that depletes in the same way that aerobic fitness can work. But, those don't necessarily follow each other.
Now I want to know which cognitive psychology experiments were successfully replicated though.
I thought we knew that these were vehicles by wannabe self-help authors to puff up their status for money. See for example “Grit” and “Deep Work” and other bullshit entries in a breathlessly hyped up genre of pseudoscience.
Note that nearly non of these studies are pure cognitive psychology. Most have intersections with social psychology (and I would deem primarily social psychology) or developmental psychology. For example the debunked study on social priming was published in Journal of Personality and Social Psychology.
This title would be much more accurate if the author omitted “cognitive” from the title.
A key factor behind psychology's low replication rate is the absence of theories that define the field. In most science fields, an initial finding can be compared to theory before publication, which may weed out unlikely results in advance. But psychology doesn't have this option -- no theories, so no Litmus test.
It's important to say that a psychology study can be scientific in one sense -- say, rigorous and disciplined, but at the same time be unscientific, in the sense that it doesn't test a falsifiable, defining psychological theory -- because there aren't any of those.
Or, to put it more simply, scientific fields require falsifiable theories about some aspect of nature, and the mind is not part of nature.
Future neuroscience might fix this, but don't hold your breath for that outcome. I suspect we'll have AGI in artificial brains before we have testable, falsifiable neuroscience theories about our natural brains.
> Claimed result: Listening to Mozart temporarily makes you smarter.
This belongs in a dungeon crawl game. You find an artifact that plays music to you. Depending on the music played (depends on the artifact's enchantment and blessed status), it can buff or debuff your intelligence by several points temporarily.
I'm surprised that dunning Kruger isn't listed.
[flagged]
<On average, african americans are taller than white men and have a higher muscular density.>
Are you comparing direct descendants of Yoruba versus descendants of Celts in America ? or mixed descendants of Bantu and Cherokee versus mixed descendants of Anglo-Saxons and Slavs ? In your study would Barack Obama be a person of color or a person of pallor ?
Or is this data you have gathered observing people at Costco. Just checking on your scientific methodology.
https://pubmed.ncbi.nlm.nih.gov/6735823/
https://www.researchgate.net/publication/12481554_Measures_o...
People at Costco would be smarter than average so not a valid sample.
Differences are hidden because (1) differences, even small ones, are used to justify discrimination (2) some feel the need to correct for stereotypes (3) these differences often don't really exist or amount to a small effect size.[0]
In the end, we're talking about distributions of people, and staring at these differences mischaracterizes all but those at the mean.
All that matters is who can pass the test.
[0]: I also encourage you to ask ChatGPT/Grok/Claude "men vs women math performance studies." You'll be shocked to find most studies point to no or small differences.
[1]: Malcom Gladwell wrote a great piece about his experience as a runner that seems appropriate to share https://www.newyorker.com/magazine/1997/05/19/the-sports-tab...
Quite often those differences exist because of systemic or cultural bias that affects the test design. Tests are often validated based off of other tests that showed a difference, but those tests often had a severe sampling bias that showed a group difference where non-existed. It then became an established theory that if you design a test that measures e.g. “emotional intelligence” (whatever that means) and it didn’t show a group difference, it was invalid and had to be adjusted until it did.
> We know for a fact that sex or ethnicity impacts body yet we seem unable to cope with the idea that there are also differences in how brains work.
Here is your error. You’re assuming that a physical difference in morphology is linked to behavioral or neural correlates. That’s not the case, since observed statistical- or group-level differences need not be driven by biology. You’re assuming biological determinism, and the evidence for direct genetic effects on behavior isn’t there.
> and the evidence for direct genetic effects on behavior isn’t there.
Yes it is. There's an entire field for studying this called Behavioral Genetics.
The easiest evidence comes from comparing monozygotic and dizygotic twins (maternal vs fraternal twins). The variance in behavior is higher among the dizygotic twins who have different genomes.
Twin studies are inherently biased. We have also learned since the 1950s that your genes are not nearly as static as previously assumed. The field of behavior genetics is very fraught indeed, full of disproven assumptions, flawed statistics, and racist pseudo-science. For the longest time behavioral genetics served to justify discrimination through eugenics.
Your parent is correct, the evidence for genetic effects only exist in pseudo-scientific fields using long debunked and flawed methodology. In other words, the “evidence” for behavioral genetics has failed to replicate.
> Twin studies are inherently biased.
How? And why do you think this completely invalidates their observations?
> We have also learned since the 1950s that your genes are not nearly as static as previously assumed
Aside from random mutations, your genes are essentially static for your lifetime. Genetic expression can change, but you don’t suddenly flip from being blue eyed to brown eyed because your genes change.
We already know that genetics predispose people to certain conditions like schizophrenia. We have ample evidence that behavioral traits are passed down via genes from centuries of animal breeding. How would anyone possibly conclude that genes have no impact on behavior?
I don't know how people like him can come up with so much ideological bullshit that is very obviously proved wrong just by observing other species or consulting history.
If any of it was wrong not only, we would just not breed and select animals for specific traits but pretty much most of our civilisation wouldn't even exist as it does.
We got there precisely by selecting and using animals as tools and food security. Our farm animals are quite passive, precisely because we selected that trait.
There are some people who pay over 30K€ for the breeding of a specific horse in an attempt to create a race winner; and then we have guys like this, supposedly smart but who keep spiting nonsense and even pretend to have the authority of science behind him.
The evidence is right under everyone's nose. It is extremely hard to "prove" in a "scientific" (at some points statistics have too much interpretation behind them to be meaningful) way but anyone who is completely blinded by ideologies.
People who wouldn't be considered "smart" here have an old saying: "the apple doesn't fall far from the tree". They may not be smart but have infinitely more wisdom and what they say might be true more often than the "smart" peoples.
I would argue that behavioral genetics is extremely ideologically driven. This whole sub-field was started by a white supremacist (Francis Galton) with the aim of “proving” the superiority of the white race. The early days were wrought with pseudoscientific bullshit and unlike me complaining about it on a tech forum, the ideology of behavioral geneticist resulted in an actual policy and the horrors of eugenics.
If you want to find more about what makes Behavioral Genetics such a terrible “scientific” endeavor and how the whole field is driven by ideology instead of science, there is a whole book dedicated to the subject: https://en.wikipedia.org/wiki/Misbehaving_Science
It is not about whether or not genes can encode behavior, they obviously can. But explaining behavior with genetics has never actually been done, and most attempts of doing so have been pseudo-scientific race science with the goal of showing the superiority of the white race.
Yes we can select a behavioral trade and breed animals that are more likely to behave in a certain way in a given situation. Those animals are usually also trained from birth to behave in that matter, demonstrating how important environmental interaction is to explaining behavior. Selective breeding is a proof that some behavior is heritable. But heritability does not mean that you can explain behavior with genetics, it just means that some behavior is more likely within some population, regardless of whether or not genes are the reason for the variety. Twin studies, even if they were valid and unbiased, fail to account for that, and are therefor not evidence for the wild claims of behavior genetics.
But it gets worse. Like I said before, twin studies are inherently biased and they don‘t actually show an accurate estimate of heritability as behavior geneticist claim. Twins are not a random sample of the population, twins share the same environment at least until the first minutes after birth (and quite often months or even years after birth), they often interact with each other, and even if they are separated soon after birth, they are more often adopted into similar (often high income) families. In short twin studies suffer from bad statistics resulting from junk-in junk-out.
And it gets even worse, because even looking past the fact that heritability does not offer any evidence for behavior genetics, and the fact that twin studies are fraught with bad statistics, the fundamental assumptions of twin studies are wrong. The human genome is not static throughout the live of the individual, our own genes only account for less then half of our genetic mass (the rest are from microorganisms some are there from birth, others leave and enter our bodies frequently, some we might even trade genes with, and a lot of them affect our behavior).
If you are interested, here is a 2014 report debunking twin studies as proof for behavior genetics: https://onlinelibrary.wiley.com/doi/10.1111/1745-9125.12036
EDIT: As for the claim that “[w]e already know that genetics predispose people to certain conditions like schizophrenia.”, that is not so clear either: https://slate.com/technology/2016/02/schizophrenia-genes-fou... and if you want an unopinionated review of the current status of the genetic theory of schizophrenia: this is a nice one: https://pmc.ncbi.nlm.nih.gov/articles/PMC7465115/ and concludes:
> From the impressing 80% heritability stated by Sullivan, very little has been pinpointed and confirmed in live models or come even close to the ultimate goal of targeted therapy. [...] Many confounding variables still plague all levels of testing starting from sample sizes, absence of negative studies of insufficient follow-up for high credibility.
It's not an error unless you're able to demonstrate the opposite.
I have yet to see studies that demonstrate that different sexes, hormones or even ethnicities do not impact cognitive abilities or higher proficiency in different fields.
Whereas I've seen plenty that show that women, on average, demonstrate higher cognitive abilities linked to verbal proficiency, text comprehension or executive tasks. Women also tend to have better memory than men.
Facts are that there are genetic differences in how our brains work. And let's not ignore the huge importance of hormones, extremely potent regulators of how we function.
To ignore that we have differences that, at large, help explain statistics is asinine.
And how are you able to rule out that societal or environmental effects are the primary driver? How is your argument not circular, that observed differences are therefore the result of biology?
I've never stated that biology is the primary driver.
I merely stated that biology, should not be ignored when judging very large samples.
There are cross sex cognitive tests at which women and men tend to perform differently, such as spatial awareness or speed perception and many others.
What's the environmental or cultural factor behind the fact that a female's brain, on average, is able to judge speed much more correctly than a male?
I see you edited your response after my reply. I’m not denying that you’ve read about those observed differences. I’m trying to say that those differences don’t need to be driven by biology, and evidence suggests otherwise. Behavior can’t be reduced to genetics, and the mechanistic link isn’t there. You are claiming that morphological differences explain the variation. Besides, by your reasoning, you could look at the NBA before Bill Russell and make very different claims.
> And how are you able to rule out
It is not possible to rule out unfalsifiable hypotheses.
"on average, women tend to learn languages easier than men": I'm a linguist (although not an expert on second language learning), and I've never heard that. Citation?
I am not the person you're responding to, but is this a surprising and counterintuitive claim? It holds true in my observation (n significantly larger than 32)
All the people I personally know who speak more than 10 languages are men, including me. What now?
Circular reasoning can be used to "prove" anything, so it's not helpful as a basis for policy making.
> Women have, on average, a higher emotional intelligence which is e.g. tied to higher linguistic proficiency. That helps in many different fields and, on average, women tend to learn languages easier than men.
Has this been experimentally shown to be the case with studies that don‘t fail to replicate?
Between studies that fail to replicate and pure conjecture and pseudo-science I certainly favor the former, at least actual studies that fail to replicate can be disproven, your conjectures are just race/sex science and nothing but pseudo-science. I can either take you at your word, or choose not to believe you. I pick the latter.
https://doi.org/10.1093/socpro/spv028
"Are Smart People Less Racist? Verbal Ability, Anti-Black Prejudice, and the Principle-Policy Paradox"
Simple study that implies that you should expect white people who think black people are less intelligent than white people to have about 8-11 fewer IQ points than other white people. Just a survey about racial attitudes, and a test of verbal ability that correlates well with IQ tests.
> white people who think ...
Your link says "high-ability whites are less likely than low-ability whites to report ..."