In 2011, a psychologist named Joseph P. Simmons and two colleagues set out to use real experimental data to prove an impossible hypothesis. Not merely improbable or surprising, but downright ridiculous. The hypothesis: that listening to The Beatles’ “When I’m Sixty-Four” makes people younger. The method: Recruit a small sample of undergraduates to listen to either The Beatles song or one of two other tracks, then administer a questionnaire asking for a number of random and irrelevant facts and opinions—their parents’ ages, their restaurant preferences, the name of a Canadian football quarterback, and so on. The result: By strategically arranging their data and carefully wording their findings, the psychologists “proved” that randomly selected people who hear “When I’m Sixty-Four” are, in fact, younger than people who don’t....
The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”...
June 27, 2014
"An intellectual crisis in the age of TED talks and Freakonomics."
"Can Social Scientists Save Themselves?"
Subscribe to:
Post Comments (Atom)
Posts
23 comments:
Torture the data, and you can get it to confess to anything.
I like to re-frame the discussion of "p" especially when it applied to political polls by the media in the form, "the margin of error is 4 percent," etc.
I say, "One in every 20 polls like that is off by more than 4 percent. Do you suppose this might be one of them?"
But, the science is settled!
Easy jab taken, this is why scientists are generally of the "don't trust, but verify," variety.
Excellent read. Thanks for the link.
The devil is always in the details.
As I've said many times: Scientists are human beings who are subject to all the failings of being human beings, including the very human trait of putting ones own self interest ahead of the truth. Scientists are not special or above reproof. They've been known to make a few bad assumptions.
Superficial but not totally incorrect summary of the article: "Science is f***** up"."
Better, more reflective summary of the article: "Science has both process issues and human dishonesty to deal with, but there are researchers who are doing their best to add processes that illuminate those problems and set the stage for correction to the base of knowledge created."
Summary totally missing the point of the article: "You can't trust any of these damn scientists."
There will probably be a few more categories of response to this study around the internet, but I'm willing to bet a large number of them will fit into these three.
So glad these people have jobs for life thanks to tenure.
"Outright fakery" is the best new category mentioned in the article.
It's the half fakery guys in Academia and Faux Sciences that have been the problem. Any trial lawyer can tell you that a witness who is telling half truth and half lie is easy to trip up.
It's the total liers whose stories are accepted and assailing them can easily be made to look rude, especially if the total liar fakes a childlike innocent persona.
That method has taken over the field among the under 30 story tellers. George (I did not stalk him to shoot him) Zimmerman used it.
This is kind of what happened to Michael Mann and his hockey stick. It all works fine and dandy until you try to collect new data.
The it's HIDE THE DECLINE because no primary relationship between the tree rings and global temperature existed.
I took a Social Psychology course at U. of Illinois in 1974. It took me about two weeks to realize that the results of these experiments either a) revealed something that was already intuitively obvious, or b) were so limited to the strictures of the experiment that it was of limited or no practical use.
I hated every day in that class.
This is somewhat Straussian-inpsired, but I've found it useful:
In the modern era, psychology is supposed to function as a 'science' and map our interior lives. What it can't accomplish, perhaps psychiatry and neuroscience can. The DSM grows yearly.
Sociology purports to function as a science and explain our actions in groups. Our social life is endlessly fascinating (you can see why many collectivists flock to this).
Political science is a 'science' and will explain something as ancient as human nature itself as rigorously as modern physics explains the natural laws (like the Greeks didn't come first with politics).
Of course, these fields are all different, all with specific claims to knowledge and truth, and can be highly useful in people's lives.
So, Strauss can provide some context to 'modern' thinking, a context with which I also have problems and serious doubt and skepticism (with Strauss, Nietzsche etc), but a context nonetheless.
TED has its merits and very interesting speakers, but some people are clearly trying to make culture and theories of culture out of TED, to explain love, morality, ethics, politics..all the oughts and shoulds and doubts of life that a good liberal arts and education in philosophy can help provide in another context for some people.
This is unsurprising, as well as the fact that over time, the more popular something becomes, the more full of bulls**t.
I've got my doubts, at least.
Bu..bu..but the study was peer reviewed, wasn't it? How could it be wrong?
I am touched by the childlike naivete that too many people place in peer review without knowing anything about the process.
I suspect that if I could site a peer reviewed paper saying that the man in the moon is actually a transexual woman, a lot of deep thinkers would say "Peer reviewed, huh? I guess I'll take it as fact."
In the physical sciences there are something like 3,000 peer reviewed journals. Generally, authors must pay to publish an article in them. In some cases as much as $5,000.
Does anyone really think that a journal is going to use peer reviewers who question an article that will bring $5,000 of revenue?
At the very least this pay to publish model seems to be a conflict of interest on the journal's part.
There are all sorts of schenanigans. Read about Jan Hedrik Schon who published 7 peer reviewed articles in Science Journal. All 7 were made up from whole cloth and eventually withdrawn from the magazine. This was part of the reason his university revoked his doctorate.
We should be very wary of all of these "studies" by scientists. For example that global warming causes impotence in men.
Multiply the situation in the hard sciences by 10 or 20 for the soft or social sciences like this study.
John Henry
Tibore said:
Summary totally missing the point of the article: "You can't trust any of these damn scientists."
I believe that probably 90% of hard scientists are working in good faith. That doesn't make them always right but it does make them honest.
The problem is that we have no way to know which particular scientists are in the 90% and which in the 10%.
Peer review should do that in theory but often doesn't in actuality.
John Henry
Back in the 90s(?) Alan Sokal wrote a famous paper. It was completely made up gibberish but organized in the style of social science writing.
It was published in a famous social sciences journal.
He then came forward and questioned pretty much everything the journal had published. It was an interesting experience.
Even more interesting was that some social scientists refused to believe it was gibberish. They claimed that the paper made important points about something.
This was after Sokal came forward and explained how and why he wrote the article.
John Henry
The one common thread of TED presenters is their love of statist solutions.
"Torture the data, and you can get it to confess to anything."
Waterboard the data and you can get even more. And, didn't they tell us torture never works. Well, this is an example of how well it works.
This is kind of what happened to Michael Mann and his hockey stick. It all works fine and dandy until you try to collect new data.
I see that I am not the only one to have thought of AGW research when reading this article. And, much of the basic stuff fails Bacon's 13th Century reproducibility standard. For one thing, the basic temperature data has been massively massaged (maybe for good reason, since actual data is spotty in both space and time). And, the massaging has been carefully guarded. I saw an article in a British paper last week alleging that the reality is that the 1930s were actually the warmest decade in the last century or so, and that this had been manipulated out of the data being used to support AGW (as was, to a great extent, the Medieval Warming Period).
In regards to Mann though, I remain unconvinced that he intentionally utilized faulty statistics, and rather, at least until proven wrong, assume that he was just not a good statistician. The level of statistics needed to debunk his work is essentially that of a professional statistician, likely someone with the same level of statistical knowledge that Mann has of tree rings or the like. Not the type that a PhD in a distant subject would know, but rather, the type that a PhD in statistics would know. And, interestingly, the further you get from counting tree rings, etc., and into subjects such as astrophysics and statistics, the more the AGW theory is questioned.
As for the tree rings as proxies - the Russians have long claimed cherry picking of data. Which, if the least bit plausible, goes a long way to putting a big question mark over much of the proxy work that is the basis of AGW scholarship - in a similar way that the fake study looked for correlation among all the various permutations of results, discarding the pairing that failed to show such.
The cargo cult sciences are still cargo cults, forty years after Feynman called them out? Quelle surprise.
I would be more willing to accept the idea that Mann has good intentions if he wasn't currently suing people for criticizing him. And falsely describing himself as a Nobel Prize winner while he does it.
Torture the student until he drops out.
The book A Troublesome Inheritance bases its arguments about race and the meaning of race on peer reviewed articles citing new data from genome wide analyses which have been linked with longitudinal social surveys. This is a new technique in the social sciences. It isn't possible to truly assess the validity of the results derived from this new technique this early in the game. But supporters of Wade are denouncing their opponents as PC zombies who won't accept the "science" in "peer-reviewed articles." Yet as the article shows which is under discussion in this blog post, data in the social sciences is always massaged and sometimes tortured. Furthermore, the peer review system isn't good at detecting misuse of statistic principles unless a long period of time has gone by in which the assumptions behind the "data" have been analyzed. So when you hear - as you will - of how new "scientific" techniques utilizing genome wide analysis have discovered that those same old lies about race really weren't lies, don't just soak up the new racism in a credulous way. At least regard the whole of the new theory of race and behavior as unproven until the strengths and the weaknesses of the GWA-Longitudinal studies technique is better understood.
FauxTradGuy:
"George (I did not stalk him to shoot him) Zimmerman used it. "
Oh, please, not this sh*t again. It's so discredited, and thus by extension are you.
Give it up! Stop beclowning yourself! There isn't the slightest evidence that Z's handgun was anywhere other than safely stowed in its holster up until the point M. knocked him down and was beating his head into the pavement. None!!!
John,
"I am touched by the childlike naivete that too many people place in peer review without knowing anything about the process. "
No kidding!
My take on the real meaning of peer review: "Will this embarrass us if we publish it?" That's it!
...don't just soak up the new racism in a credulous way
At least regard the whole of the new theory of race and behavior as unproven until the strengths and the weaknesses of the GWA-Longitudinal studies technique is better understood.
Excellent advice. But what if time reveals the assertions of A Troublesome Inheritance are true? Will that be "new racism", or simply a call for acceptance of greater diversity?
Post a Comment