Althouse: "If, in the end, the data do turn out to be fraudulent, does that say anything about social science as a whole?"

May 23, 2015

"If, in the end, the data do turn out to be fraudulent, does that say anything about social science as a whole?"

"On some level, the case would be a statistical fluke. Despite what news headlines would have you believe, outright fraud is incredibly rare; almost no one commits it, and almost no one experiences it firsthand. As a result, innocence is presumed, and the mindset is one of trust.... There’s another issue at play: the nature of belief. As I’ve written before, we are far quicker to believe things that mesh with our view of how life should be. [Columbia polisci prof Donald] Green is a firm supporter of gay marriage, and that may have made him especially pleased about the study.... But, perhaps ironically, it was enthusiasm about the study that led to its exposure. The events of the past few days were the result not of skepticism but of belief. Red flags were raised because David Broockman and Joshua Kalla liked the study and wanted to build on it...."

From "How a Gay-Marriage Study Went Wrong" by Maria Konnikova in The New Yorker.

49 comments:

rhhardin said...: Cognitive scientists have shown that you believe what you want to believe.

This is called the cognitive scientist effect.; 5/23/15, 10:24 AM
SteveR said...: When your science involves shaping public policy and effecting the lives of many who don't have the subject matter expertise to evaluate the data, you should be very careful. Throw in the money and its all a big game. Great answer, now find a study.; 5/23/15, 10:29 AM
madAsHell said...: Who paid for the study?

Are there any repercussions for creating a fraudulent study?; 5/23/15, 10:33 AM
Hagar said...: Social science is a contradiction of terms.; 5/23/15, 10:34 AM
Michael K said...: "Social "science" is a contradiction of terms."

FIFY; 5/23/15, 10:38 AM
m stone said...: Wrong question in my mind.

Does an controlled contact or canvassing technique have the power to sway voters on an issue like this?

People have "opinion leaders" who shape our viewpoints. Some doorknocker may be persuasive but, more than likely your mind was made up by your father or mother or some close thoughtful friend.

Yes, we can change, but it's usually glacial.; 5/23/15, 10:39 AM
Big Mike said...: I'm agreeing with Hagar. Calling them the social sciences doesn't really make them scientific.; 5/23/15, 10:40 AM
Gabriel said...: Politicians and courts will seize on studies to support their agenda that they wish to inflict on the rest of us. To that extent, fraud in the social sciences is a big deal.; 5/23/15, 10:43 AM
Chris N said...: I'll accept her reasoning.

Personally, I harbor deep doubts about the epistemological foundations of many branches of the social sciences, as to why they call themselves sciences, should they even be called such a thing.

As to media reporting of the social sciences, statistics, 'factoids' etc. it's always been a carnival.

Perhaps the criticism should be squared on the particular offender, then more broadly on those who had a hand in it along the chain.

I don't expect those who gather around this particular cause would think their chanting might have anything to do with an 'atmosphere' around the guy, though at the end of the day, he's clearly responsible for his own actions.

Further outwards, if I were at the New Yorker, I might wonder how many other beliefs I held as self-evident were actually true.; 5/23/15, 10:56 AM
kcom said...: "if I were at the New Yorker, I might wonder how many other beliefs I held as self-evident were actually true."

You might. But only if you were just walking through the building and not working there.; 5/23/15, 11:07 AM
Chris N said...: Gabriel,

Well said.

There is an entire political/social/intellectual establishment that uses the social sciences to justify their own beliefs/biases/worldview and to bolster their reputation in the marketplace and keep the chatterers chattering.

Perish the though this happens at The New Yorker.

I picture a production line along the BosWash corridor.

How can you go from an insight in a social scientist's mind to grant money, to research, to a paper, to an intellectual hot topic, to conventional wisdom, to Presidential advisers, to policy, to law...and back again to more grant money?

Why, it's green lawns all the way.

I don't expect a lot more than this given human nature, but hey...; 5/23/15, 11:10 AM
Lewis Wetzel said...: This gives a more technical account of the fraud:

http://retractionwatch.com/2015/05/20/author-retracts-study-of-changing-minds-on-same-sex-marriage-after-colleague-admits-data-were-faked/

Much of social science cannot properly be called science because you cannot reproduce the data. You are studying human behavior, after all, and of course studying human behavior is itself an aspect of human behavior.; 5/23/15, 11:11 AM
JAORE said...: "Despite what news headlines would have you believe, outright fraud is incredibly rare;"

I'm sure there is a thoroughly researched study to support this statement.; 5/23/15, 11:20 AM
Chris N said...: Much like the social sciences, it'd be interesting to see if there are heretics and defectors who leave and become arch-enemy 'neo-conservatives,' those who often marry similar humanist ideals with all sorts of other American institutions like the military.

I tend to think these publications work best when they harbor contrarians, writer's writers, and heretics.

Don't speak out against the liberal pieties!; 5/23/15, 11:23 AM
n.n said...: They manufactured evidence to create leverage. This is not a novel feature of human enterprise nor a unique feature of the [selective] trans-equivalence movement. This is not the first a false narrative was spun by the social complex, including affiliated journolists.

That said, the separation of Church (i.e. organized moral consensus) and State is a fantasy. People are poorly served when they exclusively judge a philosophy by its actual or purported philosopher. Pro-choice is a policy of selective (i.e. unprincipled or opportunistic) exclusion.; 5/23/15, 11:30 AM
Chris N said...: The readers of the Boston Evening Transcript
Sway in the wind like a field of ripe corn.

When evening quickens faintly in the street,
Wakening the appetites of life in some
And to others bringing the Boston Evening Transcript,
I mount the steps and ring the bell, turning
Wearily, as one would turn to nod good-bye to Rochefoucauld,
If the street were time and he at the end of the street,
And I say, "Cousin Harriet, here is the Boston Evening Transcript."; 5/23/15, 11:32 AM
Virgil Hilts said...: Not a good week, month, year or decade for Columbia. I remember back when it was considered kind of a prestigious school.; 5/23/15, 11:54 AM
Ken Mitchell said...: Big Mike said: "I'm agreeing with Hagar. Calling them the social sciences doesn't really make them scientific."

ANY field of study that has the word "science" in its name, isn't one.

Physics, chemistry, biology; these things are sciences.

"social science" and "political science" are not.; 5/23/15, 12:02 PM
John Burgess said...: I guess Ms Konnikova isn't familiar with Retraction Watch.

http://retractionwatch.com/; 5/23/15, 12:02 PM
Ken Mitchell said...: "Despite what news headlines would have you believe, outright fraud is incredibly rare; almost no one commits it, and almost no one experiences it firsthand."

I'm going to call "Bullshit!" on this comment. The first few dozen examples of phony studies with made-up "data" might have been surprising, but it has become all too routine these days. And how many people actually VERIFY these "studies"?; 5/23/15, 12:04 PM
Static Ping said...: Yes, yes, yes, fabrication of results is rare! So said the flim-flam man trying to sell be a bottle of snake oil. My rheumatism will be gone in no time!

On this particular topic, we have already seen (a) a clearly and obviously fraudulent study be praised by people who supposedly are experts and/or are able to function as human beings and (b) the utter condemnation of a a study that resulted in conclusions that violated the prejudices of the majority of social scientists even though the study itself was certainly no worse than many other praised studies. If fraud is more common than admitted I seriously doubt these would be the people to root it out. More snake oil for the suckers. Alas, the suckers includes the majority of social scientists, the media, and the political class. The rubes are well educated these days.

The problem is statistical sampling studies are ALWAYS dubious. ALWAYS. NO EXCEPTIONS. Statistical studies always come with a margin of error and the possibility that the study, even if done properly and in good faith, does not represent the population at all. A single study may be interesting, but it never proves anything. To show a true correlation requires many studies until the confirmation becomes undeniable. And even at that point of undeniability it only shows correlation, not causation. Causation requires more work.

It is so much easier to do a single study, sell it to the rubes in the media who all would have failed every collegiate statistics course if they had been required to take one, and enjoy the results. Real science is too much like work.

At this juncture, social science is mostly useless.; 5/23/15, 12:06 PM
Static Ping said...: And ironically by being useless it has proven so very, very useful to certain people.; 5/23/15, 12:08 PM
chuck said...: A friend of mine, very liberal, dropped out as a sociology grad student because of the dishonesty of her advisor. I expect manipulation of data and surveys is standard practice. Apart from that, sociologists putting together survey questions are so accustomed to their left environment that many questions and options never occur to them, they are effectively blind, and that biases surveys.; 5/23/15, 12:22 PM
Eric the Fruit Bat said...: One of my biased beliefs is that the point of divination is for everybody to do what the priest wants them to do.; 5/23/15, 12:24 PM
Krumhorn said...: On some level, the case would be a statistical fluke. Despite what news headlines would have you believe, outright fraud is incredibly rare; almost no one commits it, and almost no one experiences it firsthand.

Nonsense! The reverse is true. I think many but the most credulous folks assume that the results of a given study have been contrived to achieve a desired outcome.

The whole point of the scientific method is to assume that skeptics will try to disprove the thesis unless they can reproduce the results. Global warming is the perfect example of outright and continuous fraud. And yet the leftie media go to great lengths to conceal the multitude of examples of data manipulation, cherry-picked proxies, the rape of statistical methods and demonstrated coordinated efforts to 'conceal the decline'.

The credulous lefties promote anything that offers a scintilla of legitimacy to their agenda regardless of the colossal damage.

But they mean oh-so well.

- Krumhorn; 5/23/15, 12:32 PM
Fernandinande said...: Not just "Social Science":

Why Most Published [medical] Research Findings Are False; 5/23/15, 12:35 PM
Fernandinande said...: Terry said...
Much of social science cannot properly be called science because you cannot reproduce the data.

One exception is the study of cognition and intelligence. However, the results are generally ignored.

Along those lines, Stephen Jay Gould is well known - but not well enough - for falsifying data and lying about other studies in his silly book "The Mismeasure of Man", yet he is still 'respected' by people who don't know any better and/or prefer the sound of his lies, i.e. the MSM.; 5/23/15, 12:52 PM
jr565 said...: Never let facts get in the way of the agenda.; 5/23/15, 12:55 PM
Sebastian said...: "We know that studies confirming liberal thinking sometimes get a pass"

True, but beyond confirmation bias there's actual fabrication of desired results.

No social science study should be published without independent replication.

I would think many past "results" are derived from analysis of databases that no longer exist or cannot be checked. Anything that depends on data not in the public domain is suspect.; 5/23/15, 1:12 PM
Henry said...: In the social "sciences", outright fraud is not terribly rare at all.; 5/23/15, 2:07 PM
WVFarmLife said...: madAsHell asked, "Who paid for the study?"

And my answer is that I don't know in this case, but in general, almost all scientific research is paid for by our government or other governments. This has been true since at least World War II.

This has enormous implications. The odds are good that in any given subject area only a small group of people are choosing which proposals to fund. If the people choosing share an ideology then they will choose proposals that support their ideology. That's the reality of how human nature works.

And if the only proposals being funded tend towards a certain desired result, then you are going to get that result, regardless of whether "the universe" supports that result or not.

Particularly if people with divergent views and intuitions are being denied jobs and funding.

I'm not really sure how in the long-term science survives this kind of treatment.

This particular case is encouraging because the fraud was detected and the people doing the detecting are probably not going to be punished.; 5/23/15, 2:44 PM
West Texas Intermediate Crude said...: If the science is settled, it's social.; 5/23/15, 3:11 PM
MayBee said...: Social Science studies only falsely report data 2% of the time.; 5/23/15, 3:19 PM
richard mcenroe said...: I'm more inclined to believe Glenn Reynolds on the prevalence of scientific research fraud (widespread) than Ms. Konnikova ("Nonexistent when I want it to be true.")

Virgil Hilts: Not since I watched those idiots marching in the '60's.; 5/23/15, 3:29 PM
richard mcenroe said...: "There is an entire political/social/intellectual establishment that uses the social sciences to justify their own beliefs/biases/worldview and to bolster their reputation in the marketplace and keep the chatterers chattering."

The term you are looking for is "mandarinate"...; 5/23/15, 3:30 PM
clint said...: How do we know that "outright fraud is incredibly rare; almost no one commits it"?

How often are independent third parties running around and checking the primary data on social science studies?

Heck, remember Michael Bellesile's Arming America? Won the Bancroft Prize (a big deal). Supreme Court justices were openly talking about his groundbreaking research and planning to rely on it in their 2nd Amendment decision.; 5/23/15, 3:44 PM
Anonymous said...: The whole point of this article is to convince us the third sentence is true. Which its not.

These surveys and social sciences are all there to influence us. Not to tell us about us. But to shape policy and law.

And since it all works, miraculously, in the favor of progressives and Democrats, every time, and supports their idiotic beliefs (anyone remember Elizabeth Warren and medical bankruptcy?) Of course the New Yorker is going to try and convince us.

This stuff works on the low information voter.; 5/23/15, 3:59 PM
Fabi said...: I gotcher statistical fluke right here!

*grabs crotch*; 5/23/15, 5:44 PM
Dave Schumann said...: Konnikova doesn't appear to be equipped to understand what actually happened here. She has a Ph.D. in psychology but doesn't ever appear to have done actual work in the field; her background before her doctorate was in creative writing, and since then she's been an author in all sorts of popular media.

Good for her, but she just doesn't evince any understanding of how incredibly basic the fraud here was. It was the equivalent of trying to sell a dorm-room Scream poster as an actual Munch. When you read what this guy actually did to fabricate data it's f'ing embarrassing. Click through to the underlying PDF and read their 8-point summary; it's accessible for the layman.

The point being that Konnikova, for reasons I won't speculate on, ignores the fact that even the most amateurish, embarrassing, immediately obvious fraud went through a prestigious publishing process and was widely disseminated and took months to be noticed. The lesson isn't that fraud is rare. The lesson is that someone with half a brain, which this sad-sack grad student doesn't seem to have possessed, could pull off any amount of fraud with no one ever noticing.; 5/23/15, 7:44 PM
Zach said...: I have both written and reviewed articles, (in physics, not the social sciences) so I can comment a little on the reviewing process.

Peer review is not really a fact finding or adversarial process. You pretty much take for granted that the author is accurately reporting the work that they did -- it's not like you have any way of checking, even if they were lying. Most scientific apparatus is home built, and the person who built it is the expert in interpreting their own measurements. Faking results is a career ending mistake, so it's not the kind of accusation you would throw around lightly.

The specialist journals where the bulk of science is reported have, to my knowledge at least, very low levels of fraud. If you look at Retraction Watch's archives for Physical Review Letters

http://retractionwatch.com/category/physical-review-letters/

every single retraction looks like an honest mistake that the authors found themselves.

The really high profile journals like Nature and Science (those are really the only two at that level) are targets for fraud -- the incentives are higher. Looking at the archives for Science, we see a couple of examples of fraud coming up per year

http://retractionwatch.com/category/science-journal-retractions/

Notice how the authors are not retracting the articles in question -- these aren't people finding errors in their own work, these are people covering their butts. Notice also that some of these retractions are still honest mistakes -- the people retracting their article on detecting single proton spins state clearly what the problem was, and why they can't stand by their earlier results, with no questions of honesty attached.

Nature, which is not involved in this study, seems to be on a bad streak. After many years where they had one or two retractions a year, they suddenly had 13 papers retracted in just two years.

http://retractionwatch.com/category/nature-retractions/

That's an article every other issue, for a magazine that doesn't publish many articles. That's starting to get worrisome. Hopefully they can get that under control.

In discussing issues like this, you have to draw a line between honest mistake retractions and misconduct retractions. Anyone can make a mistake, and it's actually a mark in someone's favor if they shoulder some embarrassment to correct the record. But the people in the gay marriage study didn't actually do the research they claimed to do. That's just fraud.; 5/23/15, 8:07 PM
Zach said...: Green says the right thing in the New Yorker article, but the fact remains that there are two authors on this paper, and he's one of them. This, on a paper where the data was straight-up faked. Green didn't even know that LaCour hadn't raised the money he claimed to have raised -- several hundred thousand dollars!

The level of involvement of senior authors in the actual content of published papers is often scandalously low.; 5/23/15, 8:21 PM
Zach said...: The really remarkable paragraph in the article is this

A year later, in the summer of 2013, LaCour told Green that the study they’d discussed back in Ann Arbor was complete: the results suggested that talking with openly gay canvassers could produce a durable attitude shift in favor of gay marriage. “I’m used to studying prejudice, teaching prejudice, thinking about prejudice, and the literature is just suffused with pessimism about any prospect of attitude change,” Green told me. “And here we have a study that shows it has profound effects to have contact with gays.” Green was skeptical, and told LaCour that he needed to replicate the findings by sending out a second wave of Fleischer’s canvassers and surveying a second set of voters. LaCour reported back; it appeared, Green said, that “the magic happened again.” The data looked statistically solid, and the analyses seemed to back up LaCour’s claims. The survey response rates looked abnormally high, but LaCour claimed to have raised hundreds of thousands of dollars in grant money to offer bonuses to people who responded to the survey; it was reasonable to think that the money was enough to account for their continued participation. Green volunteered to help LaCour draft the study. The next year, it was published.

Here you can see the germ of a confidence scheme being played on Green. Green has knowledge and skepticism -- he knows that persuasion is usually very ineffective, and he asks for data to be replicated. But LaCour has answers to every question -- they're fraudulent answers, but that's hard to tell from New York about a study in LA. Ultimately, Green makes the mistake of telling LaCour what it's going to take to overcome his skepticism (another round of interviews), which LaCour provides.

Ultimately, all confidence scams depend on being the sole source of information for the mark. One thing that Green should have done is insist on playing around with the data himself -- if you take out the political content of this study, the really surprising result is the high response rate. That's what attracted the attention of the Berkeley group, and Green picked up on it himself. Ultimately, he accepted a lame, sounds-true argument that paying respondents was causing the high response rate, so he missed the chance to find the fraud himself.

In a perfect world, something like a super high response rate would hold up publication of the paper until everyone in the group was happy that they understood what was going on. Accepting a superficially plausible explanation without looking deeper will ultimately come back to bite you.; 5/23/15, 8:33 PM
Zach said...: Oops -- a retraction of my own. Nature is a weekly magazine, not a monthly. So six retractions in a year is one every eight issues, not one every two.; 5/23/15, 8:38 PM
Lewis Wetzel said...: Zach-
A very helpful series of comments.
I work in the sciences in a support position. I don't do research, I support research. It is shocking that this fraud appeared in a periodical like Science. Science is a flagship publication. It is difficult to believe a grad student wouldn't panic when he heard that his fraudulent research was to be published in Science.
LaCour spent no grant money on the research. I expect he turned down down grant money because it would have meant scrutiny by people he couldn't influence.; 5/24/15, 1:59 AM
Anonymous said...: Yeah, it says that "social science" is a corrupt left-wing hive mind, and that until they get a large number of conservatives and libertarians in there, nothing they produce will have any value.; 5/24/15, 2:17 AM
Aussie Pundit said...: Despite what news headlines would have you believe, outright fraud is incredibly rare

No it's not.
It's not rare and it's not even "incredible". Widespread fraud in a system that encourages fraud is very believable.; 5/24/15, 8:40 AM
Joe said...: The problem is calling it "social science". It's always been "social make-shit-up".; 5/24/15, 1:04 PM
chickelit said...: Another remarkable aspect of the story is how Green is bearing all the blame. The editors of Science are at fault here too--err at least one or two of them. They've allowed partisan politics into their editorial pages and I suppose it was only a matter of time before that contaminated the science portion of Science; 5/24/15, 2:45 PM
Bad Lieutenant said...: Wonder if their homosexuality affected their process.; 5/27/15, 8:29 AM