Althouse: "T]he average Senate poll conducted in the final three weeks of this year’s campaign overestimated the Democrat’s performance by 4 percentage points."

November 5, 2014

"T]he average Senate poll conducted in the final three weeks of this year’s campaign overestimated the Democrat’s performance by 4 percentage points."

"The Democrats’ complaints may have been more sophisticated-seeming than the skewed polls' arguments made by Republicans in 2012. But in the end, they were just as wrong. The polls did have a strong bias this year — but it was toward Democrats and not against them...."

Says Nate Silver.

Now, he tells us. Isn't it his job to figure this sort of thing out before the elections?

59 comments:

traditionalguy said...: Prophesying what has already happened is a skill. much like learning to fake sincerity.; 11/5/14, 10:58 AM
Mr. D said...: A lot of conservatives I know aren't particularly fond of being polled. In an era of caller ID and fewer landlines, it's quite likely that pollsters are struggling to find a representative sampling of the electorate. Liberals are far more likely to let their freak flag fly. It's the silent majority thing redux.; 11/5/14, 11:05 AM
tim maguire said...: Democrats' complaints about the polling in 2014 sound exactly like the Republicans' complaints in 2012. So of course the Democrats' complaints are more sophisticated-seeming.; 11/5/14, 11:06 AM
PB said...: that's the problem with what Nate does. he analyzes polls and data but never goes out and samples or verifies for himself. Perhaps he can analyze polls better that others and correlate them to other current and past polls, but at some point you have to collect some data yourself to validate what you get from others.; 11/5/14, 11:06 AM
Brennan said...: Nate Silver has only been accurate when he's leaked insider polls from Team Obama.

I didn't follow Nate's crystal ball this election. Was he making public statements that he had an insider view of the internal polling from Team Obama?; 11/5/14, 11:06 AM
Jaq said...: Since Democrats discovered the awesome power of the media for suppressing Republican votes when the networks called Florida for Gore when the most heavily red section of Florida, the panhandle, had an hour to vote, that is the purpose that polls provide.; 11/5/14, 11:08 AM
Jaq said...: Another problem with what Silver does is that he analyzes trying to find the signal in the noise, and when dealing with human behavior, like polls, where humans can dynamically adjust to information, the noise is the signal.; 11/5/14, 11:09 AM
Henry said...: Silver explains it in the article. Poll bias is normal.

Key takeaway:

Polling aggregators, including FiveThirtyEight, may be contributing to it. A fly-by-night pollster using a dubious methodology can look up the FiveThirtyEight or Upshot or HuffPost Pollster or Real Clear Politics polling consensus and tweak their assumptions so as to match it — but sometimes the polling consensus is wrong.

Silver is like a major institutional investor in the markets. He can no longer make predictions without nudging the predictions.; 11/5/14, 11:09 AM
Steven said...: The trouble is, until election day, the only data you have to compare to polls is other polls. Looks like all the pollsters overcorrected their normalizations based on being surprised in 2012, and got surprised on the backswing. But you can't know that until you get data that says that.; 11/5/14, 11:09 AM
Brennan said...: but at some point you have to collect some data yourself to validate what you get from others.

You really don't have to collect the data yourself to be fairly accurate in your predictions. I just think Nate Silver has had inside information from Team Obama that has elevated his accuracy. He's no different than Michael Mann and his "fix" for the Hockey Stick.

If you get inside information that isn't accessible to everyone than nobody can every really validate your modeling. This is the practice of a mystic.; 11/5/14, 11:10 AM
Jaq said...: Alaska now in the bag.; 11/5/14, 11:11 AM
Rusty said...: Now, he tells us. Isn't it his job to figure this sort of thing out before the elections?

That doesn't serve the narrative.; 11/5/14, 11:15 AM
Birkel said...: Unpossible, Althouse!
The polls needed no unskewing.
Democrats assumed cheating of 2012 proportions.; 11/5/14, 11:18 AM
Mark Nielsen said...: After the 2012 election, conservatives were labeled (not by Silver, as far as I know) as looney and anti-science for having believed there was a democrat skew to the polls. What this year teaches us is that there sometimes *is* a skew to the polls. Elections are about more than public opinion. They're decided by human behavior (turnout) and it's certainly not anti-science to believe that factor is hard to pin down.; 11/5/14, 11:22 AM
Anonymous said...: What Tim Maguire said.; 11/5/14, 11:22 AM
Shanna said...: The Democrats’ complaints may have been more sophisticated-seeming than the skewed polls' arguments made by Republicans in 2012.

Hilarious.; 11/5/14, 11:23 AM
Anonymous said...: Mr. D said...
A lot of conservatives I know aren't particularly fond of being polled. In an era of caller ID and fewer landlines, it's quite likely that pollsters are struggling to find a representative sampling of the electorate. Liberals are far more likely to let their freak flag fly. It's the silent majority thing redux.

and a few Dems lie to shop stewards and pollsters about voting for the Liberal...; 11/5/14, 11:24 AM
Left Bank of the Charles said...: Nate Silver missed Kansas and North Carolina, but did predict the overall Republican victory.

Were the polls skewed, as he suggests, or did the electorate move towards the Republicans at the end?; 11/5/14, 11:25 AM
n.n said...: The pols... I mean, the polls, overshot.; 11/5/14, 11:29 AM
NotWhoIUsedtoBe said...: GIGO, I guess.

To his credit, Silver was honest about possible bias for the last few weeks.

The RCP average was wrong, too, if that matters.; 11/5/14, 11:30 AM
Unknown said...: Exactly my thought. And recall that it was Nate Silver/538 who had the lowest probability (only 60 something %) of the GOP taking the Senate, compared to WashPo and NYT which were much higher (90 something %).; 11/5/14, 11:38 AM
n.n said...: The pols... I mean, the polls, lied.; 11/5/14, 11:39 AM
jr565 said...: Sometimes the pollsters are right. Sometimes they're wrong. I dont' know how accurate Nate was in THIS election, but clearly there were a lot of people who got it wrong.; 11/5/14, 11:41 AM
Jaq said...: RCP is nothing more than an aggregation of hard numbers with a heavy admixture of received wisdom, which is more than enough to screw them up.

It is like averaging all the climate models to try to guess the future because you have no basis to choose one over the other and they all disagree.; 11/5/14, 11:44 AM
Peder said...: No, it's not his job. His job (self created, true) is to simply interpret the polls that are taken. Yes, that means that he can't tell beforehand when there's a systematic bias towards one party or the other. That's why he built in such a wide error margin into his model.
I'm not sure that anyone could have told us that there was this systematic bias before the actual voting happened. That's true when the bias swings to either party. I don't think this is a fair criticism to lay at Silver's feet.; 11/5/14, 11:44 AM
Nonapod said...: Another thing to considered is that people on the right have been much more cautious about getting excited and overestimating anything favorable towards Republicans with regards to polling since what happened in 2012. One result of that could be that there was little pushback from the right when it came to scrutinizing polls that may have overestimated in the Dems favor.; 11/5/14, 11:52 AM
rehajm said...: I don't think this is a fair criticism to lay at Silver's feet.

I agree with this. You strive for accuracy, strive to identify weakness in design, and don't overstate. Accept there are limitations.

They did quite well in terms of direction. Amplitude can be forgiven.; 11/5/14, 11:54 AM
Brando said...: Silver DID tell us--in his writings he always allowed for the polls to be biased towards one side or the other, and noted that in '12 they were biased towards the GOP and in '10 biased towards the Democrats. He builds that into his projections and makes conservative estimates as a result.; 11/5/14, 11:54 AM
mccullough said...: Silver did a good job. After 2012, it was a reasonable effort by pollsters. Off-year elections seem to favor Republican turnout, but that could not be assumed.

Silver is much better at political forecasting than sports forecasting.; 11/5/14, 11:55 AM
Bobby said...: Unknown,

No, yesterday morning, when voting started, Nate/538 had the Republican chances of taking the Senate at just about 75%. He's been calling the Republican takeover to be more likely than not (at varying levels) since he unveiled his 2014 forecasting model, and the probability increased steadily as election day approached.

If you're looking for an analyst to criticize for being "biased" in this election, I'd offer up Sam Wang at PEC, who until mid-September was predicting Democratic-control of the Senate and then dramatically switched to a prediction slightly favoring the Republicans which he maintained until the end (all while frequently and repeatedly pointing out that the edge was almost indstinguishable from flipping a coin). My guess is that Sam's site was overrun with pro-Democratic commenters (many taking refuge from Silver and other analysts who were predicting a GOP victory), and Sam either consciously or subconsciously began to craft his writing in a way that comforted them rather than scaring them. It's kind of a normal human reaction, if one compares it to Schumer predicting a Democratic Senate victory in 2014 or Morris calling Romney over Obama in 2012.

--Bobby; 11/5/14, 12:22 PM
damikesc said...: Exactly my thought. And recall that it was Nate Silver/538 who had the lowest probability (only 60 something %) of the GOP taking the Senate, compared to WashPo and NYT which were much higher (90 something %).

I don't mind the lower percentage because, honestly, the races were less close than expected and people like Tillis were expected to lose. Alaska polling is notoriously terrible, so picking that would've been a challenge. And the polling never had Ernst up by a ton, so those 3 and there is no GOP takeover is my math is right.

His percentage showed it would likely happen, but polls weren't giving enough of a pad for the GOP candidate to make a percentage of certainty be super high.; 11/5/14, 12:28 PM
Real American said...: no. Silver's job is to make Democrats feel better about themselves so they don't give up hope and they still vote.; 11/5/14, 12:39 PM
RecChief said...: maybe nerd nation has reached its apogee; 11/5/14, 12:42 PM
Matt Sablan said...: I said this in another thread: The better Republicans do, the less reliable the polls seem.

I don't know if that's actually correlated, or just my shoddy memory, though.; 11/5/14, 12:43 PM
Jaq said...: Really, how could Silver have predicted that Obama in particular, and liberals in general, were going to go all hateful on average Americans regarding ebola?

Then the nurse in Maine flipping off (figuratively) the American electorate, like the true liberal she was, who could have predicted that?; 11/5/14, 12:47 PM
Wince said...: This comment has been removed by the author.; 11/5/14, 12:50 PM
Wince said...: Previously, Anthony Weiner overestimated his pole performance by 4 inches.; 11/5/14, 12:50 PM
Jaq said...: Seriously though, the most troubling thing about Silver is that he thinks what he is trying to do is possible.

If it were, he could just become a billionaire predicting the stock market, for one thing.; 11/5/14, 1:06 PM
Jaq said...: Kind of a deep slog but an interesting look at how petroleum engineers work to overcome the same kinds of cognitive biases to which Nate Silver is also prey.; 11/5/14, 1:10 PM
n.n said...: tim in vermont:

The stock market is a different problem set, where individual or minority behaviors can have overwhelming influence to determine an outcome. Politics, on the other hand, is processed through clusters, and in America there are two large clusters that exhibit the capacity to reach a majority consensus. When there is sufficient cause, and that consensus fails, we observe crossovers between clusters, and a partisan victory.; 11/5/14, 1:17 PM
n.n said...: When a consensus fails, we observe crossovers or abstention. Either phenomenon will affect the outcome. The key to political forecasts is to assess the correlation between interests on issues in the right order within changing contexts (e.g. memory, circumstance). The last factor is weighted by the scope of change and consequence. The key to predictions is sufficient characterization and representation.; 11/5/14, 1:29 PM
Jaq said...: Umm looking at the first comments I am guessing no.; 11/5/14, 1:30 PM
Jaq said...: n.n. They both come down to making judgments about what intelligent human beings, who have access to much of the same information you do, are going to do with it.

You can break it down as much as you like, but what you can't do is provide anything more than assumptions for most of the parameters in your model.

I still maintain that the Maine nurse and Obama's intransigence on ebola were a huge factor that could in no way be accounted for in advance. Nor could you have models in place to account for any of the other myriad of election turning events in place. It is not humanly possible.

There are limits to the power of reason.; 11/5/14, 1:37 PM
Curious George said...: Mitt Romney ‏@MittRomney 4h4 hours ago
Big tent Republicans win big races. Congratulations.

Brent Bozell ‏@BrentBozell
.@MittRomney Thanks, Mr. President.

Hahahahahaha

Perfect!; 11/5/14, 1:39 PM
Jaq said...: The key to predictions is sufficient characterization and representation.

I think you should be more explicit in step two.; 11/5/14, 1:40 PM
FedkaTheConvict said...: Larry Sabato got it right; he predicted Republicans would pick up 8 Senate seats.
The Crystal Ball’s Final 2014 Picks; 11/5/14, 1:55 PM
Brando said...: "I still maintain that the Maine nurse and Obama's intransigence on ebola were a huge factor that could in no way be accounted for in advance. Nor could you have models in place to account for any of the other myriad of election turning events in place. It is not humanly possible."

I wonder about that Maine nurse--she might have helped LePage a bit. It's one thing to challenge a questionable quarantine order in court, and argue in the public about it. It's quite another to smugly blow it off and defy it laughingly, even while more cases keep popping up in NYC and elsewhere.; 11/5/14, 1:56 PM
Jaq said...: If the Maine nurse affected one person in a hundred, it was huge. I heard my cleaning lady absolutely disgusted with her.; 11/5/14, 2:04 PM
Jaq said...: Nothing in the random creation of the universe dictates that it can be comprehended by any of the creatures who might follow on in the chain of creation.

This is why I am so sure many liberals, to pick on one group among many so afflicted, don't really believe in evolution and the inarguable implications of the fact of it.; 11/5/14, 2:37 PM
Unknown said...: He compared he average o the polls (at some point) with the election results, so it's an after the fact assessment.; 11/5/14, 2:52 PM
n.n said...: tim in vermont:

The point is that in politics, it is not necessary to predict individual, but rather cluster behaviors. This reduces the art of prediction to a manageable set.; 11/5/14, 3:22 PM
n.n said...: It's possible to estimate the boundary behaviors of a chaotic system, or ensemble of chaotic processes, within a limited frame of reference. The accuracy of the estimate or prediction is inversely proportionate to offsets from known sources and sinks (e.g. time and space offsets from a known origin).; 11/5/14, 3:31 PM
Jaq said...: Just don't invest too much money or time into it.; 11/5/14, 4:05 PM
James Pawlak said...: The only polls that count are those at which the People cast ballots.; 11/5/14, 4:12 PM
Jaq said...: It's possible to estimate the boundary behaviors of a chaotic system, or ensemble of chaotic processes, within a limited frame of reference.

Sure, if the rules don't change along the way.; 11/5/14, 4:19 PM
Jaq said...: Naomi Oreskes on climate models for starters.

I am sure you have overcome all of her objections though.; 11/5/14, 4:23 PM
Gabriel said...: Did anyone else notice how Nate Silver defines poll bias as the difference between the polls and the election?

What he's doing is entirely hindsight. He does not have any way to spot bias in a poll before the election.

When he says "the polls were biased" that is entirely content-free. You can't use "bias" to explain why the polls didn't agree with the election if you define bias as the polls not agreeing with the election.

To explain the bias, and spot it beforehand, you'd have to spend a lot of time studying the methodology.; 11/5/14, 9:34 PM
ken in tx said...: I hang up on polls. I have been doing it for the last ten years. I did it yesterday. I think pollsters have too much influence in politics. I don't poll, I vote.; 11/5/14, 10:01 PM
Peder said...: Gabriel, when Silver talks of bias, he's doing so strictly in the statistical sense. Which certainly strikes me as appropriate! He recognizes that sometimes (often it seems) the polls, as a group, are off in once direction or the other. Unfortunately, it's difficult or impossible to tell before you get the actual election results.; 11/6/14, 8:48 AM