Why was the polling wrong in NH?

Matt Dabrowski, a pollster and a reader, sent in a letter about what happened with the broken polling in New Hampshire

Just yesterday, some major media polls showed Barack Obama with a 10-point lead over Hillary Clinton. But Clinton won the New Hampshire primary by at least 3%. At the same time, John McCain posted a much stronger lead over Mitt Romney than polling predicated.

The polls were dead wrong tonight. (And so was I. I told many of you that Barack Obama would sail through New Hampshire on his way to the presidency.)

Why? In effect, the pollsters double-counted New Hampshire independents.

How does this work? In order to survey likely voters, pollsters ask a series of "screener" questions. Here is how one public pollster, American Research Group (ARG), asks their screener:

"Would you say that you definitely plan to vote in the 2008 Republican presidential primary, that you might vote in the 2008 Republican presidential primary, or that you will probably not vote in the 2008 Republican presidential primary?"

Typically, pollsters treat each party separately when conducting primary polls. One survey will poll the Republican primary (and all non-Republicans will be screened out), and a separate survey will poll the Democrats (and vice versa). This is the case for two reasons. First, most states don’t have open primaries, so cross-party voting isn’t a concern. Second is cost, doubly true in internal campaign polling. It would be a waste of money for, e.g., Chris Dodd to poll Republican primary voters.

Several well-regarded campaign pollsters will travel to South Carolina tomorrow to find themselves in hot water with their campaign managers.

So let’s think this through. Pollster X conducts his survey in the Republican primary. Independents are allowed into the poll, while Democrats are not. Then Pollster X conducts his survey among Democrats. Independents are allowed into the poll a second time, while Republicans are not.

This is the polling equivalent of being allowed to vote twice. You create a situation where a voter would say this: "Well, if you were asking me about the Republican primary, I’d vote for McCain. But since you’re asking me about the Democratic primary, I’ll vote for Obama." In fact, it creates the bizarre possibility that the same individual New Hampshire voter could be literally polled twice.

It comes down to what we call "sampling error." Since the same Independents were allowed into both surveys, the poll’s sample didn’t look like the actual primary electorate. The 10-point Obama lead was only a paper lead — those were actually McCain voters who were erroneously allowed into the Democratic primary polls.

Should pollsters have known this would happen? I’d argue yes. We knew that both Obama and McCain both had wide support among Independents. At first glance, none of the pollsters realized this and changed their models accordingly.

One proper way to screen would have been something like this:

"Q1. Would you say that you definitely plan to vote in the primary, that you might vote in the primary, or that you will probably not vote in the primary?

Q2. Do you plan to vote in the Republican presidential primary, or the Democratic presidential primary?"

Notice that every likely voter is allowed into the poll, and then pushed into the appropriate primary. No segment of the population has the possibility of entering both primaries.

We desperately need to re-think our turnout models in the future, or we will continue to struggle with open primaries, early voting, etc., etc…