Is Traditional Polling Underselling Donald Trump’s True Strength?
Nate Cohn | May 17, 2016
Hillary Clinton’s lead in general election polls has faded over the last few weeks. One reason might be that a growing number of Republicans are coalescing around Donald Trump’s candidacy.
Another part of the explanation? Fewer live-interview telephone polls.
So far this year, live-interview telephone surveys have tended to show Mrs. Clinton leading by a slightly larger margin than those conducted online. There hasn’t been a live-interview telephone survey since Mr. Trump beat back all his rivals early this month. So Mrs. Clinton’s seeming lead has seemed to shrink particularly fast.
Why does Mr. Trump do better in online polls? And are the online polls right? The prevailing theory of Mr. Trump’s edge in online surveys is “social desirability bias” — the idea that poll respondents might be less likely to divulge a socially undesirable opinion, like support for a controversial candidate like Mr. Trump, to a live interviewer than in an online questionnaire.
This is not an entirely new phenomenon for Mr. Trump. He fared better in online surveys during the Republican primaries than he did in telephone surveys, a tendency first identified by Jonathan Robinson at Catalist, a data firm associated with the Democratic Party. The pattern lasted all the way through the end of the primary season. The online pollster Morning Consult conducted a study indicating that social desirability bias drove his edge.
Thomas Edsall, writing in the The New York Times, raised the possibility that the same phenomenon explains Mr. Trump’s strength in the online polls right now, suggesting that Mr. Trump could outperform traditional surveys showing a wide lead for Mrs. Clinton.
Social desirability bias might well be helping Mr. Trump in online polls. But it’s a lot less clear now than it was during the primary season. And even if that is what’s going on, there’s no reason to assume the online polls are right.
The Online Polls Were Wrong
With the primary season effectively over, I think we can say, with some qualifications, that the live-interview surveys were probably more accurate than the online surveys.
Why? The actual results just weren’t as good for Mr. Trump as the balance of online surveys predicted they would be.
Mr. Trump has now won 41 percent of the popular vote — including the string of recent contests where he clearly broke through and outperformed his prior vote tallies. On Super Tuesday, he won just 34 percent of the vote — and he won about 39 percent between then and the New York primary.
But the online surveys showed Mr. Trump at 38 percent as early as the new year, broke 40 percent around Super Tuesday, and were in the mid-40s by March 15. His actual vote tallies consistently trailed the online surveys. They look a lot more like what the live-interview surveys showed at the time.
The Upshot published a series of demographic models throughout the primary season that estimated how Mr. Trump might have fared if the country had voted on that date. At the time, our estimates showed Mr. Trump did not approach 40 percent of the vote until March 15.
With the benefit of hindsight, we can do even better: Take all of the results so far, and estimate how Mr. Trump would do in a national primary held at any given stage of the race. These estimates, based on the actual results, consistently match more closely with the live interview surveys than the online ones.
This doesn’t necessarily mean that the social desirability bias theory is wrong. It’s possible that other differences between online and live-interview polls — likely voter models or sampling procedures — accounted for Mr. Trump’s additional strength.
All told, it’s not enough evidence to make me think that the online polls are more accurate about Mr. Trump than the live-interview surveys.
Trump’s Online Survey Edge Is Less Clear Now
Mr. Trump’s advantage in online polls is a lot less clear in the general election than it was in the primary.
It does seem to exist. Mrs. Clinton generally leads Mr. Trump by less in online surveys. On balance, though, Mr. Trump has about 2.5 more points in live-interview polls. He’s actually earning a smaller share of the vote in online surveys than in the live-interview polls — the exact opposite of what one might expect if he were being hurt by social desirability bias in live-interview polls.
So what’s going on? The main difference between the online and live-interview polls is there are vastly more undecided voters in the online surveys. In the live-interview surveys, there are correspondingly more supporters of Mrs. Clinton and Mr. Trump, with Mrs. Clinton apparently gaining slightly more ground.
Why are there so many undecided voters? It’s a methodological quirk, relating to the way that online pollsters have to ask questions.
Consider, for instance, how the typical NBC News/Wall Street Journal poll, a live-interview survey, asked the horse race question in March:
“And, if the election for president were held today, and Donald Trump were the Republican candidate and Hillary Clinton were the Democratic candidate, for whom would you vote?”
Notice that “undecided” or “other” or “don’t know” isn’t a stated option? Live interview pollsters can do that, since the respondent is free to say “other” or “I don’t know,” and the interviewer can record it as such. In the NBC/WSJ poll, 9 percent volunteered “other” or “neither,” and 2 percent weren’t sure.
But this is a lot more challenging for an online pollster. If you put “don’t know” or “other” as an option, a lot of people are going to take it. But if you don’t include it, you have a problem: What will the truly undecided voters do? They could provide an unreliable answer or leave the survey. If you provide them the option to “skip,” they might skip through the whole questionnaire.
In the end, many of the online pollsters give voters the option to choose undecided. Voters are taking it in huge numbers.
You can see the effect in another type of NBC News survey: those conducted online by SurveyMonkey. At the beginning of May, the first NBC News/SurveyMonkey general election poll included an explicit “don’t know” option. Mrs. Clinton led, 43 to 37, with 19 percent selecting “don’t know.”
But in their second survey, conducted after Mr. Trump emerged triumphant, “undecided” was no longer an option. Respondents could go out of their way to skip the question and leave it blank, but they couldn’t simply say “undecided.” The result? Mrs. Clinton led, 49 to 44 percent, with just 7 percent electing to skip the question.
This doesn’t mean there isn’t a social desirability bias against Mr. Trump. One could imagine, for instance, that Mr. Trump would receive notably more support in the online surveys if they had as few undecided voters as the live-interview surveys.
But it does make it harder to compare the results of the online and live-interview surveys, since there’s no way to be sure whether Mr. Trump would still be doing better if there were fewer undecided voters.
You could imagine, for instance, that a bunch of Bernie Sanders’s supporters will take the “don’t know” option, simply to spare themselves from having to support Mrs. Clinton. In the most recent YouGov online survey, 30 percent of Mr. Sanders’s supporters were selecting either “someone else,” “not sure” or “wouldn’t vote.”
But in the recent New York Times/CBS News live-interview survey, just 9 percent wound up in those categories. Mrs. Clinton was the beneficiary: She held 82 percent of Mr. Sanders’s supporters, but just 55 percent in the YouGov poll. (Mr. Trump did fare better among Mr. Sanders’s supporters in the YouGov poll, earning 15 percent of the vote, than in The New York Times/CBS News poll, where he won 10 percent.)
My guess is that Mr. Trump would still fare slightly better in the online surveys than in the live-interview polls, even if the number of undecided voters was equal. Mrs. Clinton tends to lead by a hair more among decided voters, even after removing the “undecided” voters.
But it’s not clear that it’s a social desirability effect, or that it’s more accurate.