With a very successful polling cycle behind them, some pollsters believe that a tactic that gained widespread adoption in 2022 may help them get through the upcoming presidential election. But even supporters of the tactic say it may not be a panacea, particularly if former President Donald J. Trump is back on the ballot.
Pollsters have increasingly weighted polls based on who respondents remember voting for in a previous election, as well as adjusting for standard demographics such as race and age. This tactic has long been used in other countries to improve the accuracy of surveys, but it has become widely used in the United States only in recent years.
“We are all terrified,” said Cameron McPhee, SSRS chief methodologist, CNN polling partner and pollster who pondered some of his polls on the 2022 revoked vote. He added: “We all feel good about the changes we made in 2022. , but I think there is still a big question mark” ahead of 2024.
By weighting the revoked vote, pollsters can more easily correct partisan imbalances in who responds to polls, and in recent years, Democrats have tended to respond to polls at higher rates than Republicans. Perhaps most importantly, weighting the recall vote may specifically increase the influence of Trump supporters, a group that polls had trouble measuring accurately in 2016 and 2020.
Adoption of the tactic by pollsters in the United States is far from universal. Several prominent pollsters achieved accurate results without it, including The New York Times/Siena College, which was named America’s most accurate political pollster by FiveThirtyEight after the 2022 cycle.
Overall, 2022 was one of the most accurate years for polls in recent history, according to a FiveThirtyEight analysis. Many pollsters “probably would have been right in 2022 even without that extra weighting step, because we did,” said Patrick Murray of the Monmouth University Poll.
After 2016, post-election analyzes found that polls had consistently underrepresented less educated voters, who tended to disproportionately support Trump. To get around this, pollsters widely embraced education as an added poll weight, and a cycle of accurate polling in 2018 seemed to signal a return to normalcy.
But in 2020, the polls were more skewed than in any modern election, overrepresenting Democratic support by nearly five percentage points, compared with three percentage points, a more normal amount of error, in 2016.
“I think one of the reasons 2022 was successful, and even 2018 to some degree, was that Trump himself was not on the ballot.” said Mr. Murray. “If history is any guide, we’ll probably see that lack of response in 2024 based on how the Republican nomination goes.”
The 2020 election presented a different challenge: It took place in the midst of the pandemic. Pollsters found that some Americans, stuck at home and alone, were more likely to respond to surveys. While that was initially seen as a blessing, it could have led to even more bias if it meant that uneven enforcement of stay-at-home orders added another source of bias in who answered the phone.
Pondering the revoked vote is not without its concerns.
It has been shown that voters do not remember well who they voted for or even if they voted, they are generally more likely to remember voting for the winner. A study of Canadian voters found that up to a quarter of voters did not remember who they had voted for.
This misrepresentation of past votes can push the polls in different directions depending on who won the most recent election. In 2022, that meant respondents were more likely to say they had supported Joe Biden, and pollsters using the reversal vote would end up giving them less weight, meaning Republican support was bolstered.
But with a previous winner from a different match, the effect would be reversed. An assessment by The Times found that weighting their 2020 polls using the 2016 withdrawn vote would have made them even more biased toward Biden. AND a report of the American Association for Public Opinion Research that examined how the 2020 polls could have been improved found that polls that weighted the revoked vote were no better than those that did not.
Similarly, in 2022, the weighting per retired vote would have made the Times/Siena polls less accurate. As published, without weighting the recall vote, the final polls for the Senate, gubernatorial and House races had an average error of less than two percentage points and zero bias toward Democrats or Republicans. When weighted using the recalled vote for the 2020 election results, the median error would have increased by one percentage point, and overall the polls would have been slightly skewed toward Republicans.
But that could have been a consequence of other decisions The Times makes, including weighting available demographic information in the voter file that isn’t always available to other pollsters.
Other pollsters have found that the method produces significant improvements over typical weighting schemes. SSRS used a variety of weighting methods in 2022, including vote remembered for some of its polls, and also experimented with political identification weighting. Their post-election analysis found that using revoked vote as a weight would have been the most accurate overall approach, increasing average accuracy by more than three percentage points compared to weighting for standard demographics.
“It’s a brute force approach,” said Clifford Young, president of US public affairs at Ipsos. “That is, we don’t really know what it corrects for. Does it fix only for non-ignorables? lack of response? or correct coverage bias? Or maybe a probable voter issue? Maybe all three.
Even so, pollsters are generally optimistic. “What the evidence shows is that it puts us in a much better place in our polls than not using it,” Young said, noting that he believed most pollsters would be weighing past voting in 2024. “I think the evidence so far now it suggests that it does more good than harm.”
He added: “If we use the same weighting and correction methods that we used in 2020 in 2024, we will miss the mark.”
