Updated Nov. 4, 2024 at 7:25 p.m.*
-Analysis-
Imagine this: your weather app promises a sunny fall weekend, so you head out for a hike. But halfway there, a nasty thunderstorm hits, leaving you soaked and shivering. The next week, the forecast looks bright again, so you give it another shot. Yet again, it’s cold and stormy. You barely dodge the rain, but your trip is ruined. Would you trust that forecast a third time?
[shortcode-Eye-On-Us-Box]
Millions of Americans find themselves grappling with a similar dilemma, with the rest of the world watching in suspense. The presidential election is set for November 5th, and until recently, most polls showed Kamala Harris leading against Donald Trump. Now, it’s a nail-biter. If the results in key swing states deviate by even half a percentage point from what the polls predict, the outcomes could be dramatically different. And let’s face it, American pollsters haven’t had the best track record lately.
Let’s rewind a bit: In 2016, the polls placed Hillary Clinton ahead of Donald Trump, yet she lost. Fast forward four years, and while the polls correctly predicted Joe Biden’s victory, they had him winning by a much wider margin than reality reflected — it was a close call.
Actually, the size of the miscalculation in 2020 was even greater than in 2016. In Wisconsin, for example, the polls had Biden 10 points ahead of Trump, but on election day, the margin was a mere half a point.
Not only was the public shocked, but the pollsters themselves began to second-guess their methods. Their professional association, the American Association for Public Opinion Research (AAPOR), set up a task force in 2016 and 2020 to investigate the problem. “We have learned a lot and improved a few things,” says Frauke Kreuter, President of AAPOR. “But we may have created new challenges in the process.”
Never have pollsters had access to the amount of personal data they have today, and yet forecasts have actually gotten worse in recent years. What is happening?
Picking strawberries
To figure this out, we need to dig into how election polls are conducted. The core issue for pollsters is that they can only survey a slice of the population and must make broader assumptions based on that small sample. It’s akin to picking a carton of strawberries: you choose one based solely on what’s visible on top. What if the fruit underneath has gone bad, or isn’t yet ripe?
To avoid such unpleasant surprises, pollsters try their best to build a representative sample. Respondents should be randomly selected, but reflect the entire population. If the sample does not include people under 30 or women without a university degree, additional participants must be recruited from these groups.
Just as there is usually one mushy strawberry at the bottom of the basket, representative samples are never flawless. Polls are always a few percentage points off. This happens with all polls, in every election, everywhere in the world.
But America works in a very peculiar way: Anyone who scores 50.1 percent gets all the electors in a state. Even small mistakes can change the outcome. Although Hillary Clinton received more votes nationwide in 2016, she lost the election because Trump received a razor-thin majority of votes in the decisive states.
Controversial trick
Initially, pollsters theorized about the “Shy Trump Voter”: the idea that Trump’s polarizing nature kept some supporters from openly backing him in public or in surveys.
Today, that theory is largely considered debunked. AAPOR’s findings revealed that polls for elections like those of senators and governors — where Trump wasn’t even a candidate — were even less accurate than those for the presidency.
“I’ve met plenty of Trump voters. None of them were shy,” says Doug Rivers, chief scientist at YouGov and a researcher at Stanford University.
The problem cannot be solved by simply surveying more Trump voters. What percentage would be the right amount?
A key issue in the 2016 polls was the underrepresentation of voters without college degrees—a fixable problem. But how could the 2020 forecasts have missed the mark even more?
The AAPOR task force and other experts came up with an hypothesis that initially echoed the shy voter theory: maybe, many Trump supporters simply refused to participate in polls, possibly out of resentment toward what they viewed as the establishment — including not just the Democratic Party and liberal media but also scientists and pollsters.
Two flaws
But mind that this problem cannot be solved by simply surveying more Trump voters. What percentage would be the right amount? The survey is designed to determine exactly the proportion of Trump supporters among all eligible voters. If the pollsters already knew this number, there would be no need for the whole process.
And that’s why many pollsters resort to a trick. However, this is controversial within the industry itself and could make the polls less reliable, rather than more. The trick works like this: the pollsters won’t only ask people about who they’ll be voting for, but also about who they voted for in the last election. They recruit respondents until their sample reflects the election result of the previous election, and see how, and if, the pool of voters has shifted.
The method has two flaws. First, paradoxical as it may seem, many people do not reliably remember who they voted for four years ago. Psychologists can explain this: people like to be part of the winning team. Sometimes we bend memories accordingly.
A few providers, including YouGov, interview the same people over and over again, for years. This allows them to look into the past themselves, and not have to rely on memories. But this is the exception.
The elusive undecided
The second drawback: this trick doesn’t account for those who didn’t vote last time or who weren’t eligible yet. Yet it’s precisely these groups — young and occasional voters — who often lack a clear political stance that have a decisive influence on the election.
If you want to make a good election forecast, you have to capture the undecided voters as well. The polls tend to target those who are particularly engaged in politics and who are easier to assess. In general, the most difficult thing about election forecasts is not determining which candidate people prefer, but predicting who will actually vote and who will stay home.
“People don’t like to admit that they didn’t vote or they won’t vote,” says Rivers.
“Normal people don’t take part in surveys.”
Andrew Gelman, a statistician and political scientist at Columbia University in New York, suggests that adjusting for previous voting behavior is a good idea. “It may not eliminate the bias completely, but it does improve the polls a bit,” he explains. “Just because a method isn’t perfect doesn’t mean we shouldn’t use it.”
Besides these nuances, pollsters are also debating a much more straightforward question: what’s the best way to conduct a survey? The standard answer for decades has been phone calls. But it’s becoming increasingly rare for people to answer calls from strangers. Nowadays, pollsters have to dial more than 20 numbers to get one conversation. This approach is costly and skews representativeness: those who agree to participate usually have a lot of time on their hands and a strong interest in politics. As Andrew Gelman puts it: “Normal people don’t take part in surveys.”
The alternative is online surveys. In the anonymity of the internet, people usually answer more honestly. But now the pollsters no longer decide who they contact. Instead, they have to wait for someone to click on their button. Large firms pick from these participants, while smaller companies often have to settle for whoever they can get. The quality therefore varies greatly. But four years ago, telephone and online polls proved equally inaccurate.
Given all these difficulties, it would almost be a sensation if the presidential election this time turned out to be what the pollsters predicted. “I’m not very confident that the forecasts will be any more reliable this year,” says Doug Rivers. The AAPOR has already set up a new task force, just in case it starts raining again.
*Originally published Nov. 1, 2024, this article was updated Nov. 4, 2024 with enriched media.
