It’s poll season. Leading up to November, a new political poll will likely come out every week — if not every day. And as an astute news consumer, you may even notice polls released within days of each other that show significantly different results on which candidate is in the lead for a certain race, and by how much.
So, why is that?
“Pollsters are not predictors, they're not looking into a crystal ball,” said Brandon Rottinghaus, a political science professor who conducts polls at the University of Houston.
Rather, Rottinghaus said pollsters are “just giving us a sense of what the snapshot of the electorate looks like at that moment based upon the factors that they have determined are relevant to their poll.”
That “factors they have determined relevant” part is important to keep in mind, especially when looking at polls micro-targeting specific segments of the population.
“It's never the best thing to look at national polls to figure out how a subgroup is moving because the sample is just not big enough,” Domenico Montanaro, the senior political editor for NPR, told The Texas Newsroom.
Take, for instance, recent polls showing that former President Donald Trump could win up to a quarter of the Black vote, or assumptions being made from polls that young Black men are becoming more conservative.
Pollsters interviewed for this story said Black voters are some of the hardest to gauge, mostly because they aren’t participating in polls in high numbers.
“We have trouble, for instance, getting a sizable percentage of Latino respondents. It's also hard in some communities to get African-American respondents,” Rottinghouse said. “Those are definitely problems when it comes to making inferences about population groups where they may have a very specific opinion.”
One group of Americans that may be easier to poll is “older, whiter people” according to Joshua Blank, research director at UT Austin’s Texas Politics Project, which conducts a variety of polls around politics and trending issues.
Blank said sometimes, when conducting surveys, they’ll get too many respondents in that demographic. Pollsters then have to shift the math around.
“Let's say there were 200 old people in the survey and it should’ve been 100. Well, then, instead let's count each of their responses as .5 – as one-half – of a response. And so that way we get them down to the weight,” Blank said.
“Similarly, with the young African-American men, it might be, ‘Hey, we should have had 30 in the survey but we've only got 15. Well, then instead of counting them each as one, we count them as two,’” he added.
Who are the pollsters polling?
Ideally, people conducting polls would just be able to increase their respondent sample size to get more accurate results.
In the example above, you might have asked yourself why pollsters wouldn’t just go out and talk to 15 more people. Turns out, that is much harder nowadays than it used to be.
Richard Murray is a senior research associate at the Hobby School of Public Affairs at the University of Houston who has been conducting polls since the ‘70s, what he calls “the golden age of surveying.”
Back then, people had landlines and didn’t screen their calls like we do today.
“You could buy a sample of numbers that were randomly generated, whether they were listed or not. That was a pretty accurate sample of the entire population,” he said.
As technology grew, so did the cost to do an accurate political poll.
“Typically, a phone survey now probably is going to cost you $100 per completed interview. If it's done over the Internet, maybe half of that amount,” Murray said. “As in all things in life, cost matters.”
You read that right. The effort to survey a single Texan for a poll today could cost $100.
“If you want to run a really strong poll from a well known agency to facilitate, you're probably going to spend between $40 and $60,000 for a sample of about 1,500 people,” Rottinghaus said.
(In fact, in a recent Think interview, Time Magazine’s Philip Elliot estimated it would cost “close to a million dollars to get a statewide poll in Texas that would give us an accurate count for county by county.”)
That price tag means large, well-funded organizations are usually the ones with the best polls.
Among the top is the The New York Times and Sienna College poll. They are rated No. 1 by FiveThirtyEight, a website focused on opinion poll analysis. They rank pollsters by how close their results are to the outcome of an actual election.
Murray said another reason The New York Times is ranked so highly is because they’ve kept polling traditional.
“The New York Times is still calling people in their homes, but they had to make a hundred calls to get a single interview. And for most of us, that doesn't work. Can't, don't have the money,” Murray said.
All that considered, pollsters interviewed for this story recommend taking polls with a grain of salt.
Next time you see a splashy story on a new poll flashed across the TV screen (or hear about one on the radio), they recommend you pay attention to the amount of respondents, the margin of error, and what organization conducted the poll.
“It comes down to is it easy for you to find out what they did and how they did it? It should be,” Blank said. “And if it's not, I think you immediately have to ask questions about the quality of the work or about the intention behind it.”
