What Makes the Tea Party So Hard to Survey?

While most everyone has heard of the tea party, few people realize it is not a political party at all. Instead, many sociologists call it a “social movement,” a group of individuals and organizations working to enact or resist social or political change.

Social movements are typically fractured and poorly organized, and the tea party is no exception.

“The tea party is not a unified thing,” said Vanessa Williamson, doctoral candidate at Harvard University’s Department of Government and Social Policy and co-author of The Tea Party and the Remaking of Republican Conservatism, one of the earliest academic publications on the movement.

“It’s actually three things,” Williamson said. “It is grassroots activists, it is elite activists, and it is conservative media. Sometimes they work together, but often they don’t.”

Elite activists include national political organizations, such as the Tea Party Patriots, as well as conservative advocacy groups, such as Americans for Prosperity, HeritageAction and FreedomWorks. While these larger groups provide resources for events and messaging, none of them singularly represent or provide leadership for the entire movement.

The tea party remains a loose confederation of national and local groups, which makes developing statistics about the entire movement somewhat problematic. When assessing the results from any survey about the tea party – or any other subject, for that matter – survey methodologists recommend paying close attention to three things: the definition of the population, sample size, and the probability of being in sample.

Population: Who is a tea partyer?

Good surveys clearly define their populations of interest.

“There is no way to do a sound survey without a proper description of what the population is,” said James Lepkowski, professor in the Department of Biostatistics and director of the Program in Survey Methodology at the University of Michigan.

But the fractured nature of the tea party makes it difficult to define. Are tea party members only those who pay dues and go to regular meetings? Do they include people who simply declare themselves tea partyers? Or do they include anyone who sympathizes with or is supportive of tea party ideals?

Because different surveys define who is in – and out – of the tea party movement in different ways, estimates of the number of tea party members vary considerably, largely due to the screening questions they use.

For example, if you ask likely Republican primary voters if they belong to a local tea party organization, pay dues and attend regular meetings, that group is probably very small.

“Only about 6 to 7 percent of the people I poll say they are members of the tea party,” said David Woodard, professor of Political Science at Clemson University.

But if you ask those same likely Republican primary voters if they agree with the goals of the tea party, the percent can jump to as high as 70 to 80 percent who support it, Woodard said.

Sample size: How many tea partyers?

Survey methodologists also emphasize adequate sample size. This is especially important for groups that represent just a small percentage of the population.

Why? Most polls only include between 500 and 1,500 completed interviews. If you want to estimate the characteristics of dues-paying members of the tea party, 1,000 completed interviews would only include about 50 members, assuming they represent about 5 percent of the population surveyed.

When the sample gets below 100 individuals, it becomes too small to be statistically representative, said Michael Traugott, professor of Communication Studies and Political Science at the University of Michigan. This is also when the problem of subgroup analysis “rears its ugly head.” Something as simple as examining the differences between men and women – which would divide the already small number of interviews in half – results in too few of each to determine if any differences are real.

Given the number of completed interviews in a typical poll, it is not surprising many surveys analyzing the tea party define the movement broadly. A general survey of the adult population would likely include more tea party supporters than members, resulting in a bigger number of cases for analysis, making comparisons among interesting subgroups possible.

Probability: An equal chance for all?

Credible surveys also use probability-based sampling. This means that all members of a defined population will have an equal chance of being selected into a sample.

Generating a random sample is important because it removes the bias researchers might have when choosing survey participants.

“Carefully constructed samples that use randomization make the selection as objective as possible,” Lepkowski said. “That’s one of the major reasons for the randomization.”

An additional advantage of probability-based sampling is that you can generalize your findings to the entire population the sample represents. You can also estimate the size of the error that is due to having drawn a sample rather than interviewing the entire population. So if a well-designed survey finds that 30 percent of its sample of voting age adults identify as Republicans, that is likely the percent for the entire country, given a margin of error.

But what about the results from studies where researchers attend meetings and interview tea party members?

“It’s clearly a nonprobability sample, meaning you can’t generalize it to anything beyond the particular groups you actually talked to,” said David Cantor, associate research professor in the Joint Program in Survey Methodology at the University of Maryland and survey methodologist with Westat, a research firm based outside Washington, D.C.

“You might find valuable insights by talking to those people about various topics,” Cantor said, “but it doesn’t really tell you in any quantitative way what the views of the tea party are in a representative fashion. It’s not clear what it represents.”

These are just some of the challenges facing those who wish to define, study or write about the tea party. Broadly speaking, since this is more of a social movement and less of an affiliation, the challenges of deciding who fits the sample are as broad as the coalition of people claiming to adhere to its tenets.

And while coming up with a representative sample is not impossible, surveying actual dues-paying tea party members – who likely represent a small percentage of the voting population – would require a very large sample, larger still if regional variations were to be considered. Large surveys are costly – to the point of being prohibitive to most research organizations. Hence, the lack of good data on the tea party.

Elizabeth M. Grieco
American University
May 3, 2014