Both the Prop 8 campaign in 2008 and the subsequent Question 1 campaign in Maine in 2009 teach us a lot about the limitations of polling.

The Basic Problem:

Even smart pollsters find this a tough issue to get right . . .

First, the issue of same-sex marriage is obviously a difficult issue on which to poll compared to other issues, even for methodologically sophisticated pollsters. In California, well-run polls got wildly different results, even when asking identical questions at the same time to similar audiences. For example, PPIC and Field showed No on 8 heading toward a landslide victory in September, at the same time that Lake found a dead heat. Similarly, as Election Day loomed, PPIC and Field showed No on 8 ahead, though by a narrowing margin, at the same time that Binder was finding a dead heat and Lake was correctly predicting a loss. By Election Day, Binder had never once found Yes on 8 at 50% or above, even though Lake had Yes on 8 at 50% or higher consistently throughout most of October and only a shade below 50% (and still well ahead of No on 8) as October closed. Of all the polls, only Lake predicted the Election Day outcome correctly, that No on 8 would suffer a loss. Every other poll taken around that time (not only PPIC, Field, and Binder, but also other private polls shared with the No on 8 campaign) showed No on 8 narrowly ahead.

Methodological choices made by pollsters affect their results. But there is no pattern to their choices that explains the varying poll results. Both Lake and PPIC employed random digit dialing to contact and select the voters they included in their poll; both Binder and Field took the other commonly used approach and used a voter list to contact and select voters they included in their polls. All pollsters make their own assumptions about the likely composition of the portion of the electorate that will actually vote; Lake for instance weighs its results to include key demographic subgroups to reflect those assumptions; others, including Binder Research, do not. Within the polling profession, there is ongoing discussion about the pros and cons of these approaches. What is interesting in Prop 8, however, is that even pollsters who made similar methodological choices did not have their findings converge.

The paper by Prof. Gregory B. Lewis and Dr. Charles W. Gossett “Why Did Californians Pass Proposition 8?” examines PPIC and Field polling data on Prop 8. Lewis and Gossett make the case that wrong-way voting (a phenomenon described in Finding 7 and Appendix K) explains much of the discrepancy between their polls that showed No on 8 consistently ahead and the outcome where Yes on 8 prevailed comfortably. Go to http://papers.ssrn.com/Sol3/Delivery.cfm/SSRN_ID1463485_code101796.pdf?abstractid=1451709&mirid=5
to download the full text of the Lewis and Gossett research. Perhaps part of the reason Lake Research more closely predicted the election result is that it took greater pains to minimize the ways in which wrong-way voting affected its numbers in its Be Clear question.

But the Lake standard horse race question also had, in its own way, difficulty predicting the outcome. It understated support for No on 8, painting a bleaker picture than the one that emerged on Election Day.  After speaking with people from Lake and Binder who were deeply involved in the polling on Prop 8 and reading the brief statements issued by Field and PPIC, there is not yet a clear reason for all of the discrepancies.

. . . yet there are signs of progress

Some polling efforts have fared better at gauging the power of the kids argument immediately after one of these ballot measure votes. Third Way, polling in Maine after the passage of Measure 1, found that 63% of voters in the moveable middle thought it was likely that schools would teach about homosexuality if Measure 1 failed, 74% were concerned about schools teaching homosexuality, 40% thought kids would be more likely to experiment with homosexuality, and 58% were concerned about that possibility.

In addition, although the Lake message battery testing was flawed, as discussed in Finding 5, its benchmark poll found that “gay couples” topped the list of most trusted messengers, with 58% of likely voters saying they had a “great deal” or “some” interest in hearing from them on the issue of same-sex marriage. Runners-up were “parents of gay people” at 49%, “local religious leaders” at 47%, “Sen. Diane Feinstein” at 42%, and “Ellen DeGeneres” at 40% (a total of fourteen messengers were tested, and these were the only ones breaking 40%). Note that No on 8 aired ads with Feinstein and DeGeneres but never tried one with gay couples speaking for themselves. A September 15 memo from Lake Research and Wild Swan Resources, however, concluded that gay couples were not the optimal messenger choice without noting the source of any data driving that conclusion. The memo may reflect data that emerged in focus groups, but its failure to say so means that other factors, including the consultants’ gut sense of things, may have carried the day. If that is true, it means that more positive findings about LGBT spokespeople first have to convince the pollsters before they can influence campaign communications.

When to Poll

How to decide whether or when to poll—and whether and when to believe poll results? I offer ten guidelines that may better guide the LGBT community and our allies in the future as we decided whether and when to poll, as well as whether and when to consider additional methodological options to collect better data on smaller voter subgroups.

I also offer one guideline, at the very end, on when to believe and rely on public polls.

Guideline 1: Accept that polling is prone to error.

Polling on the issue of same-sex marriage ballot measures is difficult and prone to serious error. Use polling, but don’t rely on it by itself to assess the value of a particular message or approach. Don’t assume that hiring a second pollster solves the problem. Polling itself, while valuable, is sometimes an inherent part of the problem.

Guideline 2: Compare apples to apples.

Polling is better at identifying trends that emerge in serial polls conducted by the same firm than in predicting absolute outcomes. As a community, we should discount an isolated poll by any one firm or an isolated finding on any one question.

Guideline 3: Poll on the present, not the future.

Polling is much better at providing a snapshot of the present—what voters believe right now—than it is at prediction. Voters often fail to accurately describe what they will do in the future, or how they might respond to an argument in the future. No wonder that polling, relying on their answers, falls short as well.

Guideline 4: Avoid the hypothetical.

Polling on same-sex marriage is at its most problematic when posing hypothetical arguments and asking whether and how voters would react to them. Polling begins to approximate push polling when a series of questions after a horse race question ask the equivalent of whether the voter would change their mind if a particular thing was true. Leading questions too often get the voter being polled to give in whether or not they would actually respond that way in the real-world election. In addition, polling overestimates the value of messages that require voters to think, reason, analogize, or respond to abstract ideas, eg, “keep government out of all our lives” or “regardless of how you feel about marriage, it is wrong to treat one group of people differently from another.” See Finding 5 for a fuller discussion of the danger that polling overstates the power of de-gayed arguments in particular.

Guideline 5: Acknowledge that polling likely understates the impact of emotional arguments.

On the other hand, polling underestimates the power of emotional messages, such as the Yes on 8 argument about kids. Perhaps the reason for the disparity is an artifact of the polling process itself. Most of these polls, and all of the major polls discussed in detail in this report, are conducted over the telephone. A phone caller, asking a question in a neutral tone of voice, does not evoke the emotional power of the “Princes” ad even if they clearly state the idea behind it. A voter may not be lying when they say the idea behind “Princes” doesn’t affect them, but when the same voter sees it played out emotionally in a TV ad like “Princes,” then they react to it. The fundamental problem could be methodological. A telephone polling conversation may be unable to convey the true gist of an emotionally gripping ad.

Guideline 6: Don’t let polling drive the rest of the research.

For the above reasons, it is often a mistake to use polls to find the winning argument; polling can easily identify the wrong one. Similarly, it is often a mistake to use polls to narrow the ideas and messages that will receive further testing. Here, I am offering advice that is the opposite of standard practice: most campaigns begin their research by polling first precisely with this aim in mind.

No on 8 made choices consistent with this conventional approach. The campaign had Lake Research conduct a benchmark poll in May, and then used focus groups for additional research in June through August. With the Lake poll results understating the power of the kids and schools argument, it is not surprising that the argument received less attention in the focus groups, in some of them being relegated to the last few minutes, according to the moderator’s guide. No on 8 used focus groups more consistently to test other potential Yes on 8 arguments that polled better. At the same time, Lake Research recognized the possibility that its polling was understating the power of the kids argument based on its own experience with past same-sex marriage campaigns. It is beyond the scope of this report to ascertain fully how momentum developed that moved the kids issue as fully to the sidelines as it ended up.

Unsurprisingly, however, in many campaigns (not only No on 8), a self-reinforcing dynamic emerges: when an argument polls poorly, it receives relatively cursory examination in later research, which reduces the likelihood that the power of the argument will become clear.

Further, in the case of the kids and schools argument, it is also likely a bit of a relief for campaign consultants when it polls poorly. It is comforting to find that less emotionally charged arguments are the ones we have to counter, rather than the not-yet-solved argument and fears about kids and schools.

Guideline 7:Don’t base poll questions on wishful thinking.

Don’t poll on messages that would never survive a real-life campaign.  Many pro-LGBT campaigns devote much of their benchmark polling to test messages that could only work if our opposition fails to campaign and if voters live in a bubble where their only understanding of the ballot measure is what we tell them about it. They ask whether the voter would prefer (a) a stock argument that avoids mentioning that LGBT people are the ones most affected by the vote at hand or (b) a specific argument that uses the word “gay” or some other equally clear reference to the reality of the subject matter. Not very surprisingly, voters prefer option (a). It’s like asking voters if they’d prefer chocolate ice cream or chocolate ice cream with added gay. Few pick the latter. “Gay” is stigmatized, perhaps even for many of those who support LGBT equality, and certainly for many voters who are undecided or persuadable on the issue of same-sex marriage.

It is not realistic for our campaigns to proceed as though we control whether or not voters learn that the ballot measure up for a vote is about LGBT people. Whether we like it or not, most voters will learn that from the opposition campaign, reading the paper or watching the news, over a water cooler conversation at work, or over the dinner table at home. Our omission of the information doesn’t guarantee widespread ignorance of the reality of what’s on the ballot. Yet consultants often conduct our campaigns as though that’s true, perhaps out of habits developed in the larger part of their practice where they run campaigns for incumbent election officials who can completely overwhelm their challengers.

The No on 8 ad “Unfair” is a classic example of an attempt to reframe as non-gay the measure on the ballot. It relies on vague but negative adjectives, endorsements from various reputable entities, a dark screen, a bright red “No” on the screen, and ominous background music to convey the undesirability of Prop 8 and to encourage a no vote. All of this is standard operating procedure in ballot measure campaigns of any kind in California seeking a no vote.

But this reframing is most effective at defeating ballot measures on unknown, obscure, or confusing topics, where no opposition group is making counterarguments. It is least effective when the topic is clear, voter interest is high; voter opinions are firmly held; voters feel knowledgeable about the issue itself; and when the opposition is making clear, specific counterarguments. In other words, an ad like “Unfair” is least likely to have an impact in a campaign exactly like Prop 8.

Many of the ads by Maine’s No on 1 campaign are vulnerable to a similar critique. No on 1 faced the same anti-LGBT messaging by Yes on 1 as California experienced with Yes on 8. The Yes campaigns focused on kids and schools and a purported danger to kids. Specifically, the Yes campaigns raised the specter that straight kids would be indoctrinated inappropriately to view being gay as normal. The Yes on 1 ads in Maine repeatedly used the verb “push” in a highly pejorative way, eg, “Vote Yes on Question 1 to prevent homosexual marriage from being pushed on Maine students.” See Appendix E for the full transcripts of all of the Yes on 8 ads, and Appendix R for Yes on 1 ads. But the No on 1 campaign in Maine succumbed to the same temptation to avoid the issue. See Finding 4 for more discussion of avoidance in both the No on 8 and the No on 1 campaigns.

Guideline 8: Focus groups have many of the same deficiencies as polling.

Focus groups may also, for similar reasons, bias pro-LGBT campaigns against seeing the power of the kids argument. Most focus group research is done in groups, as the name implies. Well-run focus group research makes choices to try to maximize opportunity for diverse ideas to emerge in the group; most groups are all women or all men, for instance, to make sure that one or two men don’t dominate group discussion or shut down others’ ideas.

But several dynamics can emerge in focus groups that make it less likely they will give us a true read on an issue like kids and schools. Well-run focus groups keep participants unaware of which side is sponsoring the focus group, because many people want to give the answer desired by the focus group sponsor and please the moderator of the group. But early in many focus groups, it’s obvious which side has sponsored the event and is paying them. It’s evident that we need their help, and are asking for them to figure out how to beat the anti-gay side. Participants may try so hard to please us that we walk away thinking we have found something that works when we haven’t. This is especially true if in the focus group we haven’t fully made the most powerful anti-gay argument that we will later encounter in the campaign.

Another focus group dynamic can disable the group for an almost opposite reason. In many of the No on 8 focus groups, especially those with men, one or two strongly opinionated participants offered anti–same-sex marriage opinions early on. This often surprised the moderator of the group because the participants had been prescreened (by telephone) and identified as undecided on Prop 8. But either the telephone screening suffered from some of the same deficiencies as telephone polling or the screening failed to predict what the participants would articulate once placed in a group of their peers. Either way, the result was that peer pressure made the focus group environment so much less hospitable to different opinions that the groups yielded less insight than hoped for.

One possible remedy to consider in future campaigns is to do focus group research one-on-one, rather than in small groups. At least one attempt with this methodology, in Houston in 2005, was much more successful in uncovering difficulties with pro-LGBT messages and ads than other focus groups were.

Another possible remedy may be empirical experimentation. One example of this is the one-on-one door-to-door canvassing described in Recommendation 3, above. Taking time to talk with individuals who voted Yes on 8 and live in swing neighborhoods may yield insight into messages that move some (by no means all) of those who in 2008 voted against us.

This is not to suggest that one-on-one canvassing will teach us all we need to know. A combination of research approaches deserves consideration, and polls and focus groups are very likely to be useful components of a research plan. The work of Lake Research for No on 8 illustrates both the usefulness of polling (eg, the daily tracking and the Be Clear question to supplement the standard horse race question) and the potential for polling to be misleading, even rigged against discovery of voters’ true views (eg, the failure to identify kids as a potent argument for Yes on 8 and the weakness of the message testing, including spending disproportionate time on de-gayed arguments unlikely to carry the day once the opposition makes its case). Case studies, history, and a wide range of new experiments are likely to be necessary as well. By themselves, neither polling nor focus groups are likely to tell us enough.

Guideline 9: Be picky when you hire.

Choose an intellectually honest pollster, one who will tell you the truth even if you don’t want to hear it. Choose a pollster who is available on a reasonable basis. Choose a pollster who provides analysis both orally and in written memos and reports.

One of the great virtues of Lake Research was its willingness to report pessimistic numbers to the No on 8 campaign, even when more optimistic numbers would have been welcomed. It’s smart to hire a pollster who is willing to tell you what you don’t want to hear, particularly when their work is accurate. The opposite choice—easily available—is comforting, but only until Election Day; then it becomes an excruciating experience when the campaign underperforms.

Yet, both No on 8 pollsters could have provided, and should have been required to provide, written analysis of the results of each poll they conducted. Future campaigns should insist from the first part of the hiring process that any potential pollster agree to provide on a regular basis written analysis, not just tables and raw results.

A brief written analysis should accompany every single poll, including daily tracking polls, otherwise why conduct the poll? Little will be learned from pages and pages of undigested tables and no analysis.

Oral analysis is helpful but evanescent and therefore insufficient. If someone on the decision-making team misses the conference call, they have no access to the pollster’s thinking. If someone wants to look back at a previous poll, they have no written analysis available. If someone joins the campaign later, they are in the dark and it’s a challenge to bring them up to speed.

Written analysis has to include complete sentences. A series of PowerPoint slides does not constitute analysis and will not be helpful enough after the fact. PowerPoint slides alone suffer from the same deficiencies as oral analysis.

Capture the best thinking of the pollster in written form, and it is much more likely that the campaign team—as a team—will think about and use the new information, ask more questions of the pollster, and recognize the need for additional data sooner than they otherwise would.

Bring up all of the above topics in all pollster interviews. At the beginning of the relationship explain what you want and why you want it, and the pollster is much more likely to meet your expectations later.

Guideline 10:Learn from the limitations that affected the data and analysis in this report.

The polling by Lake Research (and the more limited polling done for No on 8 by David Binder Research) made possible some of the most important findings in this report. The No on 8 campaign made a smart decision spending almost $1.2 million on research, most of it for polling. The data provide a mostly complete—but not absolutely complete—opportunity to explore the differences among all the subgroups that play important roles in California elections. It’s worth noting some of the limitations, however, so that future campaigns can consider whether and how to overcome them.

African-American voters: pros and cons of periodic oversampling

For example, African-American voters, while represented proportionally in all of the Lake polls, number under 100 respondents in all but one of the polls, because they represent only 5% to10% of the California likely voter electorate. Any group numbering fewer than 100 completed poll interviews constitutes a sample that is too small for reliable trend analysis from poll to poll; apparent changes in the margin between their support and opposition on Prop 8 are smaller than the (very large) possible margin of error.

Only in the initial Lake benchmark poll in May 2008 did No on 8 spend the money to oversample African-Americans. Future campaigns should consider oversampling African-American voters on a regular or periodic basis to gauge voter movement or lack of movement. That would allow a more rigorous assessment of whether we are gaining support among these voters, and whether the data helps us do better. Oversampling would require the pollster’s callers to complete a minimum of 100 completed respondents per survey; this would permit trend analysis among these voters. The additional cost would amount to $5,000 more for each daily tracking survey. If No on 8 had chosen to do this twice a week for three weeks, the cost would have been an additional $30,000; if No on 8 had done the oversampling throughout the twenty-four-day tracking series from October 5 through 30, it would have incurred an additional $120,000 in survey expenses.

Monolingual voters: pros and cons of translating surveys into Spanish and Asian languages

The Lake and Binder polls for No on 8 were conducted only in English. As a result, only Latino voters relatively comfortable and fluent in English are included in the results. The reason for this decision is that California has very few monolingual Spanish speakers who are also likely voters; therefore in California, Lake Research can provide bilingual interviewing but does not do so routinely. Latinos are 5% to 6% of likely voters on ballot measures in California. Monolingual Spanish speakers are only 7% of all Latino voters; 90% to 95% of California Latino likely voters are comfortable communicating in English, in Lake’s experience. That means that only 2% to 3% of all completed interviews in a typical poll of 600–800 likely voters would require Spanish. This is too small a group from which to generalize results to monolingual Spanish speakers as a separate group. It is also likely that this group would disproportionately include some of the least persuadable older voters. By contrast, in Florida, where monolingual Spanish speakers are a much larger percentage of the electorate, Lake routinely arranges for bilingual interviewing.

On the flip side, the cost of Spanish-language bilingual interviewing is only the cost of translation of the questionnaire, approximately $1,000. Given the low cost, future campaigns might want to explore bilingual polling, and test oversampling among monolingual Latino voters to see if useful insight emerges. 

Similarly, it is likely that only Asian-Pacific islander voters relatively comfortable and fluent in English are included in the Lake polling results. The situation is similar to the one described above regarding Latino voters, except that the cost of multilingual interviewing is much higher, approximately $100 extra per completed interview because an outside firm specializing in this work would need to be brought in. Overall, the additional cost of oversampling API likely voters, including but not limited to those requiring interviews in Asian languages as well as those who could participate in English, would cost roughly the same as oversampling African-American voters noted above.

Cell phone–only households: pros and cons of exclusion

One truth about polling is that cost makes it difficult to avoid excluding some subgroups. One group entirely excluded from No on 8 polling, for instance, was cell phone–only households, and those with no landline whatsoever. Including these doubles the cost of each poll for a variety of reasons: phone company restrictions on access, legal rules governing the calls, and costs associated with the fact that a number of people retain their old cell phone numbers even when they move from one state to another. While in Lake’s estimation this exclusion was not material in 2008, this will no longer be the case by 2016, when cell phone–only households are expected to be much more numerous. In other words, each campaign has to choose which voter subgroups to prioritize in its research, since research dollars are limited.

Yet, even with the limitations, the No on 8 polling made a serious attempt to include a wide variety of groups and subgroups. As a result, the No on 8 polling provides important insight, particularly into some subgroups of voters.

Guideline 11:Don’t rely on public polls.

Read them, enjoy them, think about them—but don’t assume they have got it right. Some do a terrific job on a variety of issues. But many struggle to establish a reliable track record on same-sex marriage. Others make methodological choices can easily produce misleading results.

The first page of the Recommendations section lists three of the reasons that even some of the best public polls may err, but here are just a few of the poor methodological choices to watch out for before taking a public poll too seriously.

·         Including in the poll the opinions of anyone other than likely voters, eg a poll of all adults whether or not they are likely voters. In many elections, the majority of adults don’t vote. Even in the highest turnout elections, a large number of eligible adults don’t vote. Including them in a poll biases the results, quite possibly in our favor.

·         Asking the same-sex marriage question with multiple answer options, eg marriage, civil unions, or neither. This is not the way the question will appear on the ballot. It is impossible to know how to allocate those voters who choose an option in the poll that will not be available to them when they vote.

·         Forcing those polled to fit their answers into a very limited number of categories when an open-ended question would be more appropriate. When asking people “why” they hold a particular view, it is particularly easy for a pollster to inadvertently exclude some of the most important points of view that truly capture poll participants’ truest responses.

·         Failing to put numbers in context, eg indicating that 5% or 10% of poll participants is a “small” number; the question we have to ask is, small compared to what? A “small” number of votes might nevertheless be enough to make the difference between victory and defeat, in which case we will need the “small” number on our side and be misled by a poll that dismisses it.

The list above is not comprehensive. But it came to mind when the Public Religion Research Institute (PRRI) released their July 2010 poll finding that only one in five Californians say Proposition 8 is a “good thing” and claiming majority support for same-sex marriage. The poll suffers from all four of the deficiencies noted above.