I recently saw some polling from Data for Progress on New Hampshire voters’ views on President Biden’s American Jobs Plan (AJP) proposal. The poll was accompanied by a blog post titled “New Hampshire Voters Strongly Support the Key Provisions of the American Jobs Plan,” which argues exactly what it says in the title.
Unfortunately, this polling is pretty bad and probably designed the way it is so that it can be used by progressives as a talking point in favor of the AJP as an alternative to the bipartisan infrastructure bill. But it does provide useful lessons on when to ignore polls.
Yes-men and leading questions
One of the biggest problems with issue polling is the phenomenon of acquiescence bias. According to Wikipedia:
Acquiescence bias, also known as agreement bias,[1] is a category of response bias common to survey research[2] in which respondents have a tendency to select a positive response option[1][3] or indicate a positive connotation disproportionately more frequently.[2][4][5] Respondents do so without considering the content[3] of the question or their 'true' preference…
Research suggests that the proportion of respondents who carry out this behaviour is between 10% and 20%.[2]
In plain English, if you run a poll on to gauge support for vaguely good-sounding ideas such as “Investing in American innovation and manufacturing,” “Modernizing the electricity system by 2035 and making it more reliable,” or “Creating new tax incentives to research and develop new energy technologies,” most people are going to say that they support these things without really having a handle on the specifics.
On the other hand, if you ask voters if they support “The government playing favorites and meddling in the economy,” “Banning fracking,” or “Giving taxpayer money to solar energy companies like Solyndra,” you’re going to get much more negative responses. But these are just two different ways to spin the same policies.
Issue polling questions are often phrased in a manner that leads voters to indicate support for whatever policy is being polled. As you can see, the New Hampshire polling describes the AJP in very positive terms while not mentioning anything negative about the plan; likely a reason for the strong overall support:
The impact of partisanship
When people see a policy like “Cleaning up abandoned mines and oil and gas wells” there may be broad support. But once you tell people that this is a Democratic proposal, support polarizes across party lines. The New Hampshire polling neglects to mention that the AJP is the Democratic White House’s proposal and that Republicans oppose it. This makes sense — the extremely high support for the AJP in the poll would seem to indicate that New Hampshire is a strongly blue state; in reality, the Granite State went for Biden while also re-electing Republican Gov. Chris Sununu in a landslide and handing the GOP a majority in the legislature despite Biden carrying a majority of seats.
The one part of the Data for Progress poll that does implicitly include partisan framings is the question regarding Biden’s approval rating — pretty much everyone knows who Joe Biden is (and that he’s a Democrat). As such, his approval rating is 53-46, very closely tracking his 53-45 victory in the state.
How to spot a shoddy poll
Some good rules of thumb for spotting bad issue polling is as follows:
Lack of partisan framings — this can include failing to mention which party supports a major bill or not including arguments for and against the proposal1.
Read the question phrasing — check to see if the wording seems biased towards a particular answer
Account for acquiescence bias — recall from earlier that support is likely to be overstated by about 10-20% points. For ballot initiative polling, unless the measure is above 60% it’s likely to fail.
Consider the sponsor’s goal — a lot of polling is funded by interest groups that have a stake in making their preferred policies. Think about whether the sponsor of the poll has an incentive to make an issue seem more or less popular2.
This isn’t to say that issue polling is useless — it can be quite handy! But keep an eye out for misleading polling designed to push a particular agenda rather than gauge actual public opinion.
To be fair, Data for Progress is usually pretty good about including pro and con arguments.
For example, a campaign’s internal polls tend to overstate support for their candidate by 4 to 5 points for House races, according to FiveThirtyEight.