Are French pollsters cheating? The jury is still out

By D.R.

THE first rule of polling analysis is to take outlier results with a liberal helping of salt. Whenever a large number of surveys are conducted using proper random sampling techniques, a handful are all but guaranteed to yield results far from the consensus: of every 20 polls with a margin of error of three percentage points, you’d expect one to miss by six or more. Rather than suffering whiplash any time an atypical poll is published, seasoned election forecasters recommend focusing on the slower-moving averages.

Rarely has such sage counsel proven less relevant than in the 2017 French presidential campaign, where outlier polls have been closer to extinct than endangered. Of the 11 surveys taken in the past five days, every single one had Marine Le Pen between 21.5% and 23% of the vote, Emmanuel Macron between 22% and 24%, François Fillon between 18% and 21%, Jean-Luc Mélenchon between 18% and 22% and Benoît Hamon with either 7.5% or 8%. Such metronomic consistency has been the rule, not the exception: just look how closely the dots (each an individual survey) cluster around the trend lines in our poll tracker below. During the past two months, the average poll result for a single candidate has differed from the average of all other surveys taken in the surrounding time period for that candidate by a paltry 0.7 percentage points.

On the surface, such emphatic agreement might seem to bolster our confidence in polls’ accuracy. Among savvy analysts, however, it primarily induces dread. Although pollsters eagerly depict themselves as passive messengers of the electorate’s preferences, their results depend nearly as much on methodological choices as they do on respondents’ answers. In one striking example, the New York Times gave four esteemed experts the exact same raw data, and found that their conclusions differed by up to five percentage points. And while there is plenty of room for genuine disagreement about demographic weighting procedures, likely-voter screens and the like, there is also little to prevent pollsters from manipulating these fine-grained decisions in order to reverse-engineer a desired result. The most common example of such behaviour is “herding”, the tendency to shift an outlier figure towards the consensus in order to avoid recriminations if it proves wrong.

Herding has a long and shabby history in the polling world. It represents a classic collective-action problem, making any individual pollster (particularly a bad one) more accurate but the overall average less so. And given how the numbers from eight different polling firms with regularly published results (BVA, Elabe, Harris, Ifop-Fiducial, Ipsos, Kantar, Sofres, Odoxa and OpinionWay) have moved in near lockstep, FiveThirtyEight, a statistical-journalism website, is sounding the alarm. Both Harry Enten, a political writer, and Nate Silver, the site’s founder, recently tweeted well-founded concerns about the lack of variation in French election polling. The first round is already extremely volatile: in our current election forecast, all four leading candidates have between a 30% and 70% chance of advancing to the run-off (see chart below). Adding a dose of herding-driven uncertainty would render it a pure crapshoot.

Without a mole or whistle-blower lurking in the depths of a wayward polling firm, herding is virtually impossible to prove. However, the practice does leave statistical footprints if it is sufficiently prevalent: a heavily herded election will show a far narrower range of poll results than one would predict from the inescapable sampling error inherent in relying on small groups of respondents to proxy for the views of large populations. The bigger the gap between the expected and observed amount of variation, the greater the risk that pollsters are herding, and the less confidence we should have in survey averages that superficially appear rock-solid.

Assessing the degree of disagreement among polls is straightforward: you simply take the standard deviation, which measures how bunched-together or spread-out a series of numbers are, around the overall average for surveys taken at a similar point in time. In contrast, calculating the predicted variation, or standard error—a precursor to the familiar “margin of error” frequently reported alongside poll results—is a bit trickier. It depends both on the number of people in each survey (fewer respondents yield bigger errors) and on the vote share of each contender. The closer a candidate is to 50%, the greater the uncertainty around the estimate: with 1,000 answers, the margin of error surrounding 10% support for a candidate is 1.9 percentage points; for one backed by half of respondents, the figure is 3.1.

As it happens, both of these variables point to very steady results in France. Whereas nationwide polls in the United States typically contact around 1,000 respondents, and state-level surveys roughly half that, the median French poll during this cycle has reached over 1,500 people, with some online efforts catching ten times as many. Moreover, in this wide-open, five- then four-way race, no candidate has even approached 30% support. As a result, based on sampling error alone, the leader’s vote share should fall within a 4.5-point range 95% of the time.

Because of this deceptively low benchmark, the tight band of French poll results looks like plenty of smoke but no fire. Among 75 surveys taken over the past month, sampling error suggests that 32% of estimates for individual candidates should fall more than 0.93 percentage points away from those contenders’ true support at that point in time. Using the overall polling average during the surrounding two weeks as the closest estimate of the underlying reality, 32% actually landed beyond a range of…0.91 percentage points. Rarely do theory and practice line up so well.

Two other peripheral data points cast further doubt on the notion that French pollsters are up to no good. 2017 is not the only election cycle in which first-round surveys have looked this much alike: the numbers in the 2012 campaign were equally steady, and predicted the result almost perfectly. Moreover, polls for this year′s run-off scenarios have been far noisier: in the potential matchup between Mr Macron and Ms Le Pen, his recent vote shares have ranged from 58.5% to 71%, while Mr Fillon′s against Ms Le Pen have stretched from 52.5% to 64%. These wider bands are exactly what you would expect as proportions move closer to 50%.

Nonetheless, these seemingly reassuring findings aren’t sufficient to refute concerns about herding entirely. First, because of the difficulties of contacting hard-to-reach groups of voters, modern polls are not simple random samples. This makes their true expected range of variation far less certain. Moreover, there are many other sources of error beyond sampling that can cause survey results to diverge, and no two pollsters use the exact same method. All of these factors should increase variation between polls. The fact that the French data exactly match sampling-based expectations, rather than exceeding them, is itself mildly suggestive of modest herding.

The other cause for continued alarm is that the overall average obscures differences between the candidates (see chart above). The figures for the two leaders, Mr Macron and Ms Le Pen—the ones that set off FiveThirtyEight’s herding radar—have indeed been eerily consistent. During the past month, Mr Macron’s numbers have clustered together around 18% more than expected, and Ms Le Pen’s a downright disturbing 30% more. Those results have been balanced out by an unusually large degree of disagreement among pollsters regarding support for the surging Mr Mélenchon: his estimated vote shares have diverged 28% more than sampling error would indicate.

It’s easy to spin a narrative to account for this discrepancy. When dealing with the favourites to advance to the run-off, who have received fairly stable support from voters and attract the most interest, pollsters in CYA mode have been loath to veer off the consensus. In contrast, they could not be bothered to doctor the results of a candidate with more uncertain support levels (Mr Mélenchon), who was until recently regarded as an extreme long-shot.

Then again, with a sufficiently severe case of confirmation bias, you can drum up a story to fit just about any fact pattern. The safest conclusion is that the evidence for herding is mixed at best, and that we should regard the current polling averages as no less reliable than those in previous French presidential campaigns. But given the small margins separating the top four contenders, that still means that any of them could plausibly win.