I will attempt to answer the questions in the title through an examination of the dataset that accompanies Jennings and Wlezien (2018), Election polling errors across time and space. The main purpose of the article is to investigate the question as to whether polls have become less reliable over time. One of their key findings can be summarized from the following brief excerpt:
We find that, contrary to much conventional wisdom, the recent performance of polls has not been outside the ordinary; if anything, polling errors are getting smaller on average, not bigger.
A secondary task of Jennings and Wlezien is to ask whether the institutional context matters for polling accuracy. This sort of question is just what this virtual orchard exists for, and I was not satisfied with the treatment of electoral systems in the article. Fortunately, their dataset is available and is in Stata format, so I went about both replicating what they did (which I was able to do without any issues) and then merging in other data I have and making various new codings and analyses.
My hunch was that, if we operationalize the electoral system as more than “proportional or not”, we would find that more “permissive” electoral systems–those that favor higher party-system fragmentation and proportionality–would tend to have larger polling errors. I reasoned that when there are more parties in the system (as is usually the case under more permissive systems), voters have more choices that might be broadly acceptable to them, and hence late shifts from party to party might be more likely to be missed by the polls. This is contrary to what the authors expect and find, which is that mean absolute error tends to be lower in proportional representation (PR) systems than under “SMD” (single-member districts, which as I always feel I must add, is not an electoral system type, but simply a district magnitude). See their Table 2, which shows a mean absolute error in the last week before electoral day of 1.62 under PR and 2.28 under “SMD”.
The authors also expect and show that presidential elections have systematically higher error than legislative elections (2.70 vs. 1.83, according to the same table). They also have a nifty Figure 1 that shows that presidential election polling is both more volatile over the timeline of a given election campaign in its mean absolute error and exhibits higher error than legislative election polling at almost any point from 200 days before the election to the last pre-election polls. Importantly, even presidential election polls become more accurate near the end, but they still retain higher error than legislative elections even immediately before the election.
This finding on presidential elections is consistent with my own theoretical priors. Because presidential contests are between individuals who have a “personal vote” and who are not necessarily reliable agents of the party organization, but are selected because their parties think they can win a nationwide contest (Samuels and Shugart, 2010), the contest for president should be harder to poll than for legislative elections, all else equal. That is, winning presidential candidates attract floating voters–that is pretty much the entire goal of finding the right presidential candidate–and these might be more likely to be missed, even late in the campaign.
To test my own hunches on the impact of institutions on polling errors, I ran a regression (OLS) similar to what is reported in the authors’ Table 3: “Regressions of absolute vote-poll error using polls from the week before Election Day.” This regression shows, among other results, a strong significant effect of presidential elections (i.e., more polling error), and a negative and significant effect of PR. It also shows that the strongest effect among included variables is party size: those parties that get more than 20% of the vote tend to have larger absolute polling errors, all else equal. (I include this variable as a control in my regression as well.)
The main item of dissatisfaction for me was the dichotomy, PR vs. SMD. (Even if we call it PR vs. plurality/majority, I’d still be dissatisfied). My general rule is do not dichotomize electoral systems! Systems are more or less permissive, and are best characterized by their seat product, which is defined as mean district magnitude times assembly size. Thus I wanted to explore what the result would be if I used the seat product to define the electoral system.
I also had a further hunch, which was that presidential elections would be especially challenging to poll in institutional settings in which the electoral system for the assembly is highly permissive. In these cases, either small parties enter the presidential contest to “show the flag” even though they may have little chance to win–and hence voters may be more likely to defect at the end–or they form pre-election joint candidacies with other parties. In the latter case, some voters may hedge about whether they will vote for a candidate of an allied party when their preferred party has no candidate. Either situation should tend to make polling more difficult, inflating error even late in the campaign. To test this requires interacting the seat product with the binary variable for election type (presidential or legislative). My regression has 642 observations; theirs has 763. The difference is due to a few complex systems having unclear seat product plus a dropping of some elections that I explain below. Their findings hold on my smaller sample with almost the precise same coefficients, and so I do not think the different sample sizes matter for the conclusions.
When I do this, and graph the result (using Stata ‘margins’ command), I get the following.
I am both right and wrong! On the electoral system effect, the seat product does not matter at all for error in legislative elections. That is, we do not see either the finding Jennings and Wlezien report of lower error under PR (compared to “SMD”), nor my expectation that error would increase as the seat product increases–EXCEPT: It seems I was right in my expectation that error in presidential contests increases with the seat product of the (legislative) electoral system.
The graph shows the estimated output and 95% confidence intervals for presidential elections (black lines and data points) and for legislative (gray). We see that the error is higher, on average, for presidential systems for all seat products greater than a logged value of about 2.75, and increasingly so as the seat product rises. Note that a logged value of 2.75 is an unlogged seat product of 562. Countries in this range include France, India, the Dominican Republic, and Peru. (Note that some of these are “PR” and some “SMD”; that is the point, in that district magnitude and formula are not the only features that determine how permissive an entire national electoral system is–see Shugart and Taagepera, 2017.)
I have checked the result in various ways, both with alternative codings of the electoral system variable, and with sub-sets, as well as by selectively dropping specific countries that comprise many data points. For instance, I thought maybe Brazil (seat product of 9,669, or a logged value just short of 4) was driving the effect, or maybe the USA (435; logged =2.64) was. No. It is robust to these and other exclusions.
For alternatives on the coding of electoral system, the effect is similar if I revert to the dichotomy, and it also works if I just use the log of mean district magnitude (thereby ignoring assembly size).
For executive format types, running the regression on sub-samples also is robust. If I run only the presidential elections in pure presidential systems (73 obs.), I still get a strong positive and significant effect of the seat product on polling error. If I run only on pure parliamentary systems (410 obs.), I get no impact of the seat product. If I restrict the sample only to semi-presidential systems (159 obs.), the interactive effect holds (and all coefficients stay roughly the same) just as when all systems are included. So it seems there is a real effect here of the seat product–standing in for electoral system permissiveness–on the accuracy of polling near the end of presidential election campaigns.
I want to briefly describe a few other data choices I made. First of all, legislative elections in pure presidential systems are dropped. The Jennings and Wlezien regression sample actually has no such elections other than US midterm elections, and I do not think we can generalize from that experience to legislative vs. presidential elections in other presidential systems. (Most are concurrent anyway, as is every presidential election in the US and thus the other half of the total number of congressional elections.)
However, I did check within systems where we have both presidential and legislative polls available. All countries in the Jennings-Wlezien regression sample that are represented by both types of election are semi-presidential, aside from the US. In the US, Poland, and Portugal, the pattern holds: mean error is greater in presidential elections than in assembly elections in the same country. But the difference is significant only in Portugal. In Croatia the effect goes the other way, but to a trivial degree and there are only three legislative elections included. (If I pool all these countries, the difference across election types is statistically significant, but the magnitude of the difference is small: 2.22 for legislative and 2.78 for presidential.)
The astute reader will have noticed that the x-axis of the graph is labelled, effective seat product. This is because I need a way to include two-tier systems and the seat product’s strict definition (average magnitude X assembly size) only works for single-tier systems. There is a way to estimate the seat product equivalent for a two-tier system as if it were simple. I promise to explain that some time soon, but here is not the place for it. (UPDATE: Now planted.)
I also checked one other thing that I wanted to report before concluding. I wondered if there would be a different effect if a given election had an effective number of parties (seat-winning) greater than expected from its seat product. The intuition is that polling would be tend to off more if the party (or presidential) contest were more fragmented than expected for the given electoral system. The answer is that it does not alter the basic pattern, whereby it makes no difference to legislative elections (in parliamentary or semi-presidential systems). For presidential elections, there is a tendency for significantly higher error the more the fragmentation of the legislative election is greater than expected for the seat product. The graph below shows a plot of this election; as you can probably tell from the data plot, the fit of this regression is poorer than the one reported earlier. Still, there may be something here that is worth investigating further.