No truth stands alone.

Has Australian Polling Accuracy Changed Over Time?

The State of the Polls

As I write this, Labor’s 2-party lead in the opinion polls has expanded to about 54% against the Coalition. The combined major-party voting intention is either fairly static (Newspoll – 73%, Essential 75.5%) or down somewhat on the 74.8% they won last time (Morgan 69.5%, Ipsos 68.8%, Resolve 67%) – nowhere near enough to significantly threaten Labor’s odds of a majority. The government’s 2pp in the polls is lower than all of its predecessors at the same point but Howard ’07 (44.1%), Fraser ’83 (44.8%) and Whitlam ’75 (44.7%), with its deficit in the polls twice that of the worst-polling government to be returned (Howard ’98, 47.9% – won despite losing the popular vote). All 2pp estimates calculated using last-election preference flows.

I treat the 1975 election as having a Labor government going into it, because 1) Fraser didn’t manage to do much (if any) governing before the election was called, and 2) it’s fairly clear in the polls that the Whitlam government was the one being punished for the horrendous economy (stagflation) of the late 1970s.

Despite all of that, many people are still jaded about Labor’s chances of winning due to the unusually large polling error in 2019, which re-elected the Morrison government despite (in my view over-inflated) expectations of a Labor victory. I’m not here to tell you that this won’t happen again – if I wanted to over-sell confidence, I would be in the fake supplements market.

But I am here to tell you that even if the polling error of 2019 repeats itself, Labor is likely to end up ahead anyway.

Before we go into that – what are the odds of such a polling error in the first place? This is the first of a two-part piece, where I analyse whether we should be worried about the polls in the first place; next we’ll project the electoral map if the polls bungle it up again anyway.

What have Australian pollsters changed in response to 2019?

Overall there has been a shift towards more transparency, with several major pollsters forming the Australian Polling Council (APC), which mandates a minimum standard of methodology disclosures. Three pollsters who conduct regular federal polling have joined the APC and these appear to have been their methodology changes:

  1. YouGov (conducts Newspoll): YouGov has dropped the interactive voice response (IVR, aka “robopoll”) component of Newspoll, which it described as being “erratic” in the lead-up to the 2019 election. Additionally, it stated that failure to weight by education was partly responsible for the 2019 error and has adopted weighting by education in response. This finding is in line with findings from similar polling errors overseas and from the AMSRO Inquiry into the Performance of Opinion Polls.
  2. Essential: Essential appears to have adopted weighting by party identification, though I’m not entirely clear on how they source the party ID data they weight to. Their methodology disclosure says that their weighting target source is the “ABS, AEC”. However, neither asks about party identification or provides information on party ID as far as I am aware. This appears to be a new weighting frame in the context of Australian polling and may be the reason why Essential has tended to suggest a lower vote for Independents and Other Minor Parties than other polls taken at a similar time.
  3. Ipsos: Ipsos appears to have added an online sample to their previously phone-only poll; it’s unclear if they made any additions or changes to their weighting frames in response to 2019. At minimum, this appears to have reduced the previous Ipsos poll’s tendency to over-estimate the Green vote and under-estimate the Labor primary.

We also have a new pollster, Resolve Strategic, which appears to be trialing a simulated ballot form of surveying (where respondents rank preferences in a simulated ballot paper) and Roy Morgan, neither of whom have opted to join the APC. It’s unclear if Morgan has made many changes in response to the 2019 polling error; it has shifted away from face-to-face and finally adopted the last-elections-preference-flow method of estimating two-party-preferred. However, neither change seems to directly targeted at the causes of the 2019 error – most of the 2019 error was due to error between the Labor/Coalition primary, and the shift away from face-to-face polling occurred at the start of the pandemic.

So to recap: three pollsters have made changes in response to 2019, one is brand new but is attempting to test a hypothesis of how to poll voting-intention better than the pollsters did in 2019, and one has at least shifted away from methods known to over-estimate Labor even if we don’t know whether they did so in response to 2019. Given these changes, it’s unlikely that the exact same type of error that happened in 2019 will repeat itself, though it doesn’t preclude a different error affecting the polls.

Wait, but what if all of this is window dressing? What if the polls are getting worse, and the changes made by the pollsters above aren’t going to solve declining response rates and worsening non-response effects?

Has the accuracy of Australian polling declined over time?

This is a simple hypothesis to test – we can just plot the absolute error of each final poll at each election versus when the election was held: I define final as being conducted within a week of the election date, and being the last voting-intention poll published by the relevant pollster/under the relevant polling brand.

2pp absolute polling errors at state and federal level over time
Errors were calculated from the pollsters’ published 2pp estimate if available. If a linear trend was not statistically significant at p < 0.05, it was plotted with a dashed, translucent line.

Well, that’s a bit of a narrative destroyer. There isn’t much of a trend either way (note that neither trend above is statistically significant at the usual threshold of p < 0.05); there were some pretty big errors in the 1990s and 2000s too.

What about skew to either side of politics?

2pp polling errors at state and federal level over time
Errors were calculated from the pollsters’ published 2pp estimate if available. If a linear trend was not statistically significant at p < 0.05, it was plotted with a dashed, translucent line.

The finding that state polling has tended to over-estimate the Coalition over time (p = 0.035) probably needs to be taken with a grain of salt here given how few state polls we have from the 1980s – 1990s. Additionally, I use the pollsters’ published 2pp estimates here due to the difficulty of hunting down preference flow data for state. Some pollsters have used respondent-allocated preferences to estimate 2pp, which tends to skew to Labor by 0.6% on average relative to assuming that preferences from minor party voters will flow as they did at the last election. Hence, I’d advise against assuming there’s a skew to Labor solely off the above graph.

However, as I’ve noted elsewhere, the average size of polling errors doesn’t matter as much if those polling errors “cancel out”. Or in other words, the polls as a whole can still be predictive with large errors, as long as some of those errors over-estimate Labor and some over-estimate the Coalition. Are the errors on polls conducted at the same election getting more similar?

We can indirectly test this by analysing the average absolute error of the polling average instead:

2pp polling average absolute errors at state and federal level over time
Errors were calculated from the pollsters’ published 2pp estimate if available. Elections where only one final poll was available not included.

If you squint, there might be some, very weak relationship between the absolute error on the polling average over time. However, it’s not even statistically significant at p < 0.05, and I suspect we’re missing a few of the larger polling errors from the late 1980s and early 1990s as I have not been able to find any archive of opinion polling that goes back that far.

What about skew to the Coalition? Has the polling average gotten more skewed over time?

2pp polling average errors at state and federal level over time
Errors were calculated from the pollsters’ published 2pp estimate if available. Elections where only one final poll was available not included.

I’ll repeat my caution above about assuming the polls are skewed to Labor solely off this graph, due to the use of respondent-allocated preferences by some of the polls included.

However, there appears to be basically no relationship; if anything the polls have been shifting away from a historical skew to Labor in recent years (with some notable exceptions).

(if you want to skip my hypothesising about why it may appears as though the polls got worse and get to what this means for 2022, click here)

If polling accuracy hasn’t really changed over time, why the narrative of the polls getting worse? My guess is that it’s some combination of:

  1. There was a genuine period of remarkable poll accuracy, which led to over-inflated expectations. If you look at the absolute error plots, you can see that the pollsters had something of a decennium mirabilis in the decade centred on 2016, with very few large polling errors and most polls landing fairly close to the mark. Without the context of the fairly large polling errors of the early 2000s, this may have resulted in people over-estimating the accuracy of opinion polling.

  2. Even when there were large polling errors, they happened to not affect who won. An example is the 1996 federal election, where the published 2pp estimates on the polls averaged out to L/NC 51.5 but the result was L/NC 53.6, for a polling error of 2.1% (for context, the error in 2019 was 2.9% by the same metric). The average error on the 2pp estimates was actually larger than in the 1993 election; I get 1.6% by the same metric; polling error estimates may differ if weighting, house effect adjustments, trend functions (LOESS etc) and/or last-election preference flows are used instead. however since Howard won anyway we all remember 1993 as the “unloseable election” where “the polls got it wrong”.

  3. The pollsters made genuine improvements to their methodology which fixed some of the issues seen at past elections. While we don’t know a lot about historical pollster methodologies, but there are some changes that stand out. Namely, the shift away from using respondent-allocated preferences to estimate the two-party-preferred (if Newspoll had done so in 2004, it would have an error of just 1.3% rather than 2.7%) and from using face-to-face polling methodologies which appeared to skew Labor.

  4. Where large errors happened, they were rarely highly correlated. With a few notable exceptions (Morgan 2001), we haven’t had many large polling errors at the federal level. And of the elections where we’ve seen large polling errors, the most common outcome was that a different poll would err in the opposite direction, at least partly “cancelling out” the error of the inaccurate poll.

    This not only reduced the error on the polling average, it also helped dampen any expectations that either side would win – an issue with expectations of a Labor win at the 2019 election as well as coverage of polling accuracy afterwards. Have a drink every time you hear the statistically-meaningless (and inaccurate) phrase “no opinion poll showed the Coalition ahead through its term” and you’ll very quickly see what I mean (or not see anything at all, depending on how many articles you go through).

  5. The federal Newspoll has had a genuinely above-average run of predictive accuracy. Since its inception, the federal Newspoll has only called the popular-vote winner wrong just once (2019) in its published 2pp estimate. This is a combination of 2) large errors not affecting who won and a lower error versus other pollsters (Newspoll federal errors circled in black):

    2pp absolute polling errors at state and federal level over time, Newspoll highlighted

    Newspoll pretty much anchored the decennium mirabilis of Australian opinion polling, nailing the 2pp to within 1% from 2007 to 2016 (and it was reasonably accurate on primary votes in 2004 too, even if its 2pp estimate was bungled due to respondent preferences). As Newspoll tended to be the pollster which received the most attention from the media and politicians, large errors by other pollsters failed to receive the attention they probably should have gotten, resulting in an unrealistic portrayal of Australian polling accuracy.

It’s entirely possible that the continued decline in response rates and selection effects in who chooses to respond to an opinion poll will finally produce a sustained worsening of poll accuracy in Australia. However, the historical data suggests that the polling errors of recent elections appear to be a reversion to the mean after a decade of genuinely remarkable polling accuracy, not a sustained decline in polling accuracy over time.

Of course, this could either reflect minimal change in the difficulty of constructing a representative sample (maybe the landline polls of the 2000s weren’t as good as we think?) or pollsters getting better at using weighting and sampling to construct a representative sample despite the increased challenges of response rate, response effects etc.

What about 2022?

In the context of the upcoming federal election, how unusually inaccurate would the polls have to be for the Coalition to be re-elected? Here’s the plot of the error on all polls, with the current (15/May/2022) poll lead and the final 2pp poll lead in 2019:

2pp absolute polling errors at state and federal level over time, poll lead from 2019 (51.4) and 2022 (54.1) plotted in

In 2019, the polling error meant that the Coalition won with a reasonable 2pp lead when they were behind by roughly the same amount in the final polls. It’s unlikely something like that happens again this time; for the Coalition to overturn Labor’s popular vote lead they would require a polling error of the size only seen in four polls of 116, or a probability of about 1-in-29 (3.4%).

It’s even less likely when you consider that many of those polling errors were either rogue polls (Morgan 2001) or where they were the only poll at that election (Newspoll, QLD 1995). An average of multiple polls is generally more accurate; which is reflected in the plot of polling average absolute errors:

2pp polling average absolute errors at state and federal level over time, with 2019 poll lead (51.4) and 2022 poll lead (54.1) included.

So far, we have never had an election with more than one pollster and the resulting polling average being so inaccurate that the side polling 54% or more would lose the popular vote.

This is not to say that it couldn’t happen – if polling errors are getting more correlated over time, then the probability of such an error would approach the ~3% seen in individual polls. But it still doesn’t suggest a particularly high probability of the Coalition winning the 2pp.

Of course, that’s not the exact same thing as saying the Coalition can’t win the election overall – the Howard government was re-elected in 1998 despite losing the 2pp (by about 49-51). Could the current Morrison government pull off a similar hat trick using a smaller polling error?

Tune in next time, when we explore what the electoral map would look like if Labor gets hit by a repeat of the 2019 polling error.

Edit (19/May/2022): Since this piece was published, some pollsters have begun to show Labor’s lead tightening to roughly where it was at the start of the campaign; bringing the unadjusted polling average in from ALP 54.1 to ALP 53.4 (only weighted for time of release). This makes a polling error big enough to overturn Labor’s popular vote lead somewhat more likely though still uncommon by historical standards:

2pp absolute polling errors at state and federal level over time, poll lead from 2019 (51.4) and 2022 (53.4) plotted in
2pp polling average absolute errors at state and federal level over time, with 2019 poll lead (51.4) and 2022 poll lead (53.4) included.

2 thoughts on Has Australian Polling Accuracy Changed Over Time?

  1. Congratulations on your modelling.

    One question about your forecasting model, does it take into account postal and prepoll votes cast each day and apply your estimated 2PP%’s for each of those days?

    1. (Don’t congratulate us on the model yet, our predictors might turn out to be completely wrong!)

      The model doesn’t forecast or take into account postal and pre-polls. If you had a god’s-eye-view of the whole thing, you might be able to go “hey there’s more postals and pre-polls than usual, that should imply reduced uncertainty”, but realistically none of our predictors should be influenced by postal or pre-poll voting.

      A voter who has voted can just respond with whoever they voted for in an opinion poll; they can still say whether they approve of the PM or the OL, and if economic factors drove their vote, well, the economic factors usually don’t change that much in the couple of weeks between the opening of early voting and election day. Since early votes aren’t even counted until election night, I don’t think there’s anything that can be gleaned from rates of postal and pre-poll votes.

Add Your Comment

× Home Politics Economics Projects
×

You appear to be using an outdated browser, for which this site is not optimised.

For your security, we strongly recommend you download a newer browser.

Although any of the latest browsers will do, we suggest the latest version of Firefox.

Download Firefox here

(Click on the button in the top-right to close this reminder)