class: left, top, title-slide # Taking the pulse of ‘the pulse of democracy’ ## The promise and peril of poll aggregation ### <strong>G. Elliott Morris</strong> ### Mar 22, 2023 | Charlottesville, VA --- <img src="figures/cover.jpg" width="50%" /> ??? - here to talk about my book strength in numbers, about how polls work and why we need them for a healthy democracy - but before we get there: a quick recent history of polls in 2016, 2018, 2020 and now 2022 - 2016 was in many ways the catalyst for writing this book - Then, I was still in college at the university of Texas, but I had reverse-engineered the 538 forecasting model in my spare time as a way to learn computer programming --- class: center, middle <img src="figures/538_2016.png" width="75%" /> ??? - You may remember, however, that polls that year were not very accurate - Had Clinton winning in Michigan, Wisconsin, pennsylvania, Florida and NC - All of which she lost --- class: center, middle <img src="figures/wapo_2016_states.png" width="90%" /> ??? - Overall, polls missed by 5 points in the states - One of the biggest misses ever --- class: center, middle <img src="figures/538_2016_states.png" width="90%" /> ??? - Looking at polls that were released only in the final 21 days of the campaign and only in competitive states, the average state poll missed Hillary Clinton’s margin over Donald Trump by about 3 points - But even worse, most of those polls systematically underestimated Trump — they had high bias not just high error - In fact, the average poll was more biased than at any point since 1998 --- class: center, middle ## The fundamental problem with polling <img src="figures/phone.jpeg" width="70%" /> ??? - To understand the reason for that miss you must understand the fundamental problem that pollsters have to solve: nonresponse. - That is: the people who answer a poll may be different — demographically, politically, culturally — than the people who don’t. - And it turns out that solving that equation is very hard, in part because there is a lot of randomness in polls due to low response rates. --- ### Pollsters use statistical algorithms to ensure their samples match the population on different demographic targets - Race, age, gender, and region are most common - Political variables (sometimes) - Can use weighting (raking) modeling (MRP), w various tradeoffs .pull-left[ <img src="figures/raking.jpg" width="100%" /> ] .pull-right[ <img src="figures/mrp.jpg" width="100%" /> ] ??? - Pollster get around nonresponse by adjusting (or “weighting” their samples to be demographically representative of the electorate. - If their poll should have 70% white voters, for example, but only 35% of respondents are white, then all the white voters get a weight of 2 and everyone else gets halved. They repeat that process for a bunch of traits: race, age, education, gender, region, etc. --- class: center, middle ### Pew used 12 weighting variables in 2020! <img src="figures/pew_weights.png" width="65%" /> ??? - Pew Research Center had 12 weighting variables in 2020! - All of this is try to make the data pollsters actually get representative of the data they should have got, in theory --- class: center, middle ## 2016: Demographic nonresponse bias <img src="figures/weighting_education.jpg" width="100%" /> ??? - The canonical explanation now is that in 2016, the nearly uniform bias in polls came higher rates of nonresponse among non-college-educated voters - And you can see that with this graph. - The NYT estimates the if all national 2016 had weighted their data to be representative of race and education, their error would have dropped from 9 points to 2 points. Even though that’s not perfect, it is a big improvement! --- class: center, middle ## 2020: Partisan nonresponse bias <img src="figures/gq_rs_polls.jpg" width="80%" /> ??? - That’s because the problem in 2020 was not demographic nonresponse, but political nonresponse. Pollsters got the wrong mix of Trump voters within demographic groups. This time, the white non-college voters that answer their polls were systematically too democratic. --- ## Partisan nonresponse bias -- - ### Problem reaching Trump voters overall -- - ### And _within_ demographic groups -- - ### Something you cannot fix with weighting -- - #### Pollsters can adjust for past vote, but the electorate changes, and certain _types_ of voters may not respond to surveys ??? - And that’s something that is very, very hard to adjust for after the fact. - For one thing, there is no official record from the Census of how many democrats there are in the country. - And while you could run models on past polling data and try to weight to the right percentage of white 2016 Trump voters, that doesn’t help you among people who changed their minds over time. --- <img src="figures/tnr_2020_polls.png" width="100%" /> --- class: center, middle <img src="figures/538_2020_states.png" width="75%" /> ??? - At the state level, polls in 2020 underestimated Donald Trump by 4 points. That was an even larger bias than in 2016! - That was a shock to many in the industry who had changed their methods since the last time around. Even the pollsters who were adjusting their polls to be demographically representative of the population had large errors. --- class: center, middle, inverse ## Bias vs variance --- ## On average, _error_ in polls is low <img src="figures/aapor.png" width="80%" /> ??? - Reference earlier slide - So while on average the absolute error of polls and averages is low by historical comparisons --- ## But a good aggregate needs polls with low _bias_, -- ### especially at the state level. -- <br> <img src="figures/bias_variance.png" width="100%" /> ??? - What election forecasts really need is unbiased surveys - BC electoral college especially at the state level --- class: center, middle, inverse # Problem: bias in polls has been increasing ??? - and that's where pollsters run into their big problems --- ## Problem: bias in polls has been increasing... <img src="figures/potus_poll_error_bias.png" width="60%" /> ??? - Since 2000 bias in the polls has been growing - At least in presidential election years --- ## ... and bias is correlated across levels and states <img src="figures/poll-bias-states-time.png" width="70%" /> ??? - and bias correlated across states, too - Explain scatter plot --- class: center, middle, inverse # Solution 1: Less biased polls! --- ## Less biased polls -- - Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval) -- - Something you cannot fix with standard weighting. -- - So... -- #### Options: -- 1. More weighting variables (NYT) -- 2. More offline and off-phone data collection (Pew NPORS, SSRS, NORC) -- 3. Mixed-mode samples (promising, but not yet popular among public pollsters) --- ## The polls in 2022 <img src="figures/cohn_2022.png" width="95%" /> ??? - Maybe this worked out? - Nate Cohn --- ## The polls in 2022 <img src="figures/economist_2022.jpeg" width="95%" /> ??? - In fact all pollsters had a good year --- class: center, middle <img src="figures/ekins_2022.jpeg" width="95%" /> ??? - From traditional pollsters to some new ones --- class: center, middle # The problem with solution 1: ### No clear evidence that pollsters who changed methods did better than adapters -- ### Just as likely as pollsters got lucky as it is that any individual overperformed --- ### What happens when pollsters get unlucky again? ??? - But there is no evidence this is due to methods changes - Just getting lucky --- class: center, middle, inverse # "Solution" 2: Let the aggregation model debias the polls --- ## Case study: _Economist_ model -- <img src="figures/econ_bayes.png" width="60%" /> ??? - We are fully Bayesian --- <img src="figures/econ_stan.png" width="100%" /> ??? - In fact, here is our code! --- <img src="figures/econ_bayes.png" width="60%" /> ??? - We are fully Bayesian - We start with a fundamentals-based "prior" about the election outcome nationally - Mention what goes into that (ML regularization) - Decompose it into constituent state forecasts - Measure historic error with leave-one-out cross-validation - That gets plugged into the modeli n every state --- ## Case study: _Economist_ model -- #### i. State-space model: Latent state trends (vote shares) evolve as a hierarchical random walk over time -- #### ii. Polls are weighted by their historical error *and bias* * Based on past relationship between a pollster's lagged historical bias and performance of the aggregate -- #### iii. Polls are observations with constant random effects to "debias" based on: * Pollster firm (so-called "house effects") * Poll mode * Poll population -- #### iv. Polls are also adjusted for potential partisan non-response * Each poll has a covariate for whether it weights by party registration or past vote * Effect is allowed to change over time * Adjusts for biases that remain AFTER removing the other biases ??? - Walk through slide --- <img src="figures/economist_backtest_rw.png" width="100%" /> ??? - Show how the model works graphically --- ## Notable improvements! <img src="figures/states-vs-results.png" width="80%" /> ??? - Point out how model does in past --- class: center, middle # In 2016... -- ## ... But not 2020 -- <img src="figures/2020-economist-histogram.png" width="80%" /> ??? - This works in 2016 - But not in 2020 --- class: center, middle ## The problem with solution 2: -- ### 1. Pollsters change their methods, making adjustments outdated -- ### 2. Even sophisticated adjustments don't work if there are no unbiased surveys! ??? - Problem with solution 2 is that - Pollsters change their methods over time, messing with the bias adjustment - And bias adjustments cannot work if there are no unbiased polls! --- class: center, middle, inverse # Solution 3: Conditional forecasting! --- class: center, middle ## Solution 3: Conditional forecasting! -- ### - Present aggregates assuming some amount of polling _bias_. -- ### - As a way to explain to readers how bias enters the process of polling -- ### - And what happens to forecasts if bias _now_ does not follow historical distributions --- ## Conditional forecasting: -- .pull-left[ ## 1. Debias polls <img src="figures/conditional_forecasting_one.png" width="80%" /> ] -- .pull-right[ ## 2. Rerun simulations <img src="figures/conditional_forecasting_two.png" width="80%" /> ] --- # 2. Rerun simulations <img src="figures/conditional_forecasting_two.png" width="80%" /> --- # 2. Rerun simulations <img src="figures/conditional_forecasting_two.png" width="50%" /> ### Advantage: leaves readers with a much clearer picture of possibilities for election outcomes _if past patterns of bias aren't predictive of bias now_ (2016, 2020) ??? - Explain the advantage --- ## Conditional forecasting: <img src="figures/conditional_forecasting_three.png" width="90%" /> ??? - Here is what this could have looked like in 2020 --- ## Conditional forecasting: ### Advantage: Leaves readers with a much clearer picture of possibilities for election outcomes _if past patterns of bias aren't predictive of bias now_ (2016, 2020) ### Disavantage: When polls are right, readers feel misled (2018, 2022) ??? - But only useful really in years with high uniform bias in the polls --- <img src="figures/cohn_2022.png" width="95%" /> --- class: center, middle, inverse ## The state of the polls ??? - So let's take stock --- ## The state of the polls -- ## "The pulse of 'the pulse of democracy'" -- #### 1. Individual surveys subject to high degree of error -- #### 2. Industry-wide bias is unpredictable and potentially large -- #### 3. Not enough information to aggregate away this bias -- #### 4. But low variance from aggregation gives the (false) impression of precision -- #### 5. Solution is not in statistics, probably, but visualizaation and communication ??? - Walk through slide --- # Thank you! ### Questions? (Suggestions?) .pull-left[ <img src="figures/cover.jpg" width="50%" /> ] .pull-right[ **Website: [gelliottmorris.com](https://www.gelliottmorris.com)** **Twitter: [@gelliottmorris](http://www.twitter.com/gelliottmorris)** **Newsletter: [gelliottmorris.substack.com](https://gelliottmorris.substack.com)** ] --- .bottom[ _These slides were made using the `xaringan` package for R. They are available online at https://www.gelliottmorris.com/slides/_ ]