Polling aggregation in an era of high (potential) bias

class: left, top, title-slide

# Polling aggregation in an era of high (potential) bias
## From Strength in Numbers: How Polls Work + Why We Need Them
### G. Elliott Morris | Oct 19 2022 | NYAAPOR

---

---
## On average, _error_ in polls is low

---
## But a good aggregate needs polls with low _bias_,

### especially at the state level.

---
class: center, middle, inverse

# Problem: bias in polls has been increasing

---
## Problem: bias in polls has been increasing...

---
## ... and bias is correlated across levels and states
<img src="figures/poll-bias-states-time.png" width="70%" />

---
class: center, middle, inverse

# Solution 1: Less biased polls!

---
## Less biased polls

- Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)

- Something you cannot fix with standard weighting.

- So...

#### Options:

1. More weighting variables (NYT)

2. More offline and off-phone data collection (Pew NPORS, SSRS, NORC)

3. Mixed-mode samples (promising, but not yet popular among public pollsters)

---
class: center, middle, inverse

# Solution 1: Less biased polls!
--

# or

--
# Solution 1: Less biased polls?

---
class: center, middle, inverse

# "Solution" 2: Let the aggregation model debias the polls

---
## Case study: _Economist_ model

#### i. Latent state-level vote shares evolve as a random walk over time

#### ii. Polls are weighted by their historical error *and bias*
* Based on past relationship between a pollster's lagged historical bias and performance of the aggregate

#### iii. Polls are observations with constant random effects to "debias" based on:
* Pollster firm (so-called "house effects")
* Poll mode
* Poll population

#### iv. Polls are also adjusted for potential partisan non-response
* Each poll has a covariate for whether it weights by party registration or past vote
* Effect is allowed to change over time
* Adjusts for biases that remain AFTER removing the other biases

---

## Notable improvements!

---
class: center, middle

# In 2016...

## ... But not 2020

---
class: center, middle

# The problem with solution 2:

# 1. Pollsters change their methods

# 2. Not all adjustments work

---
class: center, middle, inverse

# Solution 3: Conditional forecasting!

---
class: center, middle

## Solution 3: Conditional forecasting!

--
### - Present aggregates assuming some amount of polling _bias_.

--
### - As a way to explain to readers how bias enters the process of polling

--
### - And what happens to forecasts if bias _now_ does not follow historical distributions

---

## Conditional forecasting:

--
.pull-left[
## 1. Debias polls

<img src="figures/conditional_forecasting_one.png" width="80%" />
]

.pull-right[
## 2. Rerun simulations

<img src="figures/conditional_forecasting_two.png" width="80%" />
]

---

# 2. Rerun simulations
<img src="figures/conditional_forecasting_two.png" width="80%" />

---

# 2. Rerun simulations

### Advantage: leaves readers with a much clearer picture of possibilities for election outcomes _if past patterns of bias aren't predictive of bias now_ (2016, 2020)

---
class: center, middle

# We will see if this helps...

---
class: center, middle, inverse

# Further questions:

---
# What if that doesn't work?

### 2022 a critical test: does surveys get better or stay the same — or do they get worse?
### What if the DGP remains biased?
### What if the quality of the average poll continues to fall?

---

### Can we trust polls to be precise in close elections?
### If not, what are they good for?

---
class: center, middle

# How Polls Work and Why We Need Them

---

.pull-left[
<img src="figures/cover.jpg" width="90%" />
]

.pull-right[

# Thank you!

### _STENGTH IN NUMBERS_ is available now.

**Website: [gelliottmorris.com](https://www.gelliottmorris.com)**

**Twitter: [@gelliottmorris](http://www.twitter.com/gelliottmorris)**

### Questions?

]

---

_These slides were made using the `xaringan` package for R. They are available online at https://www.gelliottmorris.com/slides/_