Sampling Error

• Review
• Random Sampling Error and Severity
• Sampling Error vs. Measurement Error
• Non-citizen Voting

Was Charlie Kirk assassination a “wake-up call”?

• Some claim that “concern about political violence among Americans increased after the assassination”

Key terms:

population: full set of cases (countries, individuals, etc.) we’re interested in describing

sample: a subset of the population that we observe and measure

inference: description of the (unmeasured) population we make based on the (measured) sample

and there is uncertainty about what is true about the population, because we only measure a sample

The population:

All American adults (18 years of age or older)

The sample:

\(1381\) (June 2025), \(1477\) (September 2025) Americans 18+ “contacted … by text or online. Probability-based sampling frames include random-digit-dialing cell phone sample plus cellphone sample based on billing address.”

The inference:

Americans say politically-motivated violence is a major problem: - 73% (\(\pm 2.3\)%) in June 2025 - 77% (\(\pm 2.1\)%) in September 2025

random sampling: sampling cases from the population in a manner that gives all cases an equal probability of being chosen.

This procedure creates samples that:

• give unbiased inferences about the population (regardless of sample size) (no sampling bias)
  • unbiased in that, across all samples, on average the sample averages are the same as the population average
• has random sampling errors with a known size: produces known uncertainty (described by the field of statistics)

sampling error:

The difference between the value of the measure for the sample and the true value of the measure for the population

\[\mathrm{Value}_{sample} - \mathrm{Value}_{population} \neq 0 \xrightarrow{then} \mathrm{sampling \ error}\]

• Just like measurement error, there are two types: one that is bias and one that is random
• Sampling error is a kind of measurement error. If our measure requires inference about a population, sampling error is measurement error.

sampling error:

When thinking about random sampling error and sampling bias, we should think about…

the sampling distribution: See here

• the results (e.g., sample average) from all possible samples we could have gotten, using this sampling procedure.
• we only ever get the result from the one sample we draw, but can imagine the results could have been different (counter-factually)
• it is not something we actually know, except in simulations

The inference:

Americans say politically-motivated violence is a major problem:

• 73% (\(\pm 2.3\)%) in June 2025
• 77% (\(\pm 2.1\)%) in September 2025 (after Kirk assassination)

Can we say concern about politically-motivated violence has gone up?

• What might a skeptic say?

If we assume random sampling, no sampling bias (or at least, the same sampling bias):

Skeptic might say:

“It could be that concern about political violence remained at 73%, and the increase is due to random sampling error!”

Random sampling always involves random sampling error \(\to\) uncertainty

How might interpretation of evidence for claims be changed due to this random sampling error?

• We want to know whether we find evidence in support of some claim, EVEN IF THAT CLAIM WERE FALSE, due to random sampling error (by chance).
• We want to know “how severe” is this evidence? How likely could it lead us to wrong conclusions, due to chance?

(board, contrast w/ bias)

Unlike sampling bias, random sampling error \(\to\) a knowable uncertainty:

When sample size (\(n\)) \(\uparrow\)…

• random errors \(\downarrow\) in size
• sampling distribution \(\to\) approximately normal/“bell-shaped”

See here

One approach to assess severity is:

hypothesis tests: compare an “alternative” (claim is true) versus a “null” hypothesis (claim is false). Tell us probability of observing data for a claim by chance, if the claim were false. (error probability)

alternative hypothesis (claim is correct)

null hypothesis (claim is incorrect)

hypothesis test asks: “how likely are we to see this evidence for the claim, assuming that the claim is false”

• we assume the null hypothesis is right \(\to\) true concern is 73%
• we assume a random sampling procedure \(\to\) take random samples of \(1477\)
• with these two assumptions, we “make” the sampling distribution (in stats class, you plug into formulas)

We can imagine truth is 73% concerned about political violence; generate thousands of samples of size \(1477\); record the results

If true concern about political violence is 73%, by chance we see 77% concern or more with probability of 0.000172.

hypothesis test asks: “how likely are we to see this evidence, assuming that the claim is false”

• \(p\)-value: probability of accepting claim (rejecting the null hypothesis) due to chance, if it were false
• probability of incorrectly concluding that concern about political violence increased, due to random sampling error. (error is 0.0172% of the time)

\(p\) values:

• it is an error probability (probability of incorrectly accepting the claim)
• it is advertised severity of evidence (against random sampling error)
• \(p\) value of \(< 0.05\) is not magic. \(\to\) “could observe this 1 in 20 times if null hypothesis is true/claim is wrong”
• only meaningful if assumptions are correct (here: no bias, random sampling)

If we assume random sampling, no sampling bias:

Skeptic might say:

“It could be that concern about political violence remained at 73%, and the increase is due to random sampling error!”

null hypothesis: there has been NO increase in concern (claim is wrong)

But we can respond:

“We saw 77% of people concerned about political violence. If you were right, we’d see this only 0.0172% of the time

What is a margin of error/confidence interval, then?

• We choose an error probability that is “acceptable” \(\to\) 1% (how severe do we want our evidence to be?)
• Compare observed data (77% concerned about political violence) against all possible null hypotheses (0% to 100%).

If true concern about political violence is 75%, by chance we see 77% concern or more with probability of 0.036425. Error rate of 3.6425\(\%\) \(> 1\%\) \(\to\) cannot reject 75%.

What is a margin of error/confidence interval, then?

• We choose an error probability that is “acceptable” \(\to\) 1% (how severe do we want our evidence to be?)
• Compare observed data (77% concerned about political violence) against all possible null hypotheses (0% to 100%).
• Range of values for “null hypotheses” we cannot reject with chosen error rate.
• Still assumes no sampling bias and use of random sampling

• 77% concerned \(\to\) reject truth being outside of range from 74.2% to 79.8% with error rate of 1% or less

What is a “margin of error” or “confidence interval”:

• If survey says it is \(50\% \pm 2\%\) with \(95\%\) confidence

• The true value is in the range of \(48-52\) with \(5\%\) error probability

If we have random sampling errors:

• we can put a number on probability of making an error \(\to\) quantify severity

If we have sampling bias…

• we’d need to think about: direction and magnitude to see if it could lead to weak severity

Surveys said voters preferred Biden to Trump by \(8.4\) percent. Biden actually won by only \(4.5\) points.

What might explain this?

It depends: if this is going on, then sampling bias

It depends: if this is going on, then sampling bias

It depends: if there are “shy” Trump voters, then measurement bias.

• when is there sampling error vs measurement error?
• what is sampling error a measurement error?

In addition to winning the Electoral College in a landslide, I won the popular vote if you deduct the millions of people who voted illegally

— Donald J. Trump (@realDonaldTrump) November 27, 2016

This “problem” used to justify policy changes:

• requirements to show proof of citizenship with ID when voting
• collect records on voter files at a national (not state) level
• federal investigations and prosecutions of alleged voter fraud
  
• threats against voter turnout NGOs

Claim: Widespread voter fraud: “14% of non-citizens voted”

Concept

• Non-citizen; voter fraud \(\xrightarrow{}\)

Variable(s)

• Fraction of non-citizens who voted \(\xrightarrow{}\)

Measure(s)

• how do we measure non-citizen voting? \(\xrightarrow{}\)

Answer?

Richman et al:

• two large, random sample of adult Americans in 2008, 2010 (~33,000 and 55,400 people)
• identify a sample of non-citizens: respondents who indicate on the survey that they are non-citizens (\(N = 489\), or about \(1\%\) of people in 2010)

Richman et al:

• two large, random sample of adult Americans in 2008, 2010 (~33,000 and 55,400 people)
• identify a sample of non-citizens: respondents who indicate on the survey that they are non-citizens (\(N = 489\), or about \(1\%\) of people in 2010)
• count who among the “non-citizen sample” voted (\(13\)) (validated voting measure - matched to public voter file)
• conclude that 3.5% of non-citizens voted in 2010 (~700k), up to 14.7% in 2008 (~2.8 million people)

So, was Trump right?

Discuss: Do you find this persuasive? Why or why not?

Problem One: Individual Measurement Error

The political scientists who run the CCES survey point out:

• Citizenship question suffers from (low) measurement error.
• Those surveyed in both 2010 and 2012: \(99.7\%\) gave the same answer on citizenship, \(0.19\%\) went from “non-citizen” to “citizen” (maybe true), \(0.11\%\) went from “citizen” to “non-citizen” (definitely false)
• measurement error: misclassifies \(0.1\%\) of people (1 out of a 1000)

• to the board

Problem Two: Sampling Error

measurement error of individuals as citizens/non-citizens, leads Richman et al to sample of “non-citizens” that include citizens and non-citizens:

• citizens \(\gg\) non-citizens \(\to\) many more citizens who are misclassified as “non-citizens”
• We have sampling error… the sample does not reflect the population Richman et al want to make inferences about.
• It could be that the “non-citizen” voting is driven entirely by voting among citizens who clicked the wrong button on the survey.

Nobody who consistently reports being a non-citizen votes.

Measurement Error (of individuals’ citizenship)

\(\Downarrow produces\)

Sampling Error (sample that should be of non-citizens includes citizens)

\(\Downarrow produces\)

Measurement Error (about the population of non-citizens)

\(\Downarrow\)

authors make incorrect inference that hundreds of thousands of non-citizens vote illegally.

1. Richman et al/ Trump want to make claims about the population of all non-citizens. They use a sample.
2. The sample is generated based on individual survey responses to a questions that are wrong for \(0.1\%\) of individuals (individual measurement error)
3. This generates sampling bias as the sample systematically includes citizens in sample of “non-citizens”
4. Because Richman et al/ Trump are using the sample to make inferences about the population of non-citizens, the sampling bias produces measurement bias (of population): systematically over-estimating non-citizen voting.

When is sampling error not measurement error?

• If you are not trying to make an inference about a population, sampling error doesn’t generate measurement error.

People try to persuade you there is a problem in need of fixing:

• illegal actions by migrants
• increasing crime
• levels of political violence
• declines in democracy

Is this problem real?

• Concept \(\to\) is definition systematic/transparent?
• Variable/validity \(\to\) Does variable capture concept? Does choice of variable lead us to support claim even if it is false?
• Measure/measurement error \(\to\) Do measurement strategies lead to error? What kind of error? Does this error lead us to support claim even if it is false?