(1) Recap

(2) Validity

• recognizing it
• examples

(3) Measurement Error

• Bias
• Random

In 2020: Breonna Taylor

In 2024: Laken Riley

Two “diagnostic claims”

1. “There is racial bias in the use of police violence against Black Americans.” (EXAMPLE 1)
2. “There is widespread violence perpetrated by illegal immigrants.” (EXAMPLE 2)

• Activists point to highly publicized events as evidence for their claims.

• Are these individual deaths compelling evidence for the descriptive claims? Why/Why not?

We want our evidence to:

• be capable of showing claim to be wrong (weak severity)
• stand up to multiple checks on where it could be wrong (strong severity):
• pass tests that can EASILY find claim to be wrong.

We want to be sensitive to:

• what properties of evidence \(\to\) absence of weak severity
• what are the failure points that we can check \(\to\) strong severity

variable: A measurable property of cases that corresponds to a concept or part of a concept and can potentially take on different values across cases and time (it varies across cases).

• something we could observe in principle
• Chosen to indicate membership in category/presence of attribute (concepts)
• take on values for cases at a specific point in time
• Variation across cases and/or over time.
• General (e.g., “fraction of shared news that is false”, not “fraction of shared news on Facebook in Canada in 2024 that was false.”)

measure: A procedure for determining the value a variable takes for specific cases through observation.

• Measures aim to determine the values a variable takes for some cases
• Target some some specific cases we want to know about (e.g., a procedure for estimating the fraction of shared news on Facebook in Canada in 2024 that was false.)

“Country X isn’t the most politically corrupt”

Concept: Political Corruption or “the use of power by government officials for illegitimate private gain”

Variable: Fraction of political officeholders in a place prosecuted for corruption

Measure: Match criminal court defendants in corruption prosecutions to list of politicians.

validity is the extent to which variable (what we hope to observe) matches the concept:

lack of validity arises when:

• what we observe in the variable is partly or wholly due to a different concept than intended

  • must have explanation for why what we observe is due to some other concept.

• what we observe in the variable only partially captures the concept

  • must have explanation for why important part of concept does not influence the what we would observe

“The risk of being a victim of a violent crime is less in Canada than the United States”

Concept: risk of violent crime “likelihood of experiencing a crime that involves a threat or damage to one’s body”

Variable: Number of violent crimes

Measure: Tabulations of violent crimes created by police agencies

• Any problems with validity here? (Board: make a table!)

1. “There is racial bias in the use of police violence against Black Americans.” (EXAMPLE 1)

For every 10,000 black people arrested for violent crime, 3 are killed

For every 10,000 white people arrested for violent crime, 4 are killed

I'm going to keep tweeting this until someone can explain to me how this is possible if there is truly pervasive racial bias in policing

— Leonydus Johnson (leave/me/alone) (@LeonydusJohnson) June 1, 2020

• Claim? Concepts? Variables? Measures?

Claim: “Racial bias in policing is not pervasive”

Concept: racial bias in policing defined as racial disparity in police use of force in excess of “reasonable” considerations such as “objective threat” posed by suspect

Variable: difference by race in number of people killed by police per persons arrested for violent crimes

Measure: count press-reported police-shootings by race, FBI data on arrests by crime-type and race

Discuss: Does this variable have validity?

BOARD

1. Variable “overlaps” multiple concepts.
2. Different factors cause or produce what we observe!

(board)

lack of validity:

• requires a story about why what we would see as values of the variable are the result of something other than the concept we are interested in.

lack of validity:

Just because we have a story about why variable lacks validity:

• does not mean we are right: \(\to\) must collect new evidence capable of seeing if this story is false.
• “checking” this story \(\to\) “strong severity”

• Test: do police discriminate against Black Americans in stops/charges?

lack of validity:

Just because variable lacks validity:

• does not necessarily mean absence of weak severity
• if lack of validity makes it harder to accept claim/easier to reject claim: evidence for the claim passed a hard test

Claim: “Illegal immigrants commit murder at higher rates”

Concept: “illegal immigrant”, “murder”

Variable: correlation between fraction of people in a city who are undocumented immigrants, fraction of people in a city who are murderers

Measure: Pew Research estimate of undocumented migrants, FBI data on murders per capita

undocumented and murderers in US cities (each dot is a city)

• Does this have validity?

One common type of validity problem:

Ecological Inference and Ecological Fallacy

ecological inference: using correlations of attributes at aggregate levels (e.g. country, province, city) to making inferences about individual behaviors. (undocumented migrants in a city, murderers in a city)

• If assumptions correct, this can work, but usually not.\(^*\)

One common type of validity problem:

Ecological Inference and Ecological Fallacy

ecological fallacy

• even if we have perfect data on aggregate variables (e.g. undocumented migrants, murders), aggregate patterns \(\to\) incorrect conclusions about individual patterns
• relationship between these attributes look different at individual vs aggregate level, because individual behaviors differ across and within cities/provinces/countries

“There is widespread violence perpetrated by illegal immigrants.” (EXAMPLE 2)

Light et al 2020 investigate claim: “undocumented migrants are prone to violent crime”

concepts: undocumented, violent criminals

variable: conviction rates for violent crime (homicide, assault, robbery, sexual assault) for US-born citizens, legal immigrants, undocumented immigrants

measure: individual crimes listed in arrests in the Texas Computerized Criminal History database, individual immigration status as determined by DHS and ICE using biometrics database, numbers of undocumented migrants using Census data

Not the end of the story: Kennedy et al at Center for Immigration Studies dispute these findings:

Complaints focus on:

• how does Texas identify undocumented migrants in arrest and prison records?
• how do DHS/ICE identify undocumented migrants?

Kennedy et al, argue:

• It takes time for undocumented immigrants in custody to be identified.

  • \(\to\) undercounting of arrested undocumented
  • \(\to\) more undercounting in recently arrested

• Only people in custody for longer periods of time for serious crimes likely to be thoroughly checked:

  • \(\to\) undercounting is lower/minimal for homicide convictions
  • \(\to\) need to use DHS and Texas prison (TDCJ) checks on migration status

• Are Kennedy et al pointing to a lack of validity in the “undocumented” variable?

• By “fixing” measurement procedures, conclusions reversed.
• This is a claim about measurement error

Validity is about link between variable and concept

Measurement Error is about link between measure and variable.

measurement error

is a difference between the observed value of a variable for a case (produced by the measurement procedure) and the true value of the variable for that case.

\[\mathrm{Value}_{observed} - \mathrm{Value}_{true} \neq 0 \xrightarrow{then} \mathrm{measurement \ error}\]

If what we observe is different from the true value for a case (difference is not 0), then there is measurement ERROR

What is the incidence of sexual misconduct defined here at UBC?

Let’s say a variable is the number of breaches of Sexual Misconduct Policy in a given year.

Measure: Reporting from the UBC Investigations Office.

What is the incidence of sexual misconduct defined here at UBC?

Let’s say a variable is the number of breaches of Sexual Misconduct Policy in a given year.

Measure: Reporting from the UBC Investigations Office.

That implies \(15\) incidents in 2025-2026 Academic Year (last available data).

45 reports \(\to\) 26 investigations \(\to\) 20 completed investigations \(\to\) 15 breaches found

• Is this observed value too high? too low? correct? Why?

\[\mathrm{Sexual \ Misconduct }_{observed} - \mathrm{Sexual \ Misconduct}_{true} \neq 0\]

\[\xrightarrow{then} \mathrm{measurement \ error}\]

• Most likely \(\mathrm{Sexual \ Misconduct }_{observed} - \mathrm{Sexual \ Misconduct}_{true} < 0\)

Two varieties of measurement error

• bias/systematic measurement error
• random measurement error

Differ in the patterns of \(\mathrm{Value}_{observed} - \mathrm{Value}_{true}\) that we see.

Measures may suffer from both.

bias or systematic measurement error: error produced when our measurement procedure obtains values that are, on average, too high or too low (or incorrectly labelled) compared to the truth.

• Key phrase is “on average”: error is not a one-off fluke, will happen systematically even if you repeat the measurement procedure.
• can have an upward (observed value too high) or downward (observed value too low) bias
• can occur for nominal/categorical values: e.g., systematically misclassifying undocumented migrants as legal
• not “politically” biased

Kennedy et al argue that Light et al’s measurement procedures lead to, on average:

\[\mathrm{Migrant \ Homicide \ Rate }_{observed} - \mathrm{Migrant \ Homicide \ Rate}_{true} < 0\]

\[\xrightarrow{then} \mathrm{measurement \ bias}\]

Though, this debate over measurement isn’t over.

bias different in different subgroups

random measurement error: errors that occur due to random features of measurement process or phenomenon. So even if observed values are sometimes wrong, they are, on average, correct

• Due to chance, we get values that are too high or too low
• There is no tilt one way or another (no bias): “noise”
• random here means: errors unrelated to true value; errors equally likely to be over- or under- estimate.
• In aggregate, values that are “too high” are balanced out by values that are “too low” compared to the truth

Variable: relative change in COVID-19 infections

Measure: “Composite wastewater influent is collected over a 24-hour period from wastewater treatment plants (WWTPs). Samples are collected 2-3x per week at each WWTP and are transported by the BCCDC PHL for analysis. Wastewater samples are concentrated by ultracentrifugal filtration, nucleic acids extracted and SARS-CoV-2 envelope gene (E gene) is detected by real-time quantitative polymerase chain reaction (RT-qPCR).”

Day-to-day variation in:

• wastewater volume (e.g. rain, snowmelt, showering, dishwashing)
• fecal matter (e.g. diet, exercise, other diseases)

can lead to errors in measurement, but these errors…

• probably unrelated to amount of true rate of COVID infections
• probably equally likely to lead to over and under count
• likely cancel out in the long run.

Lack of Validity:

• story for why what we would observe does not match concept (unrelated to data-collection procedures)
• may or may not lead to failing weak severity
• ecological inference/fallacy is a common problem

Measurement Error

• observed values differ from true values of the variable
• Bias/systematic measurement error
• Random measurement error