(1) Recap
- Evaluating Descriptive Claims
- Variables vs. Measures
- Validity
(2) Validity
- recognizing it
- examples
(3) Measurement Error
- Bias
- Random
October 2, 2025
We want our evidence to:
We want to be sensitive to:
Concepts not transparent/systematic \(\xrightarrow{\xcancel{weak \ severity}}\)
Variable does not map onto concept (lack of validity) \(\xrightarrow{\xcancel{weak \ severity}}\)
Procedure does not return the true values (measurement error) \(\xrightarrow{\xcancel{weak \ severity}}\)
A measurable property of cases that corresponds to a concept or part of a concept and can potentially take on different values across cases and time (it varies across cases).
A procedure for determining the value a variable takes for specific cases based on observation.
validity is the extent to which variable (what we hope to observe) matches the concept:
lack of validity arises when:
what we observe in the variable is partly or wholly due to a different concept than intended
what we observe in the variable is only partially captures the concept
“Country X isn’t the most politically corrupt”
Concept: Political Corruption or “the use of power by government officials for illegitimate private gain”
Variable: Fraction of political officeholders in a place prosecuted for corruption
Measure: Match criminal court defendants in corruption prosecutions to list of politicians.
“The risk of being a victim of a violent crime is less in Canada than the United States”
Concept: risk of violent crime “likelihood of experiencing a crime that involves a threat or damage to one’s body”
Variable: Number of violent crimes
Measure: Tabulations of violent crimes created by police agencies
For every 10,000 black people arrested for violent crime, 3 are killed
— Leonydus Johnson (leave/me/alone) (@LeonydusJohnson) June 1, 2020
For every 10,000 white people arrested for violent crime, 4 are killed
I'm going to keep tweeting this until someone can explain to me how this is possible if there is truly pervasive racial bias in policing
Claim: “Racial bias in policing is not pervasive”
Concept: racial bias in policing defined as racial disparity in police use of force in excess of “reasonable” considerations such as “objective threat” posed by suspect
Variable: difference by race in number of people killed by police per persons arrested for violent crimes
Measure: count press-reported police-shootings by race, FBI data on arrests by crime-type and race
Claim: “Racial bias in policing is not pervasive”
Concept: racial bias in policing defined as racial disparity in police use of force in excess of “reasonable” considerations such as “objective threat” posed by suspect
Variable: difference by race in number of people killed by police per persons arrested for violent crimes
Measure: count press-reported police-shootings by race, FBI data on arrests by crime-type and race
Even if we perfectly measure violent crime arrests by race, does variable match concept?
Claim: “Racial bias in policing against Black Americans is not pervasive.”
Concept: racial bias in policing defined as racial disparity in police use of force in excess of “reasonable” considerations such as “objective threat” posed by suspect
Variable: difference by race in number of people killed by police per persons arrested for violent crimes
If our alternative story of how this variable is generated (how are values produced) is correct, then fail weak severity: variable cannot show claim to be wrong
Variable “overlaps” multiple concepts.
Average propensity for violent crime
Racial bias in arrests/charges
Claim: “Illegal immigrants commit murder at higher rates”
Concept: “illegal immigrant”, “murder”
Variable: correlation between fraction of people in a city who are undocumented immigrants, fraction of people in a city who are murderers
Measure: Pew Research estimate of undocumented migrants, FBI data on murders per capita
undocumented and murderers in US cities (each dot is a city)
One common type of validity problem:
ecological inference: using correlations of attributes at aggregate levels (e.g. country, province, city) to making inferences about individual behaviors. (undocumented migrants in a city, murderers in a city)
One common type of validity problem:
ecological fallacy
lack of validity:
Light et al 2020 investigate claim: “undocumented migrants are prone to violent crime”
concepts: undocumented, violent criminals
variable: conviction rates for violent crime (homicide, assault, robbery, sexual assault) for US-born citizens, legal immigrants, undocumented immigrants
measure: individual crimes listed in arrests in the Texas Computerized Criminal History database, individual immigration status as determined by DHS and ICE using biometrics database, numbers of undocumented migrants using Census data
Not the end of the story: Kennedy et al at Center for Immigration Studies dispute these findings:
Complaints focus on:
Kennedy et al, argue:
It takes time for undocumented immigrants in custody to be identified.
Only people in custody for longer periods of time for serious crimes likely to be thoroughly checked:
is a difference between the observed value of a variable for a case (produced by the measurement procedure) and the true value of the variable for that case.
\[\mathrm{Value}_{observed} - \mathrm{Value}_{true} \neq 0 \xrightarrow{then} \mathrm{measurement \ error}\]
If what we observe is different from the true value for a case (difference is not 0), then there is measurement ERROR
What is the incidence of sexual misconduct defined here at UBC?
Let’s say a variable is the number of breaches of Sexual Misconduct Policy in a given year.
Measure: Reporting from the UBC Investigations Office.
What is the incidence of sexual misconduct defined here at UBC?
Let’s say a variable is the number of breaches of Sexual Misconduct Policy in a given year.
Measure: Reporting from the UBC Investigations Office.
That implies \(13\) incidents in 2024-2025 Academic Year (last available data).
39 reports \(\to\) 21 investigations \(\to\) 19 completed investigations \(\to\) 13 breaches found
\[\mathrm{Sexual \ Misconduct }_{observed} - \mathrm{Sexual \ Misconduct}_{true} \neq 0\]
\[\xrightarrow{then} \mathrm{measurement \ error}\]
Differ in the patterns of \(\mathrm{Value}_{observed} - \mathrm{Value}_{true}\) that we see.
Measures may suffer from both.
bias or systematic measurement error: error produced when our measurement procedure obtains values that are, on average, too high or too low (or incorrectly labelled) compared to the truth.
Kennedy et al argue that Light et al’s measurement procedures lead to, on average:
\[\mathrm{Migrant \ Homicide \ Rate }_{observed} - \mathrm{Migrant \ Homicide \ Rate}_{true} < 0\]
\[\xrightarrow{then} \mathrm{measurement \ bias}\]
Though, this debate over measurement isn’t over.
bias different in different subgroups
random measurement error: errors that occur due to random features of measurement process or phenomenon. So even if observed values are sometimes wrong, they are, on average, correct
Variable: relative change in COVID-19 infections
Measure: “Composite wastewater influent is collected over a 24-hour period from wastewater treatment plants (WWTPs). Samples are collected 2-3x per week at each WWTP and are transported by the BCCDC PHL for analysis. Wastewater samples are concentrated by ultracentrifugal filtration, nucleic acids extracted and SARS-CoV-2 envelope gene (E gene) is detected by real-time quantitative polymerase chain reaction (RT-qPCR).”
Day-to-day variation in:
can lead to errors in measurement, but these errors…