February 11, 2021


(1) Measurement Error

  • Bias/Systematic
  • Random
  • Sources and Solutions

(2) Sampling Error

  • Sampling Bias
  • Random Sampling Error

Measurement Error

Measurement Error

measurement error

is a difference between the true value of a variable for a case and the observed value of the variable for that case produced by the measurement procedure.

\[\mathrm{Value}_{observed} - \mathrm{Value}_{true} \neq 0 \xrightarrow{then} \mathrm{measurement \ error}\]

Measurement Bias

measurement bias or systematic measurement error: error produced when our measurement procedure obtains values that are, on average, too high or too low (or, incorrect) compared to the truth.

  • Key phrase is “on average”: error is not a one-off fluke, will happen systematically even if you repeat the measurement procedure.
  • can have an upward (observed value too high) or downward (observed value too low) bias
  • not “politically” biased
  • bias might not be the same for all cases or different across subgroups
    • example: economic evaluations and partisanship in surveys

Random Measurement Error

random measurement error: errors that occur due to random features of measurement process or phenomenon and the values that we measure are, on average, correct

  • Due to chance, we get values that are too high or too low
  • May be lots of errors
  • There is no systematic tilt one way or another (no bias)
  • In aggregate, values that are “too high” are balanced out by values that are “too low” compared to the truth

Example: Facebook and Hate Crime

Mueller and Schwarz (2018) ask:

Is social-media hate speech related to real-world violence?

  • Are there higher levels of anti-refugee violence in places with more social media access in weeks with more social media anti-refugee hate speech?
  • Address this question in the context of Germany (2015-2017)

Example: Facebook and Hate Crime

Need three concepts/variables/measures:

  1. Anti-refugee violence
  2. Anti-refugee rhetoric on Facebook
  3. Exposure to Facebook

Example: Facebook and Hate Crime

concept: Anti-refugee Violence

variable: Attacks against refugee persons and property per 10k refugees

measure: (for each week in each municipality)

Example: Facebook and Hate Crime

concept: Anti-refugee rhetoric on social media

variable: Number of anti-refugee posts on Facebook per week


Example: Facebook and Hate Crime

Example Facebook posts:

Example: Facebook and Hate Crime

concept: “Exposure to Facebook”: persons who have an active Facebook account

variable: Active facebook users in a municipality per 10k people.

measure: Followers of Nutella Germany on Facebook (who share their location information) per capita

  • out of ~\(63,000\) Nutella followers, only ~\(22,000\) shared their location

Example: Facebook and Hate Crime

In groups, discuss each of these measures:

  • Could they suffer from measurement bias? If yes, why?
  • Could they suffer from random measurement error? If yes, why?

Systematic Measurement Error/Bias


(\(1\)) Researcher subjectivity/interpretation: Researcher systematically biased in how she evaluates cases

  • e.g.: Resume experiments: HR professionals exhibit gender, racial bias.
  • explicit or implicit stereotyping

Systematic Measurement Error/Bias


(\(2\)) Obstacles to observation

  • social norms may discourage revelation of information; downward bias in “undesirable” phenomena
    • e.g. survey measure of racism or drug use \(\xrightarrow{}\) social desirability bias
  • incentives to hide/misrepresent: political actors have strategic reasons to conceal information from each other
    • e.g. police use-of-force encounter reports on “objective threat” of black suspects (Fryer 2019) (upward bias)
    • e.g. wealthy people may misrepresent assets to avoid taxation (downward bias)

Example: Immigration

If we surveyed Canadians and asked them:

“And would you support or oppose stopping all immigration into Canada?”

They can choose “oppose”, “support”, “neither support nor oppose”

Do you think this survey response would suffer from measurement bias?

Example: Immigration

List experiments