(1) Concept Review

(2) Variables and Measures

(3) Variables

• Validity

• What is the “problem” they are trying fix?

• What are some descriptive claims implied by the “problem”

• What do we need to “check” these claims?

• What concepts? Definition?

Concepts: define our terms in a way that is transparent and can be used systematically. If concepts are opaque or idiosyncratic \(\to\) STOP! (board)

Variables: translate concepts into something that we can (in principle) observe. If variables do not correspond to the concept / correspond to other concepts \(\to\) STOP!

Measurement: devise transparent and systematic procedures with known uncertainty to variable in specific cases. If procedure is opaque, suffers from bias, has high uncertainty \(\to\) STOP!

“Misinformation is know it when I see it”:

• If I want to show that misinformation has gotten worse… no transparent and systematic definition let’s me label content in line with my claim \(\to\) fails weak severity

“Misinformation is factually inaccurate information OR whatever is on social media”:

• If I want to show misinformation has gotten worse… absence of systematic definition (loophole for social media) leads me to find more misinformation whenever there is more social media (even if social media content were truthful) \(\to\) fails weak severity

What is misinformation?

• false/factually inaccurate information that may be shared intentionally or unintentionally

What is disinformation?

• intentional dissemination of known false/factually inaccurate information

How can we observe misinformation in a way that lets us evaluate whether:

“Misinformation has become more widespread in recent years.”

How can we observe disinformation in a way that lets us evaluate whether:

“Disinformation has become more widespread in recent years.”

• this points us toward variables and measures

variable: A measurable property of cases that corresponds to a concept or part of a concept and can potentially take on different values across cases and time (it varies across cases).

• something we could observe in principle
• Chosen to indicate membership in category/presence of attribute (concepts)
• take on values for cases at a specific point in time
• Variation across cases and/or over time.
• General (e.g., “fraction of shared news that is false”, not “fraction of shared news on Facebook in Canada in 2024 that was false.”)

measure: A procedure for determining the value a variable takes for specific cases through observation.

• Measures aim to determine the values a variable takes for some cases
• Target some some specific cases we want to know about (e.g., a procedure for estimating the fraction of shared news on Facebook in Canada in 2024 that was false.)

NOTE: whats the difference?

Variables can be more or less precise: “level of democracy” vs “ratio of price per area of ads vs classified ads in printed newspapers”

• precise variables may look like a measure.
• but measures always say how we see something

What is the tallest mountain on Earth?

A descriptive question:

What is the tallest mountain on the North Shore?

We need to:

• define a the concept of “tall”
• create a variable that matches that definition
• develop a procedure to obtain values for mountains on the North Shore (measure)

Concept: tallness (of a mountain)

Elevation (distance from peak to sea level)

\(\xrightarrow{}\) Variable:

Vertical distance in meters from mean sea level to the top of the peak

\(\xrightarrow{}\) Measure:

Use difference in barometric pressure at Burrard Inlet and peak to calculate difference in elevation

Are you going to climb the mountain? Prominence might be a better concept of height.

Concept: tallness (of a mountain)

the elevation of a summit relative to the highest point to which one must descend before reascending to a higher summit

\(\xrightarrow{}\) Variable:

Vertical distance in meters from top of the peak to lowest contour line surrounding it and no other higher peaks.

\(\xrightarrow{}\) Measure:

Satellites using radar interferometry create topographical maps; computer algorithm to find lowest contour

Different concepts \(\to\) different variables

Different variables \(\to\) different measures

Different Answer:

• Using elevation from sea level, West Lion is taller
• Using prominence, Seymour is taller.

Claim: “Canadian exposure to misinformation has increased in recent years.”

• What concepts do we need to define?

Claim: “Canadian exposure to misinformation has increased in recent years.”

Concept: exposure to misinformation

Variable: Proportion of people who believe it is hard for them to distinguish between true and false information.

Measure: Ask random sample of Canadians to answer whether they find distinguishing true vs. false information is “harder”, “easier”, or “unchanged” now compared to 3 years ago.

• What values can this variable take? What could go wrong here? (Any risk of incorrectly accepting claim ?)

Claim: “Conservative Twitter/X users are exposed to more misinformation.”

• What concepts do we need to define?

Claim: “Conservative Twitter/X users are exposed to more misinformation.”

Concept: falseness of a message

Variable: rating of a statement on a scale of (True, Mostly True, Half True, Mostly False, False, Pants on Fire)

Measure: Ratings of Fact-checked statements on PolitiFact.com. Methodology Here.

• What values can this variable take? What could go wrong here? (Any risk of incorrectly accepting claim ?)

Claim: “Conservative Twitter/X users are exposed to more misinformation.”

Concept: message exposure

Variable: Number of tweets posted by “elite” (e.g., politicians, bureaucrats, famous personalities, advocacy groups, and media organizations) accounts followed by the user

Measure: Twitter/X API to find follows, tweets.

• What values can this variable take? What could go wrong here? (Any risk of incorrectly accepting claim ?)

Concept: exposure to misinformation

Variable: The average falseness of messages by elite (e.g., politicians, bureaucrats, famous personalities, advocacy groups, and media organizations) X users followed by a person, weighted by the number of Tweets made by each elite.

• What could go wrong here?

Check your “exposure to misinformation” here!

Claim: “Conservative Twitter/X users are exposed to more misinformation.”

Concept: political ideology

Variable: A score between -1 (liberal) to 1 (conservative) based on the ideology of accounts a user follows on Twitter.

Measure: Elite accounts labeled with known ideology; users scored by ideology of accounts they follow.

• What values can this variable take? What could go wrong here? (Any risk of incorrectly accepting claim ?)

Different issues:

• variables may not capture the concept we want.
• variables may capture OTHER concepts, as well.
• measures may be poor.

Even if we develop a concept that is transparent and systematic…

variables may not correspond to the concept

This may mean…

• at best: what we observe is irrelevant.
• at worst: regardless of measurement procedure, we are going to find claim to be true, even when it is not (fails weak severity)

validity: Degree of “fit” between a variables the concept the variable is intended to capture.

• When a variable “captures” or “maps onto” the concept we want to observe in \(\to\) “validity”
• When a variable “captures” or “maps onto” other concepts we do not want to observe \(\to\) lack of “validity”
• lack of validity may \(\to\) make incorrect conclusions

In our discussion: some of the concerns were about measurement:

How do you know if the problem is with validity:

even if we are able to perfectly observe (measure) something, we think still don’t capture the concept from the claim.

“State of Bihar isn’t the most politically corrupt in India”

Concept: Political Corruption or “the use of power by government officials for illegitimate private gain”

Variable: Fraction of political officeholders in a place prosecuted for corruption

Measure: Match criminal court defendants in corruption prosecutions to list of politicians.

• What factors influence what we’d see for this variable?

Concept: Political Corruption or “the use of power by government officials for illegitimate private gain”

Variable: Fraction of political officeholders in a place prosecuted for corruption

Measure: Match criminal court defendants in corruption prosecutions to list of politicians.

Problems

• Places with lots of corruption do not prosecute corruption
• Places with low corruption successfully prosecute corruption
• Places low corruption have stricter corruption laws.
• Yet measure will give correct values for the variable

(board)

Claim: “The risk of being a victim of a violent crime is less in Canada than the United States”

Variable: Number of violent crimes

• What factors influence what we would see with this variable?
• Would using this variable permit us to find the claim wrong? (board)

Where do validity problems come from?

• failure to correspond to the concept

• attributes other than our concept affect what we observe.

  • sometimes this is due to wrong level of measurement.

Stats Canada found that in 2025: 47% of Canadians said “it was … more difficult than it was three years earlier to distinguish between true and false information”

Can we conclude that “misinformation is getting worse”?

Claim: “Canadian exposure to misinformation has increased in recent years.”

Concept: exposure to misinformation

Variable: Proportion of people who believe it is hard for them to distinguish between true and false information.

Measure: Ask random sample of Canadians to answer whether they find distinguishing true vs. false information is “harder”, “easier”, or “unchanged” now compared to 3 years ago.

• Do you have validity concerns here?

Measuring exposure to misinformation from political elites on Twitter

Concept: exposure to misinformation

Variable: The average falseness of messages by elite (e.g., politicians, bureaucrats, famous personalities, advocacy groups, and media organizations) X users followed by a person, weighted by the number of Tweets made by each elite.

• Do you have validity concerns here?

Check your “exposure to misinformation” here!

Claim: “Conservative Twitter/X users are exposed to more misinformation.”

Concept: political ideology

Variable: A score between -1 (liberal) to 1 (conservative) based on the ideology of accounts a user follows on Twitter.

Measure: Elite accounts labeled with known ideology; users scored by ideology of accounts they follow.

• Do you have validity concerns here?

It is not enough to just imagine that there might be a validity problem:

• either we have strong reasons to think there are obvious problems
• or we need to suggest what we should observe that would rule out the validity concern
• evidence where possible problems with validity are taken seriously and ruled out with other evidence is stronger (survived more attempts to find errors)

1. Distinguish between Variables and Measures

• variables are what we could observe about cases that matches the concept
• measures are procedures to find values of variables for specific cases (data collection)

1. Variables:

• different levels of measurement
• variables may lack validity: even if data perfectly captures what we want to observe, our conclusions are flawed
• lack of validity may lead to failure to meet weak severity