1. Recap
2. Before/After Assumptions
3. Beyond Before and After

Every way of using correlation as evidence for causality makes assumptions

• FPCI cannot be solved without assumptions
• With assumptions, can say confounding/bias is not a problem

Yesterday

Back in 2017…

Did Trump’s rhetoric affect anti-Muslim hate crimes?

We can make Before and After comparisons: What is it?

before and after: examine the change in \(Y\) in a single case (or group of cases) where \(X\) changes over time (from before to after)

Taking the same data from Feinberg, Branton, and Martinez-Ebers…

• we focus only on counties that ever had a Trump (Clinton) rally
• compare the month after the rally \((X = \text{Rally})\) to the month before the rally \((X = \text{No Rally})\)

Conditioning removes confounding by:

• identify possible confounding variables
• measure confounding variables
• relationship b/t \(X\) and \(Y\) for cases with similar value of confounding variables \(W\).

Design-based solutions (like Before and After) remove confounding by:

• selecting cases for comparison in order to eliminate many known/unknown as well as measurable/unmeasurable confounding variables.
• the nature of the comparison holds constant classes of confounding variables, not specific confounding variables.
• by a “class” we mean all confounding variables that share certain properties

How does it work?…

All confounding variables (affect whether a rally occurs; affect hate crimes) that are unchanging over time (before to) are held constant.

• because held constant, cannot produce confounding
• e.g., demographic features, political leaning, location/geography, long-term economic trends, 8chan white nationalists
• any variable that does not change in the time period of the comparison (in this case, two months) held constant
• does not matter if we can think of or even measure the confounders

Just like experiments and confounding, Before and After comparisons plug in for MISSING potential outcomes.

We assume \(\color{red}{\text{Hate Crimes}_{c,After}[\text{No Rally}]} = \\ \color{black}{\text{Hate Crimes}_{c,Before}[\text{No Rally}]}\)

That is: if \(X\) had not changed, \(Y\) would not have changed.

External Validity

• We can examine “real world” causes of interest
• BUT… limited set of cases (only find effects of cases with change in \(X\) over time)

Not quite as widely applicable as confounding… \(\to\) Higher, but not Highest external validity

Assumptions:

• assume that counterfactual potential outcomes of \(Y\) without \(X\) after \(X\) happens, same as factual \(Y\) without \(X\) before \(X\) happens
• equivalently: assume there are no variables \(W\) that affect \(Y\) and change over time with \(X\).

Any \(W\) that affects \(Y\) and changes with \(X\) will produce confounding even if it does not cause \(X\).

• this is a new source of confounding (result of our comparison)
• we eliminate confounding from unchanging variables, but open up possible new confounding from changing variables.

Must assume there are no variables \(W\) that affect \(Y\) and change over time with \(X\).

Why might this assumption fail?

• Did Trump rallies take place in places that are already trending toward having more hate crimes?
• Is it possible that Trump wanted to avoid controversy and waited to hold rallies in places until they had a month with a lower-than-usual number of hate crimes? (board)

• We can address these concerns by looking at longer-term trends…

Mostly flat trend; change when rallies occur

Mostly constant upward trend; no change when rallies occur

When does this assumption fail?

Over-time comparison, we can create confounding from variables that do not cause \(X\) to change, if they also change with \(X\) over time…

• Does rally change measurement, but not actual number of hate crimes? (Measurement bias)

• Are there are other changes over the same time-frame (change at the same time as \(X\), rallies)?

  • Less of a problem when comparing across very short time periods (fewer variables change)

Looking at the 2017 Truck attack: by DAY

Looking at the 2017 Truck attack: by MONTH

FBI data shows no change, compared to Anti-Defamation League \(\to\) “effect” is changing measurement of hate crimes?

Does easing restrictions on gun laws increase murders committed using guns?

• Some states in the US require all handgun purchasers to acquire a permit-to-purchase (PTP) license (LIKE CANADA).
• Only persons with a permit may purchase firearms
• In late 2007, Missouri eliminated its PTP requirement

Webster et al (2014) investigate:

• Did the removal of the PTP law increase firearms homicides in Missouri?

• Conditioning?: Lots of unique features of Missouri; no “otherwise similar” state.

Did the repeal CAUSE a change in murders using guns?

Holds all unique, unchanging characteristics of Missouri constant…

But, we have to assume that there is nothing else about Missouri that

1. changed around the same time as the PTP (gun control) repeal
2. and affected Firearms Homicides

(or more technically, assume that \(\color{red}{\text{Murders}_{MO,After}[\text{No Repeal}]} = \color{black}{\text{Murders}_{MO,Before}[\text{No Repeal}]}\))

No long-term trends, no effects on measurement,

no changes in crimes \(\to\) PTP repeal

Does this plot make it easier/harder to believe PTP repeal caused more murders? (DISCUSS)

Could be that other things were changing between 2007-2008 that confound relationship between PTP and Murders?

• Maybe an upward trend in long term?
• Maybe 2006-2007 was aberration, 2008 a return to trend?
• Did anything else happen in 2008?

It appears that \(\color{red}{\text{Murders}_{MO,After}[\text{No Repeal}]} \neq \\ \color{black}{\text{Murders}_{MO,Before}[\text{No Repeal}]}\)

Because other factors changing murders, regardless of repeal

What can we do to remove confounding from other variables that change over time, like…

• weather patterns (hot weather \(\to\) murders)
• global financial crises/economic shocks
• political events

• Another way to put this question is: what would the trend in gun murders in Missouri have been had there been no PTP repeal? What was the counterfactual trend?

We want to compare the actual trend in Missouri:

\(\begin{equation}\begin{split}\text{Trend}_{MO} ={} & \color{black}{\text{Murders}_{MO,After}[\text{Repeal}]} - \\ & \color{black}{\text{Murders}_{MO,Before}[\text{No Repeal}]}\end{split}\end{equation}\)

against the counterfactual trend in Missouri:

\(\begin{equation}\begin{split}\color{red}{\text{CF Trend}_{MO}} ={} & \color{red}{\text{Murders}_{MO,After}[\text{No Repeal}]} - \\ & \color{black}{\text{Murders}_{MO,Before}[\text{No Repeal}]}\end{split}\end{equation}\)

\(\small{\begin{equation}\begin{split} = {} & \overbrace{\{\text{Murders}_{MO,After}(\text{Repeal}) - \text{Murders}_{MO,Before}(\text{No Repeal})\}}^{\text{Missouri observed trend}} - \\ & \underbrace{\{\color{red}{\text{Murders}_{MO,After}(\text{No Repeal})} - \text{Murders}_{MO,Before}(\text{No Repeal})\}}_{\color{red}{\text{Missouri counterfactual trend}}}\end{split}\end{equation}}\)

• Before and After assumes the counterfactual trend is always 0

Many possible counterfactual trends…

Which counterfactual is right?

Which counterfactual is right?

We can’t know the counterfactual trend in Missouri…

but we can observe the trends in other states that did not change their gun purchasing laws (no change in Gun Control, \(X\)).

• We can plug in the \(\text{factual TREND}\) in an “untreated” case (no change in \(X\)) for the \(\color{red}{\text{counterfactual TREND}}\) in the “treated” case (where \(X\) did change).

Arkansas has a different history that Missouri, so there are differences that are unchanging between them.

But, if Arkansas experiences same regional economic, political, cultural, weather trends as Missouri, they likely share the same trends over time.

Then, we can plug in

\(\small{\begin{equation}\begin{split} = {} & \overbrace{\{\text{Murders}_{MO,After}(\text{Repeal}) - \text{Murders}_{MO,Before}(\text{No Repeal})\}}^{\text{Missouri observed trend}} - \\ & \{\underbrace{\text{Murders}_{AR,After}(\text{No Repeal}) - \text{Murders}_{AR,Before}(\text{No Repeal})\}}_{\text{Arkansas observed trend}}\end{split}\end{equation}}\)

Missouri/Arkansas different in 2007, but if Missouri had same trend (counterfactually) as Arkansas, what would we expect Murders to have done in 2008 w/out the repeal?

Missouri’s counterfactual trend is parallel to / same as Arkansas’s factual trend

With your neighbors, discuss: Do you believe this is evidence of causality? What confounding does this address? What confounding does it not address?

Like before and after, difference in differences comparisons are design based

• careful comparison rules out groups of confounding variables

What is it?

• Compare changes in “treated” cases before and after “treatment” to before and after changes in “untreated” cases

Why is it called difference in differences?

\(\small{\begin{equation}\begin{split} = {} & \overbrace{\{\text{Murders}_{MO,After}(\text{Repeal}) - \text{Murders}_{MO,Before}(\text{No Repeal})\}}^{\text{Missouri observed trend}} - \\ & \{\underbrace{\text{Murders}_{AR,After}(\text{No Repeal}) - \text{Murders}_{AR,Before}(\text{No Repeal})\}}_{\text{Arkansas observed trend}}\end{split}\end{equation}}\)

So:

• \(\mathrm{Difference \ 1} = Murders_{After} - Murders_{Before}\) gives us trend in murders in a \(State\)…
  • holding unchanging attributes of state constant (difference over time)
• \(\mathrm{Difference \ 2} = \mathrm{Difference \ 1}_{Missouri} - \mathrm{Difference \ 1}_{Arkansas}\) gives us change in murders in \(Treated\) over time, compared to trend in \(Control\)
  • holds changing attributes of both states constant (difference in trends)

How does it work?

• Hold constant unchanging attributes of cases (compare same case before and after “treatment”)
• Hold constant variables that change together over time in both “treated” and “untreated” cases

We don’t need to know/measure what these variables are!

Confounding Solved…

All confounding variables (affect whether a PTP repealed; affect firearms homicides) that are unchanging over time are held constant

• comparing change over time with-in the same case

All confounding variables that change the similarly in “treated” and “untreated” case are held constant.

• By comparing change over time in “treated” to change over time in “control”

In order to infer \(X\) causes \(Y\) if \(X,Y\) correlated in difference-in-differences comparison…

Assumption

• we assume the observed trend in \(Y\) for “untreated” case is equal to the “counterfactual trend” in \(Y\) for the “treated” case.
• Equivalently: we assume “treated” and “untreated” have the “parallel trends” in \(Y\), absent the change in \(X\).
• Equivalently: no variables that affect \(Y\) and change over time differently in “treated” and “untreated” cases

Do you believe assumption of “parallel trends”? (Counterfactual Missouri trend same as factual Arkansas trend)

Confounding UNSolved…

• Arkansas and Missouri murder rates mostly move together before 2007.
• But large change in AR in 2001/2002 not in MO
• But in 2007, before law took effect, murders dipped

Perhaps there are some things that affect murder rates that change differently in these two states.

When is the assumption plausible?

• We can check to see if cases share trends before treatment, but does not prove they would have shared trends after treatment
• We should compare cases that experience many similar changes over time: comparing Missouri to British Columbia may not be helpful.