Week 12: Do Small Classes Increase the Probability of Graduating from High School?

POL269 Political Research

Javier Sajuria

2024-08-04

Do Small Classes Increase Probability of Graduating from High School?

  • The dataset comes from a randomized experiment conducted in Tennessee, where students were randomly assigned to attend either a small class or a regular-size class from kindergarten until 3rd grade

  • Given that we are analyzing experimental data, what do we need to compute to estimate the average causal effect? __________________

  • Although, we could compute it directly let’s compute it by fitting a linear model so that ______ is equivalent to it

In-Class Exercise

  1. Open RStudio

  2. Open exercise_5.R from within RStudio

  3. Run step 0

    • use the setwd() for your computer

0. Get Ready for the Analysis

  • Load the data and create any variables needed
## set the working directory to DSS folder
setwd("~/Desktop/pol269") # if Mac
setwd("C:/user/Desktop/pol269") # if Windows
## load the data and packages
star <- read.csv("STAR.csv") # reads and stores data
library(tidyverse) # loads tidyverse package
## create treatment variable
star <- star |> 
  mutate(small = ifelse(classtype == "small", 1, 0)) # creates new variable

  • Look at the data
head(star) # shows first observations
  classtype reading math graduated small
1     small     578  610         1     1
2   regular     612  612         1     0
3   regular     583  606         1     0
4     small     661  648         1     1
5     small     614  636         1     1
6   regular     610  603         0     0
  • The treatment variable is small
  • The outcome variable is graduated
  • Is the outcome variable binary? And, if not, what’s its unit of measurement? (this affects the interpretation of \(\widehat{\beta}\))
    • yes, Y is binary, so \(\widehat{\beta}\) will be measured in p.p (x100)

1. What Is the Estimated Average Treatment Effect?

  • Fit a linear model so that the estimated slope coefficient is equivalent to the difference-in-means estimator. In this case, the fitted line is: \(\widehat{\textrm{\textit{graduated}}} = \widehat{\alpha} + \widehat{\beta} \,\textrm{small}\)

  • Store the fitted model in an object called fit and then ask R to provide the contents of fit

  • R code to fit and store linear model?

  • R code to ask R to provide contents of fit? 
## 
## Call:
## lm(formula = graduated ~ small, data = star)
## 
## Coefficients:
## (Intercept)        small  
##    0.866473     0.007031
  • \(\widehat{\beta}\) = ______

  • \(\widehat{\beta}\) = 0.007

  • Direction, size, and unit of measurement of the effect?

    • An increase of about 0.7 percentage points

  • What’s the estimated average treatment effect? (Make sure to mention all the key elements: the assumption, why the assumption is reasonable, the treatment, the outcome, as well as the direction, size, and unit of measurement of the average treatment effect)

  • Answer: Assuming that students who attended a small class were comparable to students who attended a regular-size class (a reasonable assumption because the data come from a randomized experiment), we estimate that attending a small class increases the probability of graduating from high school by about 0.7 percentage points, on average

  • Why?

    • increase because we are measuring a change in \(Y\) and \(\widehat{\beta}\) (which is equivalent to the difference-in-means estimator) is positive
    • percentage points because the difference-in-means estimator is the result of subtracting two percentages: % - % = p.p. (because graduated is binary)
    • 0.7 (and not 0.007); we need to multiply the result by 100 to turn it into p.p. (because graduated is binary)

2. Is the Effect Statistically Significant?

  • That is, is the average treatment effect distinguishable from zero at the population level, statistically speaking?

    • To answer, we need to do hypothesis testing

  1. Specify null and alternative hypotheses

    • \(H_0 {:} \,\, \beta=0\) (meaning: attending a small class has no average causal effect on the probability of graduating from high school at the population level)
    • \(H_1 {:} \,\, \beta\neq0\) (meaning: attending a small class either increases or decreases the probability of graduating from high school, on average, at the population level)

  1. Compute observed test statistic and associated p-value
  • We ask R to compute both for us by running summary() where we specify inside the parentheses the name of the object where we stored the output of the lm() function
summary(fit)

Call:
lm(formula = graduated ~ small, data = star)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.8735  0.1265  0.1265  0.1335  0.1335 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.866473   0.012834  67.514   <2e-16 ***
small       0.007031   0.018940   0.371    0.711    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.3369 on 1272 degrees of freedom
Multiple R-squared:  0.0001083, Adjusted R-squared:  -0.0006777 
F-statistic: 0.1378 on 1 and 1272 DF,  p-value: 0.7105
  • The observed test statistic is ____
  • The associated p-value is ____

  1. Do we reject or fail to reject the null hypothesis? We fail to reject the null hypothesis because …

    • option A: absolute value of the observed test statistic is less than 1.96 (|0.37| \({<}\) 1.96)
    • option B: p-value is larger than 0.05 (0.71 \({>}\) 0.05)

  • Is the effect statistically significant at the 5% level?

  • Answer: No, the effect is not statistically significant at the 5% level. We do not have enough evidence to state that attending a small class is likely to have a non-zero average causal effect on the probability of graduating from high school, at the population level.

3. Can We Interpret the Effect as Causal?

  • How strong is the internal validity of this study? Have the researchers accurately measured the average causal effect on the sample of students who were part of the study?

  • Answer: Yes, we can interpret the effect as causal. The internal validity of this study is strong because the treatment (attending a small class) was assigned at random. Random treatment assignment should have eliminated all confounding variables. Students that were assigned to attend a small class should be comparable to students that were assigned to attend a regular-size class.

4. Can We Generalize the Results?

  • How strong is the external validity of this study? To what population can the findings be generalized to?

  • Answer: Given the characteristics of the study, only students from large schools in Tennessee were able to participate in the experiment. As a result, the sample of participating students was not perfectly representative of all students in Tennessee or of all students in the U.S. Consequently, we can conclude that, although we do get to observe the treatment of interest in the real world, the analysis has relatively weak external validity, especially if one wishes to generalize the study’s conclusions to all schools and students in Tennessee or in the entire United States. (See subsection 5.5.4 of the book.)

Final thoughts

  1. Be critical, but don’t abuse
  2. Call bullshit
  3. “We’re all smart. Distinguish yourself by being kind.”
  4. And…

THANKS

POL269