Statistics and Statistical Programming (Fall 2020)/pset8

Part I: Analyze and interpret a simulated study of education and income

The first set of programming challenges this week pose an open-ended set of questions about a simulated dataset from an observational study of high school graduates' academic achievement and subsequent income. Here is some information about the "study design" (note: this is not data from an actual study):

Data from twelve cohorts of public high school students was collected from across the Chicago suburbs. Each cohort incorporates a random sample of 142 students from a single suburban school district. For each student, researchers gathered a standardized measure of the students' aggregate GPA as a proxy for their academic achievement. The researchers then matched the students' names against IRS records five years later and collected each student's reported pre-tax earnings for that year.

I have provided you with a version of the dataset from this hypothetical study in which each row corresponds to one student. For each student, the dataset contains the following variables:

• id: A unique numeric identifier for each student in the study (randomly generated to preserve student anonymity).
• cohort: An anonymized label of the cohort (school district) the student was drawn from.
• gpa: Approximate GPA percentile of the student within the entire district. Note that this means all student GPAs within each district were aggregated and converted to an identical scale before percentiles were calculated.
• income: Pre-tax income (in thousands of US dollars) reported to the U.S. federal government (IRS) by the student five years after graduation.

For the rest of this programming challenge, you should use this dataset to answer the following research questions:

• How does high school academic achievement relate to earnings?
• How does this relationship vary by school district?

You may use any analytical procedures you deem appropriate given the structure of the dataset and study design. Some things you may want to keep in mind:

• ANOVAs, T-tests, and linear regression can help you test different kinds of hypotheses.
• Adjusting for multiple comparisons is important.

Part II: Trick-or-treating all over again

The second set of programming challenges this week revisit the trick-or-treating experiment we analyzed a few weeks ago.

Load up the dataset. For this exercise we're going to fit a few versions of the following model.

${\widehat {\mathrm {fruit} }}=\beta _{0}+\beta _{1}\mathrm {obama} +\beta _{2}\mathrm {age} +\beta _{3}\mathrm {male} +\beta _{4}\mathrm {year} +\varepsilon$

You may want to revisit your earlier analysis and exploration of the data as you prepare to conduct the following analyses. You may also want to generate new exploratory analysis and summary statistics that incorporate the age, male, and year variables that we did not consider in our analysis last time around.

Fit a model to test for treatment effects

Now, let's construct a test for treatment effects. For a between-groups randomized-controlled trial (RCT) like this, that means we'll focus on the fitted parameter for the treatment assignment variable ($\beta _{1}\mathrm {obama}$) which will provide a direct estimate of the causal effect of exposure to the treatment (compared against the control) condition. That said, here are a few tips, notes, and requests:

• The outcome is dichotomous, so you can/should use logistic regression to model this data (we can discuss this choice in class). For the sake of simplicity (and because it's not covered in the textbook), we're going to side-step any questions about the assumptions necessary to identify a logistic model as well as specific steps you might take to evaluate the model fit (but rest assured that both exist!).
• You may want/need to convert some of these variables to appropriate types/classes in order to fit a logistic model. I also recommend at least turning year into a factor and using a centered version of the age variable (we can discuss this in class too).
• Be sure to state the alternative and null hypotheses related to the experimental treatment under consideration.
• It's a good idea to include the following in the presentation and interpretation of logistic model results: (1) a tabular summary/report of your fitted model including any goodness of fit statistics you can extract from R; (2) a transformation of the coefficient estimating treatment effects into an "odds ratio"; (3) model-predicted probabilities for prototypical study participants. (please note that examples for all of these are provided in Mako Hill's R tutorial on interpreting the results of logistic regression])
• For the model-predicted probabilities, please estimate the treatment effects for the following hypothetical individuals:
  • a 9-year old girl in 2015.
  • a 7-year old boy in 2012.

Conduct a post-hoc "sub-group" analysis

The paper mentions that the methods of random assignment and the experimental conditions were a little different for each year in which the study was run. Fit models (without the parameter for $\mathrm {year}$) on the corresponding subsets of the data (2012, 2014, 2015).

Interpret and discuss your results

Explain what you found! Be sure to find useful and meaningful ways to convey your findings in terms of the odds-ratios and model-predicted probabilities. Make sure to address any discrepancies you observe between your original (i.e., Problem Set 5) t-test estimates, the "full" logistic model results you estimated and the sub-group analysis you conducted above.