Chair: Suzanna Linn (Penn State University)

Co-Host: Anwar Mohammed (McMaster University)

Experimental Evaluation of Computer-Assisted Human Decision Making: Application to Pretrial Risk Assessment Instrument

Download Paper

Author(s): Kosuke Imai, Zhichao Jiang, James Greiner, Ryan Halen and Sooahn Shin

Discussant: Jonathan Mummolo (Princeton University)

Despite an increasing reliance on computerized decision making in our day-to-day lives, human beings still make highly consequential decisions. As frequently seen in business, healthcare, and public policy, recommendations produced by statistical models and machine learning algorithms are provided to human decision-makers in order to guide their decisions. The prevalence of such computer-assisted human decision making calls for the development of a methodological framework to evaluate its impact. Using the concept of principal stratification from the causal inference literature, we develop a statistical methodology for experimentally evaluating the causal impacts of machine recommendations on human decisions. We also show how to examine whether machine recommendations improve the fairness of human decisions. We apply the proposed methodology to the randomized evaluation of a pretrial risk assessment instrument (PRAI) in the criminal justice system. Judges use the PRAI when deciding which arrested individuals should be released and, for those ordered released, the corresponding bail amounts

Chair: Kevin Quinn (University of Michigan)

Co-Host: Ilayda Onder (Penn State University)

Untangling Mixtures in Judicial Opinions

Author(s): Douglas Rice and Justin Gross

Discussant: Brandon Stewart (Princeton University)

Within the small deliberative voting body of the U.S. Supreme Court, prior work has regularly looked to the voting alignments of the justices in order to understand bargaining power and who wins or loses. However, little consensus has emerged over who exerts control over the Court's actual output--judicial opinions. Rather, three distinct perspectives have emerged: (1) The opinion author controls opinion content; (2) the median member of the court controls opinion content; (3) the median of the majority coalition controls the content of the opinion. In this paper, by contrast, we look for evidence of influence where it theoretically operates -- in the opinions themselves -- and identify the unique contributions of separate justices to the collective output of the Court. To do so, we build on a well-established tradition within text-as-data research on the challenge of authorship attribution, and develop a novel authorship model that leverages writing characteristics to predict the authors of individual sentences. More specifically, we introduce an approach that evaluates authorship across multiple competitive

Chair: Ludovic Rheault (University of Toronto)

Co-Host: Regan Johnston (McMaster University)

Elements of External Validity: Framework, Design, and Analysis

Author(s): Naoki Egami and Erin Hartman

Discussant: Daniel Hopkins (University of Pennsylvania)

External validity of randomized experiments has been a focus of long-standing methodological debates in the social sciences. However, in practice, discussions of external validity often differ in their definitions, goals, and assumptions, without making them clear. Moreover, while many applied studies recognize it as their potential limitations, unfortunately, few studies have explicit designs or analysis aimed towards externally valid inferences. In this article, we propose a framework, design, and analysis to address two central goals of external validity inferences — (1) assess whether the direction of causal effects is generalizable (sign-validity), and (2) generalize the magnitude of causal effects (effect-validity). First, we propose a formal framework of external validity to decompose it into four components, X-, Y-, T-, and C-validity (units, outcomes, treatments, and contexts) and clarify the source of potential biases. Second, we present assumptions required to make externally valid causal inferences, and we propose experimental designs to make such assumptions more plausible. Finally, we introduce a multiple-testing procedure to address sign-validity and general

Chair: Justin Esarey (Wake Forest University)

Co-Host: Md Mujahedul Islam (University of Toronto)

How Wide is the Ethnic Border?

Author(s): Scott Abramson, Brandon Cooley and Bethany Lacina

Discussant: Florian Hollenbach (Texas A&M University)

We explore the relationship between ethnic heterogeneity and within- and cross-country barriers to trade. We develop a spatial model of trade in which observable productivity shocks directly affect local prices. These local shocks propagate through the trading network differentially, depending on unobserved trading frictions. Coupling data describing monthly commodity prices in 227 cities across 42 African counties, remotely sensed weather data, and spatial data describing the locations of ethnic-group homelands, we estimate this model to quantify the costs traders incur when by crossing ethnic and national borders. We show that ethnic borders induce a friction approximately half the magnitude of national borders, indicating that ethnic heterogeneity is an impediment to the development of efficient national markets. Through counterfactual experiments, we quantify the effect of these frictions on consumer welfare and the extent to which colonial-era political borders have hindered African economic integration. In all, our paper suggests that trade impediments caused by ethnic heterogeneity are a substantial channel through which ethnic fractionization

Co-Host: Mikaela Karstens (Penn State University)

Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference

Download Paper

Presentation Slides

Author(s): Max Goplerud

Discussant: Justin Grimmer (Stanford University)

Estimating non-linear hierarchical models can be computationally burdensome in the presence of large datasets and many non-nested random effects. Popular inferential techniques may take hours to fit even relatively straightforward models. This paper provides two contributions to scalable and accurate inference. First, I propose a new mean-field algorithm for estimating logistic hierarchical models with an arbitrary number of non-nested random effects. Second, I propose “marginally augmented variational Bayes” (MAVB) that further improves the initial approximation through a post-processing step. I show that MAVB provides a guaranteed improvement in the approximation quality at low computational cost and induces dependencies that were assumed away by the initial factorization assumptions. I apply these techniques to a study of voter behavior. Existing estimation took hours whereas the algorithms proposed run in minutes. The posterior means are well-recovered even under strong factorization assumptions. Applying MAVB further improves the approximation by partially correcting the under-estimated variance. The proposed methodology is implemented in an open source software package

Co-Host: Justin Savoie (University of Toronto)

An omitted variable bias framework for sensitivity analysis of instrumental variables

Author(s): Carlos Cinelli and Chad Hazlett

Discussant: Jacob Montgomery (WUSTL)

We develop an omitted variable bias framework for sensitivity analysis of instrumental variable (IV) estimates that is immune to "weak instruments," naturally handles multiple "side-effects" and "confounders," exploits expert knowledge to bound sensitivity parameters, and can be easily implemented with standard software. In particular, we introduce sensitivity statistics for routine reporting, such as robustness values for IV estimates, describing the minimum strength that omitted variables need to have to change the conclusions of a study. We show how these depend upon the sensitivity of two familiar auxiliary estimates–the effect of the instrument on the treatment (the "first-stage") and the effect of the instrument on the outcome (the "reduced form")–and how an extensive set of sensitivity questions can be answered from those alone. Next, we provide tools that fully characterize the sensitivity of point-estimates and confidence intervals to violations of the standard IV assumptions. Finally, we offer formal bounds on the worst damage caused by these violations by means of comparisons with the explanatory

Chair: Suzanna Linn (Penn State University)

Co-Host: Ilayda Onder (Penn State University)

Bayesian Causal Inference With Time-Series Cross-Sectional Data: A Dynamic Multilevel Latent Factor Model with Hierarchical Shrinkage

Author(s): Yiqing Xu, Xun Pang and Licheng Liu

Discussant: Neal Beck (New York University)

This paper proposes a Bayesian causal inference method based on estimating posterior predictive distributions of counterfactuals with TSCS data. To construct the prediction model, we fully take advantage of the flexibility of multilevel modeling and Bayesian model specification to reduce dependence on modeling assumptions. We start with a multilevel dynamic factor model and adopt a Bayesian Lasso-like hierarchical shrinkage strategy for stochastic model-specification selection. Counterfactual imputation based on the posterior predictive distribution generalizes the classic synthetic control approach by assigning observation-specific weights to features of the treated units and exploiting high-order relationships between treated and control time series. With empirical posterior distributions of counterfactuals, it is convenient and intuitive to make causal

Co-Host: Md Mujahedul Islam (University of Toronto)

Retrospective causal inference via elapsed time-weighted matrix completion, with an evaluation on the effect of the Schengen Area on the labour market of border regions

Download Paper

Author(s): Jason Poulos, Andrea Albanese, Andrea Mercatanti and Fan Li

Discussant: James Bisbee (Princeton University)

We propose a strategy of retrospective causal inference in panel data settings where (1) there is a continuous outcome measured before and after a single binary treatment; (2) there exists a group of units exposed to treatment during a subset of periods (switch-treated) and group of units always exposed to treatment (always-treated), but no group that is never exposed to treatment; and (3) the elapsed treatment duration, z, differs across groups. The potential outcomes under treatment for the switch-treated in the pre-treatment period are missing and we impute these values via nuclear-norm regularized least squares using the observed (i.e, factual) outcomes. The imputed values can be interpreted as the counterfactual outcomes of the switch-treated had they been always-treated. Differencing the counterfactual outcomes from the factual outcomes can be interpreted as the effect of not having assigned treatment to the switch-treated in