I’ve Got the Power: A Survey of Issues Surrounding Statistical Power in the Design and Analysis of Survey Experiments

Clayton Webb (University of Kansas) and Cameron Wimpy (Arkansas State University)

Abstract: The power of a hypothesis test to reject a false null hypothesis is a basic concept of statistical inference that is introduced in most, if not all, introductory texts. Despite this, a systematic survey of work published in the American Journal of Political Science (AJPS), the American Political Science Review (APSR), the Journal of Politics (JOP), and Public Opinion Quarterly (POQ) between 2010 and 2020 reveals that the concept continues to be ignored and/or misunderstood by many political scientists who use survey experiments to test hypotheses about public opinions, attitudes, and beliefs. The purpose of this research note is to highlight the apparent gaps in the political science community’s collective understanding about important concepts related to statistical power relevant to the design and analysis of survey experiments. We outline basic problems in the way statistical power is discussed in existing research and review a series of more advanced issues including the design and implementation of power analyses, testing for so-called “nill effects,” and problems created by un-even randomization. We conclude by offering a set of best practices for the design of survey experiments, the analysis of experimental data, and the presentation of experimental results.

View Poster in a New Tab