Letter to the Editor: Methodology in the Faculty Survey

To the Editor,

I read your article "Majority of Faculty Support Fossil Fuel Divestment, Social Group Sanctions”, which is the first in a series of planned articles that makes use of a survey that was sent to faculty in the Faculty of Arts and Sciences.

I am very concerned with the methodology employed for the survey, as well as the interpretation of results.

The survey was sent out to members of the FAS faculty through a repeated set of emails, which invited faculty to click on a weblink. Unfortunately, anyone could click on the link and take the survey. As such, it is impossible to know how many people who took the survey were actually FAS faculty. Therefore, the claim that the survey represents the opinion of over 1,000 faculty is not correct, nor can you claim a 43% response rate.

The interpretation of the results does state that the reported numbers have not been adjusted for response bias. But, if we take response bias seriously, as survey researchers and pollsters do, then you cannot make claims, such as “Majority of Faculty Support X,” or really any other claim about the data that attempts to characterize the faculty as a whole. Assuming that everyone who took the survey was an FAS faculty member, based on the response rate, all of the claims in the article could be flipped.

Running polls and interpreting the results requires responsibility. And I applaud your efforts to take seriously some of the questions you ask about our community. Irrespective of the conclusions you reach, running another four stories on this data would be irresponsible. Rather than print additional articles that are factually incorrect, I would take a step back and think through a redo of your efforts. This would include a modified approach to fielding the survey, the pre-registration of analysis plans to prevent “fishing” for relationships, and perhaps even consultation with Harvard’s Institutional Review Board for general guidance on research involving human subjects.

Dustin Tingley is a Professor of Government in the Government Department.

Editor's Note: The Crimson has corrected the headline of the April 30, 2018 story “Majority of Surveyed Faculty Support Fossil Fuel Divestment, Social Group Sanctions” to reflect the fact that data in the article reflects the opinions of respondents to a survey conducted by The Crimson, not the Faculty of Arts and Sciences overall.

—Emmanuel R. R. D'Agostino and Cristian D. Pleters, Editorial Chairs

—Derek G. Xiao, President