Overview
The pace of change in our world today has raised concerns about the credibility of research, particularly outdated studies that may no longer be relevant or accurate. Time changes things. And the gears of research turn slowly.
This doesn’t mean older research shouldn’t be trusted. On the contrary, many ground-breaking findings, even in social science fields like economics and psychology, have stood the test of time. Nevertheless, many more studies fail to replicate (Bohannon, 2015; Camerer, et al., 2018), not only because of changing times, but also due to outdated research methods and publication standards.
But irrespective of the actual credibility of past research, we know very little about the general public’s perceptions of it. To explore whether non-academic audiences are more discerning when it comes to older research findings, we conducted a randomized controlled trial testing their perceptions of old and new research credibility.
The Experiment
In this experiment, 400 participants on Amazon Mechanical Turk were recruited to take part in a survey discreetly designed as a randomized experiment. Participants were asked to read a research finding that may or may not be accurate, then rate their believability of the research finding. Unbeknownst to them, we randomized the year in which the research was allegedly published, either 1999 or 2019.
Participants read the following statement:
The following claim may or may not be true:
A research study from [1999 / 2019] at Northwestern University suggests that people looking at higher-elevation sceneries are more willing to purchase new products.
After reading the claim, participants were asked, "How strongly do you believe this research finding is true? (1 = Not at all, 7 = Very much),” using a 1-7 survey scale.
Results
We found no significant difference in believability between the 1999 research study (avg. = 4.13) and the 2019 research study (avg. = 4.00) (p = 0.447). The 1999 study was actually rated as slightly more believable, though not significantly so. The age of participants did not affect the results. Thus, it seems that two decades of time elapsing does not affect the perceived credibility of a research finding.
Conclusion
The results suggest that everyday people do not discount research studies from 20 years ago, at least not in the context of consumer purchasing. It remains an open question whether research in other fields, like computer science, may be discounted more quickly, or whether such effects take longer than a couple of decades. Perhaps further investigation will shed light on the longevity of this believability, both in different fields and over longer intervals. Only time will tell.
References
Bohannon, J. (2015). Many psychology papers fail replication test. Science, 349(6251), 910-911.
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644.
Methods Note
We used Ordinary Least Squares (OLS) regression analysis to test for significant differences in believability between our old and new research conditions. A small p-value (less than 0.05) indicates a statistically significant difference. The survey materials and data for this study are available upon request.