Overview
A major challenge for researchers and media outlets is reporting reliable research. For everyday consumers of that research, trust can be even more elusive. There are numerous incentives for both researchers and the media to report sensational yet dubious studies that often cater to readers' political biases.
One such context where this played out was the COVID-19 pandemic. Throughout the pandemic, building trust in vaccines and scientific research was a major challenge for the Centers for Disease Control and Prevention (CDC), especially among politically conservative individuals. It was suggested that if conservative media outlets like Fox News and the Wall Street Journal (WSJ) reported on the benefits of vaccines, more right-leaning individuals may have chosen to get vaccinated. However, it’s an open question whether reporting from politically aligned media outlets would improve the believability of the research.
Given how interesting and important this question is, we decided to test it with a simple experiment.
The Experiment
We recruited 400 people from Amazon Mechanical Turk to take part in a research study. Participants were shown a media headline about a research finding attributed to either the CDC or the WSJ, randomly assigned, then asked to rate the believability of the claim using a survey scale. The research claim was about the controversial topic of whether Vitamin C reduces flu symptoms, a subject still being debated and researched.
Participants were instructed to "Imagine you’re browsing the news online and come across the following headline by the [Centers for Disease Control and Prevention (CDC) / Wall Street Journal (WSJ)]." Each participant then viewed a corresponding news snippet with a headline and brief summary from either the WSJ or CDC. Both snippets contained the same headline and summary text; only the source attribution (CDC logo or WSJ logo) differed. One-half of participant were randomly assigned to see the WSJ-attribution, while the other half saw the CDC attribution.
Participants were then asked “To what extent do you believe the [CDC’s / WSJ’s] claim? (1 = Not at all, 7 = Very much)” using a 1-7 scale. At the end of the survey, we asked participants “Generally speaking, do you usually think of yourself as a Republican, a Democrat, an Independent, or something else?” with answer options “Republican,” “Democrat,” “Independent,” “Other,” and “No preference.” We excluded the few observations from participants that selected “Other” or “No preference.”
Results
There was no significant difference in believability of the research claim between the CDC (avg. = 4.92) and WSJ (avg. = 4.78) conditions, (p = 0.374). And although Democrats tended to believe the research more than conservatives and independents (diff. = 1.18, p < 0.0001), there was no significant interaction effect between political beliefs and reporting source (p = 0.837). Republicans were no more or less likely to believe the research claim when it was reported by the WSJ than the CDC.
Conclusion
The results suggest that simply presenting research claims through more conservative media outlets like the Wall Street Journal may not be sufficient to persuade skeptics to trust the information, even when those skeptics lean to the political right. Trust in scientific claims appears to be influenced more by individual political beliefs than the source of the information.
Perhaps future research could investigate alternative approaches to increasing trust in science, such as changing the narrative or writing style, or leveraging personal relationships like family or close friends who do trust the science. Future research should also explore the role of empathy in this process.
Methods Note
We used ordinary least squares (OLS) regression analysis to test for significant differences in believability between the CDC and WSJ claims, as well as to test for interactions with political beliefs. We used a p-value of 0.05 or below as our threshold for statistical significance. The survey materials and data for this experiment are available upon request.