"There are lies, damned lies, and statistics."
~ Mark Twain
~ Mark Twain
Overview
The internet is riddled with unsupported claims. Politics, media, business, health, and other important areas have become fertile grounds for the sowing of unsubstantiated assertions. It has made it harder for truth-tellers to persuasively communicate to others. Might backing up one’s claims with credible statistics help?
Research is mixed on the effectiveness of statistics for increasing the believability of one’s claims. Conventional wisdom suggests that presentations and research papers benefit from citable statistics by credible, independent third-parties. In contrast, research from Harvard Business School shows that statistics are less than half as memorable as simple storytelling (Graeber, Roth, & Zimmermann, 2022).
Furthermore, not everything is a matter of fact. Some things are simply opinions. And even some “facts” can have multiple sides to the story. The common use of statistics to merely buttress one’s own views of the world, even when such views run counter to the preponderance of evidence, may have made many skeptical of claims backed by statistics.
To put it the test, we designed an experiment to investigate whether including a statistic increases the believability of a claim, as well as the perceived competence of the claim maker.
The Experiment
1,200 people were recruited from the research platform Prolific to take part in a survey study involving a series of short paragraphs and a few survey questions to measure their perceptions of the authors. Unbeknownst to participants, the survey was actually an experiment, a randomized controlled trial. One of the paragraphs participants read included a claim that either was or was not supported by a research-backed statistic (randomly assigned), followed by a survey question measuring the perceived believability of the claim and the perceived intelligence of the person making the claim. The context of the paragraph was also varied to test whether the persuasion effect, if any, differed depending on the topic.
Each participant was randomly assigned to read a claim about politics, work, or mental health, randomly assigned. We then also randomly assigned whether the claim was supported by a cited statistic. For brevity, we present each claim with the cited statistic below; claims with no cited statistic simply omit the numbers and parenthetical citations.
Below is a claim from an acquaintance on social media:
"I know a lot of people criticize the government, but Americans actually view federal agencies more favorably than unfavorably. For example, a recent study by Pew Research (March 2023) shows that about 80% of Americans view the U.S. Postal Service and National Park Service favorably. Overall people like the government more than we think."
"I know a lot of people criticize the government, but Americans actually view federal agencies more favorably than unfavorably. For example, a recent study by Pew Research (March 2023) shows that about 80% of Americans view the U.S. Postal Service and National Park Service favorably. Overall people like the government more than we think."
Below is a claim from an online business article:
"In the U.S. workforce, most employees are quite satisfied with their boss, according to a study by Pew Research (March 2023). For example, 62% of those surveyed are at least very satisfied with their relationship with their manager at work."
"In the U.S. workforce, most employees are quite satisfied with their boss, according to a study by Pew Research (March 2023). For example, 62% of those surveyed are at least very satisfied with their relationship with their manager at work."
Below is a claim from a college essay:
"Although many people believe that global happiness is rising due to technological and economic progress, people are likely unhappier. According to a Gallup poll, anger, stress, sadness, physical pain, and worry have steadily increased around the world, by 37.5% since 2006."
"Although many people believe that global happiness is rising due to technological and economic progress, people are likely unhappier. According to a Gallup poll, anger, stress, sadness, physical pain, and worry have steadily increased around the world, by 37.5% since 2006."
Following the claim, participants were asked two survey questions. These survey questions served to measure our outcomes of interest, specifically claim believability and perceived competence of the person making the claim.
Participants were asked, “To what extent do you believe this claim? (1 = Not at all, 7 = Very much)” and “How smart do you think this person is? (1 = Not at all, 7 = Very much)” with answer options provided using a 1-7 survey scale. We also included a set of demographic survey questions to test whether the results differed based on gender, age, education, or political beliefs.
Participants were asked, “To what extent do you believe this claim? (1 = Not at all, 7 = Very much)” and “How smart do you think this person is? (1 = Not at all, 7 = Very much)” with answer options provided using a 1-7 survey scale. We also included a set of demographic survey questions to test whether the results differed based on gender, age, education, or political beliefs.
Results
On average, even with a large sample of 1,200 readers, the data revealed no statistically significant effect of including a statistic on claim believability (p = 0.432) or the perceived competence of the person making the claim (p = 0.262). There were, however, a few notable differences depending on context and demographics.
Regarding context, the nature of the claim may affect the persuasiveness of the statistic. Including a statistic was not helpful in the political context (p = 0.477), where we claimed that people view the government favorably, nor in the mental health context (p = 0.742), where we claimed that society has become unhappier.
However, in the work context, where we claimed that most workers are satisfied with their boss, adding a statistic increased believability (p = 0.003) and perceived competence of the writer (p < 0.001), both by 13.5%. An interaction test using OLS regression formally confirmed the significance of this result (p = 0.007; p < 0.001), with an increase in the stat effect of one-half and two-thirds of a point on our 1-7 survey scales, respectively.
However, in the work context, where we claimed that most workers are satisfied with their boss, adding a statistic increased believability (p = 0.003) and perceived competence of the writer (p < 0.001), both by 13.5%. An interaction test using OLS regression formally confirmed the significance of this result (p = 0.007; p < 0.001), with an increase in the stat effect of one-half and two-thirds of a point on our 1-7 survey scales, respectively.
A major reason why it is so difficult to persuade others is confirmation bias. With confirmation bias, people seek out information that confirms their own beliefs. When confronted with information that opposes these beliefs, we tend to ignore the information or choose not to believe it.
In this study, confirmation bias likely has a larger effect on claim believability than statistics. This is highlighted in the political context segment of our study, in which we claimed that people view the government more favorably than unfavorably. While we find no effect of statistic use on claim believability, we do find a significant relationship between claim believability and participants’ preexisting political beliefs (p < 0.0001).
Compared to Independents, Democrats were about 9% more likely to believe the claim (0.40 points on a 1-7 scale), while Republicans were about 12% less likely to believe the claim (0.53 points). In contrast, Republicans were 15% more likely to believe the work context claim that workers are satisfied with their bosses (0.55 points; p = 0.012), which aligns with the more pro-business stance of political conservatives.
In this study, confirmation bias likely has a larger effect on claim believability than statistics. This is highlighted in the political context segment of our study, in which we claimed that people view the government more favorably than unfavorably. While we find no effect of statistic use on claim believability, we do find a significant relationship between claim believability and participants’ preexisting political beliefs (p < 0.0001).
Compared to Independents, Democrats were about 9% more likely to believe the claim (0.40 points on a 1-7 scale), while Republicans were about 12% less likely to believe the claim (0.53 points). In contrast, Republicans were 15% more likely to believe the work context claim that workers are satisfied with their bosses (0.55 points; p = 0.012), which aligns with the more pro-business stance of political conservatives.
These results echo a similar finding from one of our studies involving a scientific claim. Qualitative data from the study suggests that one of the most prominent factors influencing readers’ trust in the information was their preexisting beliefs about whether or not the claim was true. Such preexisting beliefs are often rooted in one’s own personal experiences and anecdotal evidence, which often carry significant bias.
Conclusion
Overall, the results suggest that including a statistic to back up your claim may not be as powerful as it seems. Two of our three contexts showed no significant increase in claim believability or perceived competence of the claim-maker. Readers’ preexisting confirmation bias may be more influential, a theory supported by the larger relationship between political beliefs and our outcomes.
Of course, another possible explanation is that some of the claims chosen for this study simply weren’t believable enough to persuade readers. After all, individuals and companies are notorious for contorting statistics to fit their viewpoints, even when the overall evidence suggests the opposite of their claims. And although it’s tempting to just choose a stat from a publication more line with a reader’s preexisting beliefs (e.g., Fox News or the Wall Street Journal for conservatives, CNN or the New York Times for liberals), our own research attests to the limitations of such an approach.
Nevertheless, the fact that one context (work) exhibited increased believability from adding a statistic suggests that the effect of statistics may be complex or nuanced. Perhaps claims with which people have few preexisting beliefs benefit more from statistics. Perhaps including more than one statistic or citation, as is the norm in research papers, could be more persuasive. One thing is for sure, future research in this area is certainly warranted.
References
Graeber, Thomas, Christopher Roth, and Florian Zimmermann. “Stories, Statistics and Memory.” Working Paper, December 2022.
Methods Note
We used Ordinary Least Squares (OLS) regression analyses to test for significant differences in claim believability and claimant competence between our experimental conditions. OLS regression analyses with interaction terms were used to evaluate differences in political beliefs and context interactions. Our statistical significance threshold was a p-value below 0.05. The data and survey materials used for this study are available upon request.