A review of ‘Motivated Numeracy and Enlightened Self-Government’ by Dan Kahan, Ellen Peters, Erica Dawson and Paul Slovic, for Behavioural Public Policy.
By Elke U. Weber – Princeton University
Back in 1984 in a paper entitled ‘Combine and Conquer,’ I argued that two analytic approaches typically used in isolation could and should be used in combination to better understand people’s interpretation of the concept of risk. Since then I have been advocating for collaboration rather than competition between disciplines, be they the physical and social sciences who can better address global environmental challenges while acknowledging and complementing their respective contributions, or psychology and economics whose respective descriptive versus normative framework of studying choice are both essential inputs to effective prescriptions for better decision making.
Today, I appeal to the authors of this paper not to create false dichotomies. Both cognitive and social psychology make important and valid contributions to our understanding of the public response to risks. Diminishing one to augment the other is counterproductive and unnecessary. The authors assert that “the number of plausible explanations for persistent public controversy over risks and other policy-relevant facts exceeds the number that can actually be true.” The truth of this statement may well depend on the definition of “plausible,” but I argue that the statement is actually proven false by the authors’ own data for the two specific explanations they set out to test.
Even for the narrow context studied (namely covariation assessment, far removed from the task of accurately assessing and responding to personal or societal risks to which the authors at least implicitly wish to extend their conclusions), the data are instead consistent with a different claim, namely that strong behavioral phenomena like covariation assessment, hindsight bias, status-quo bias, or disagreements about the seriousness of risks tend to be strong and stable across respondents and contexts because they have multiple determinants. Thus it makes neither intellectual nor practical sense to pitch cognitive against motivational explanations, as both the how and the why of judgments and decisions matter.
Respondents in this study were asked to decide whether an intervention was helpful or harmful. A 2×2 table provided them with a count of both treated and untreated units that showed either better or worse outcomes. In one version of the task those counts indicated that the intervention was helpful, in the other version that it was harmful. On top of that, the intervention was either use of a new skin cream that would make a person’s rash better or worse or a ban on carrying concealed handguns in public that would make a city’s crime rate decrease or increase. Respondents evaluated the 2×2 table of counts provided to them under one of the four scenarios, indicated whether they thought that the treatment had been helpful or harmful, and then provided demographic information including their political orientation and level of numeracy.
The difficulty in correctly judging the effectiveness of the intervention from the data provided stems from the fact that the sample size of treated versus untreated units was different. This does not affect the correct algorithmic solution (comparing the ratio of better-over-worse outcomes in the treated versus untreated sample), but results in an incorrect answer if one of two previously documented heuristics (that use a comparison of counts rather than ratios) is used. The authors for some reason link use of the algorithm rather than the garden-path heuristic to what they call the “Science Comprehension Thesis” (SCT), even though their introduction defines SCT as reliance on System 1 reasoning, such as the use of feelings or other associations in estimates of risk. They contrast this to the “Identity-protective Cognition Thesis” (ICT), which holds that “intact capacity to comprehend decision-relevant science is [willfully] disabled by cultural conflict.”
Numeracy should increase the chance that respondents use the correct algorithm, with the SCT being the only explanation for such an effect, contrary to the authors’ claim that both SCT and ICT predict such an effect for the no-cultural-conflict skin-treatment scenario. ICT is moot on the issue of processing of 2×2 tables as a function of numeracy but only addresses what happens when an issue is contentious, predicting for example an interaction between political ideology and correctly assessing the effectiveness of the concealed-handgun ban.
Here is what was found. (1) Numeracy indeed predicted the likelihood of correct contingency judgments for the skin-treatment scenario, regardless of political ideology. As I suggested earlier, this is predicted only by the cognitive SCT, not by the motivational ICT, which does not speak to cognitive handicaps and/or individual differences in their strength. (2) Political ideology colored respondents’ accuracy for the handgun-crime scenario, as predicted by ICT, and did so more for the highly numerate. SCT is moot on this effect, but says nothing that would not allow for such an effect. The two theses simply address different parts of the problem that are not mutually exclusive. Low numeracy reveals the heuristic shortcuts we all take, in situations where they result in bias when not corrected by more sophisticated operations known to the highly numerate. In addition to the comparison-of-counts (rather than ratios) heuristics, the data show evidence for an associative heuristic that connects “larger numbers” with an “increase” judgment, as shown in Figure 6 where the low numerate are 15-20% more accurate in their judgments for the “rash increases” scenario than the “rash decreases” scenario, even though they are processing the identical 2×2 table of information. And interestingly, the highly numerate recognize that both a count-comparison and a larger-goes-with-increase process can be used as an excuse to generate the judgment that they would like to be true, given with their political persuasion. Not only do liberals versus conservatives show divergent judgments in opposite directions for the handgun-crime scenario, but the divergence is larger for the crime-decreases version than for the crime-increases version.
This study provides nice evidence for motivated reasoning. It also provides extensive evidence for cognitive challenges. Enlightened government will combine and conquer, i.e., know when to address each problem to different degrees depending on targeted respondents and context.