What the World’s Most Famous “Thinking Test” Actually Measures

About 20 years ago, Shane Frederick, professor of marketing at Yale SOM, developed the cognitive reflection test (CRT), a short quiz designed to distinguish those who are inclined to stop and think from those who are not. Most fundamentally, CRT questions are designed to spark false intuitions. Consider:

If 30 elves can wrap 30 gifts in 30 minutes,

then 40 elves could wrap 40 gifts in ___ minutes.

Most people answer 40 minutes, quickly and confidently. The answer is 30 minutes.

Subsequent research has demonstrated that excelling on the CRT, which was crafted to measure a person’s willingness to pause and reflect on gut instincts, predicts many other traits, like greater patience, better job performance, and less risk aversion. But a contingent of skeptics has long suggested predictive value of the CRT is rooted in the simple fact that it tests math abilities — nothing more.

In an effort to resolve this dispute, Frederick partnered with two skeptics, Yigal Attali from Duolingo and Maya Bar-Hillel from the University of Jerusalem, to test which of these two theories is right. (Andrew Meyer from The Chinese University of Hong Kong and Daniel Kahneman from Princeton were also part of the research team.) Publishing in the Proceedings of the National Academy of Sciences, the researchers find that the value of the CRT does, in fact, lie in its ability to measure reflectiveness, not simply mathematical aptitude. “In contrast to recent claims,” they write, “the CRT is not just a math test.”

The setup was relatively straightforward. About 4,400 participants sat for two brief tests: first, an eight-question CRT; second, an eight-question math test with problems drawn from the Graduate Record Exam (GRE). The tests were designed to be comparable in difficulty, but only the CRT had a “lure” that caters to participants’ intuition. The GRE was a straightforward math exam.

After the tests, participants took a survey designed to measure four different traits: how much they believed in scientific theories and rejected the paranormal; their preferences regarding time and risk; their reflectiveness (solving non-math problems with false intuitions); and their numeracy, or math ability.

Most fundamentally, scores on the CRT proved better than the math test at predicting these first three traits but worse at predicting numeracy. To further probe these findings, and to specifically examine the role of the intuitive lures in the CRT, the researchers told participants which CRT questions they got wrong and gave them a second chance to answer.

“The second chance CRT should function more like an ordinary math test,” the researchers write. “Telling respondents which items they missed removes the need to spontaneously identify erroneous intuitions but preserves the mathematical requirements of the problems.”

This is, indeed, what they found. Providing participants a chance to revise their answers transformed the CRT into a better predictor of numeracy: those questions that participants answered incorrectly a second time were more likely measuring struggles with the math involved than with false intuitions. The CRT’s value predicting other traits, like reflectiveness, meanwhile, declined when participants were given a second shot.

Researchers looking into cognitive reflection “have typically attempted to assess [it] using tricky math items, but math items nonetheless,” Frederick and his colleagues write. The validation of the CRT as a method for measuring this trait without depending on mathematical proficiency suggests a more generally useful tool. They note that if the test could be made more reliable, perhaps by lengthening it beyond eight questions, it could be a valuable method for measuring “cognitive reflection separately from mathematical ability.”

For more information on faculty research, please reach out to ycci@som.yale.edu.