Skeptical about an AI-related comment? Use the calculator test.

The process is easy: replace the word “AI” (or whatever they are using) with “calculator” and see what it does to the comment.

And the reason it works is simple. We already know that the introduction of the calculator did not destroy humanity. It didn’t reduce the number of jobs, or make people less smart, or create any of the cataclysmic outcomes that skeptics at the time claimed it would.

This also works for positive comments. We also know that calculators didn’t propel us into a massive global wave of prosperity, where we all sit around making art and exploring the universe.

So if the argument being made against AI is the same as the one being made about the calculator, it is likely to be specious and can safely be ignored. This is a handy heuristic that allows you to focus on thoughtful, nuanced critiques of AI that are actually important.

Here’s an example of the calculator test run on the abstract from a recent (ridiculous, poorly run, self-report) paper about AI and critical thinking. The only change is a find/replace on “GenAI” and “a calculator”:

“The rise of Generative AI (a calculator) in knowledge workflows raises questions about its impact on critical thinking skills and practices. We survey 319 knowledge workers to investigate 1) when and how they perceive the enaction of critical thinking when using a calculator, and 2) when and why a calculator affects their effort to do so. Participants shared 936 first-hand examples of using a calculator in work tasks. Quantitatively, when considering both task- and user-specific factors, a user’s task-specific self-confidence and confidence in a calculator are predictive of whether critical thinking is enacted and the effort of doing so in a calculator-assisted tasks. Specifically, higher confidence in a calculator is associated with less critical thinking, while higher self-confidence is associated with more critical thinking. Qualitatively, a calculator shifts the nature of critical thinking toward information verification, response integration, and task stewardship. Our insights reveal new design challenges and opportunities for developing a calculator tools for knowledge work.”

See how it sounds both cogent and ridiculous at the same time? That is a sign that this paper can be safely ignored. It reads like a debate from the 70s, where some well-meaning alarmist predicts the death of critical thinking by arguing that people’s degree of trust in a calculator is proportional to their ability to think critically.

The argument is internally consistent and hangs together, and yet we know it to be false: calculator usage doesn’t meaningfully reduce critical thinking or turn us into information verification machines. Instead, calculators freed up a tremendous number of very smart people to do very smart things instead.

Want to try a paper worth paying attention to? Let’s take an abstract from one by Timnit Gebru.

“Rising concern for the societal implications of calculators has inspired a wave of academic and journalistic literature in which deployed calculators are audited for harm by investigators from outside the organizations deploying the calculators. However, it remains challenging for practitioners to identify the harmful repercussions of their own calculators prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source.

In this paper, we introduce a framework for algorithmic auditing that supports calculator development end-to-end, to be applied throughout the internal organization development life-cycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization’s values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale calculators by embedding a robust process to ensure audit integrity.”

Notice how this one sounds ridiculous? That’s because Gebru isn’t making a claim about calculators; she and her co-authors are saying something meaningful that is unique to artificial intelligence systems. And so you know this paper isn’t specious and should be read in full.

AI is important and so are many of the debates about how it affects our future. But there is a difference between AI being important and every comment made about AI being important. Learning how to filter AI-based arguments is key so TI-83 it and have some fun!