Compared to what? Getting back to base with claims

A first step to take when we encounter a claim is to determine if it is consequential. If believing or acting on a claim could affect people’s lives, then the claim is consequential.

The second step is to take a closer look at the claim being made and determine what evidence would be needed to support or substantiate it.

A claim that one thing causes another requires a higher evidentiary bar than a claim that suggests an association between two things. A claim like: “this nighttime supplement will improve your sleep by 30%” calls for robust scientific evidence, including a defined measure of sleep, a relevant comparison group, and evidence collected over a meaningful period of time.

While we often focus on individual claims to evaluate, we can also batch claims to discern patterns. Consider these three claims:

• “AI makes better writers.”

• “Teen anxiety is getting worse.”

• “This supplement gives you more energy.”

Any pattern to notice here? Here are three more:

• “This app is safer for younger learners.”

• “Students learn 25% faster with video.”

• “Social media makes students less focused.”

What unites all of these examples is that they imply comparisons; each employs a comparative word (better, worse, more, safer, faster, less) to claim something is different (often better or worse) than something else. This something else is the baseline for the comparison, which could be another group or intervention, an earlier period, or someone’s situation before a change or intervention.

When this baseline is missing, a claim might sound convincing, but it’s difficult or impossible to evaluate. Let’s figure out why. We start by asking: Compared to what?

Example: AI improves students’ writing

Here’s a consequential claim: “AI makes students better writers.” We ask:

AI makes students better writers compared to what? Is it:

• better than writing without any help or assistance?

• better than getting feedback from a teacher, parent, or peer?

• better in terms of grades, organization of ideas, expressive quality, or particular skills?

• better for all students or only some? (this one is compared for whom?)

Taking this step highlights how different the claim might be and what type of evidence would be needed to support it. The first two questions work with a comparison baseline, while the next one is about the outcome and the last is about the students to whom the claim applies.

There is also a key difference between a claim that AI improves particular pieces of writing and a claim that AI helps students become better independent writers.

What’s important is how working through these questions helps clarify whether or not the claim can even be evaluated. A claim, for example, about how a specific AI tool helped one group of high school students improve their overall writing scores after a particular classroom intervention can be evaluated in light of the evidence provided to support it.

If the research was conducted with sound methods in a systematic and rigorous way, the claim might be convincing. But if questionable methods were used, key terms weren’t defined clearly (writing “score”?), or the sample included just a few students, the claim would be less convincing. Working through these issues is how we flex our critical thinking muscles.

Taking a step back also helps us make a key critical thinking move, as we keep comparison in mind. In addition to Compared with what?, we also can ask:

Compared for whom? Compared over what period? How large or small is the actual difference? Is the difference even meaningful?

If none of this information or data is made available to us, then there isn’t much to evaluate even if relevant evidence might exist elsewhere.

Getting back to base

In a previous post I highlighted a simple thinking strategy, how going for the N-E-T reminds us to intentionally Notice —> Evaluate —> Test our thinking. Applied to comparison claims it looks like:

Notice the comparative language and baseline provided or missing

Evaluate the baseline and size of the difference

Test the claim against relevant sources with evidence that directly addresses the comparison being made

🌊 Ready to practice?

Choose an online post: could be a headline, ad, or even an AI response. Identify comparison words like better, safer, faster, healthier, or more effective and determine whether a baseline is provided and if the comparison actually matters.

Takeaway

Comparative claims are common online, and we don’t need to reject all of them. The goal is to identify information that can help us responsibly interpret them. We do that by identifying the comparison, clarifying its baseline, and considering what evidence would support that specific claim.

🤝 Connect & Share!

If this newsletter could help a colleague, parent, or student, or anyone you know committed to understanding how to future-proof our thinking, please share it with them.

To thinking well together!

James