To get to the moon, we didn’t build two groups of rockets and see which group made it to orbit
A quote from a new paradigm on psychology, and a story about the three different ways to figure out how the world works.
He starts by defining the term ‘Paradigm’, which is already a very helpful thing to have in your arsenal whenever someone in a random work starts talking about paradigm shifts:
a paradigm is made out of units and rules. It says, “the part of the world I’m studying is made up of these entities, which can do these activities.”
In this way, doing science is a lot like reverse-engineering a board game. You have to figure out the units in play, like the tiles in Scrabble or the top hat in Monopoly. And then you have to figure out what those units can and can’t do: you can use your Scrabble tiles to spell “BUDDY” or “TREMBLE”, but not “GORFLBOP”. The top hat can be on Park Place, it can be on B&O Railroad, but it can never inside your left nostril, or else you’re not playing Monopoly anymore.
A paradigm shift is when you make a major revision to the list of units or rules. And indeed, when you look back at the biggest breakthroughs in the history of science, they’re all about units and rules. Darwin’s big idea was that species (units) can change over time (rule). Newton’s big idea was that the rules of gravitation that govern the planets also govern everything down here on Earth. Atomic theory was a proposal about units (all matter is made up of things called atoms) and it came with a lot of rules (“atoms always combine in the same proportions”, “matter can’t be created or destroyed”, etc.). When molecular biologists figured out their “central dogma” in the mid-1900s, they expressed it in terms of units (DNA, RNA, proteins) and what those units can do (DNA makes RNA, RNA makes proteins).
As usual, Adam is a master of analogies. The monopoly example comes back later, to summarize the three ways to do science:
Naive research: When you do pure empirical tests without having any clue about the units or rules.
Impressionistic research: you make up a bunch of hand-wavy words and then study them.
Making guesses about the units and rules, and then checking whether they hold up.
On Naive Research:
The name is slightly pejorative, but only slightly, and for good reason. On the one hand, some proportion of your research should be naive, because there’s always a chance you stumble onto something interesting. If you’re locked into a paradigm, naive research may be the only way you discover something that complicates the prevailing view.
On the other hand, you can do naive research forever without making any progress. If you’re trying to figure out how cars work, for instance, you can be like, “Does the car still work if we paint it blue?” *checks* “Okay, does the car still work if we...paint it a slightly lighter shade of blue??”
(As SMTM puts it: “To get to the moon, we didn’t build two groups of rockets and see which group made it to orbit.”)
I like that last quote so much I made it the title of my post, because often times we do need to do naive research in business too, e.g. in marketing. In cases where we need to know an effect compared to a counterfactual. I wrote about that 2 weeks ago:
But, indeed, you won’t get to the moon that way. To get to the moon, we’ll need to figure out the rules of gravity, combustion, aerodynamics, etc and calculate how big of a rocket we need to build to get us to the moon1. If you can make a good prediction for how the rules work, you’ll try things from a much more informed place2.
The part about Impressionistic research is the funniest, and for anyone who has read pseudoscience books like Freakonomics, or Outliers, or any Psychology research, it will be recognizable instantly:
If you’re studying whether “action-awareness merging” leads to “flow”, or whether students’ “math self-efficacy” mediates the relationship between their “perceived classroom environment” and their scores on a math test, or whether “mindfulness” causes “resilience” by increasing “zest for life”, you are doing impressionistic research.
The problem with this approach is that it gets you tangled up in things that don’t actually exist. What is “zest for life”? It is literally “how you respond to the Zest for Life Scale”. And what does the Zest for Life Scale measure? It measures...zest for life. If you push hard enough on any psychological abstraction, you will eventually find a tautology3 like this. This is why impressionistic research makes heavy use of statistics: the only way you can claim you’ve discovered anything is to produce a significant p-value.
Almost all of post-modernism was about using very difficult words and sentence structures to make it as hard as possible to figure out that the text actually was entirely tautological, and made no statements about anything4. Unfortunately, I think post-modernist influences are still quite large in certain fields.
Here is the monopoly analogy, applying both methods to the problem of understanding the game:
Naive and impressionistic research are often respectable-looking ways to go nowhere. For example, if you were trying to understand Monopoly using the tools of naive research, you might start by correlating “the number that appears on the dice” with “money earned”. That sounds like a reasonable idea, but you’d end up totally confused—sometimes people get money when they roll higher numbers, but sometimes they roll higher numbers and lose money, and sometimes they gain or lose money without rolling at all. These inconsistent results could spawn academic feuds that play out over decades: “The Monopoly Lab at Johns Hopkins finds that rolling a four is associated with an increase in wealth!” “No, the Monocle Group at UCLA did a preregistered replication and it actually turns out that odd numbers are good, but only if you fit a structural equation model and control for the past ten rolls!”
The impressionistic approach would be even more hopeless. At least dice and dollars are actual parts of the game; if you start studying abstractions like “capitalism proneness” and “top hat-titude”, you can spin your wheels forever. The only way you’ll ever understand Monopoly is by making guesses about the units and rules of the game, and then checking whether your guesses hold up. Otherwise, you might as well insert the top hat directly into your left nostril.
The last very important point on paradigms are that the units and rules need to be at the right level to be useful:
We’re going to get to psychology in a second, but first we have to avoid a very tempting detour. Whenever I talk to people about the units and rules of psychology, they’re immediately like, “Oh, so you’re saying psychology should be neuroscience. The units are neurons and—”
Lemme stop you right there, because that’s not where we’re going.
Let’s say you’re trying to fix the New York City transit system, so you’re thinking about trains, stations, passengers, etc. All of those things are made of smaller units, but you don’t get better at designing the system by thinking about the smallest units possible. If you start asking questions like, “How do I use a collection of iron atoms to transport a collection of carbon and hydrogen atoms?” you’ll miss the fact that some of those carbon and hydrogen atoms are in the shape of butts that need seats, or that some of them are in the shape of brains that need to be told when the train is arriving.
The post then dives into the actual content that puts forward a potential new paradigm for psychology, that behavior is ruled by thermostats. Some of it makes sense, but as Adam also writes it is going to be mostly wrong. The very important point he makes, and the one I wanted to most amplify, is that a wrong theory is much, much better than no theory, or a tautological one. It made me think more in terms of units and rules, and remind me that this ‘mechanistic’ approach can be a very helpful way to approach problems in the world.
Other stuff
Speaking of Rockets, I watched Fly Me to the Moon and found it an extremely fun and wholesome movie. Great romantic comedy, not too long. Recommend!
Grok (AI) started bringing up White Genocide in almost every reply, and here is an analysis of that.
Read up on Tsiolkovky’s Rocket Equation if you want to learn some Rocket Science
A/B testing can still be important when you try to validate your theory
Trying to read the wikipedia page of Tautology reminded why logic is only fun up to a point. But the upshot is that a tautology is always true, you cannot falsify it, e.g. “the ball is green or not green.” That statement does not actually say anything about the ball’s color.
I have a very short list of things I very deeply despise: cynicism, de-growth and post-modernism. See also, the Sokal Affair, which I love