Thought-Terminating Jargon

On thought being a vehicle for language rather than language being a vehicle for thought

(Reminder: congress might soon obliterate most state animal welfare protections, being the largest setback for animals in history. You can help stop it! Call your senator and donate! If you’re interested in making a substantial donation but want more information, feel free to get in touch via substack dm or at untrappedzoid@gmail.com.)

Richard Feynman famously said that you don’t really understand something unless you can explain it to a five-year-old. This is false. I have yet to hear anyone explain to a five-year-old the justification for this claim. Similarly, I’m pretty sure that I understand the argument for the self-indication assumption, but a five-year-old would never get it. I sometimes wonder whether the people who say this have ever interacted with a five-year-old.

But there is a grain of truth to Feynman’s claim nonetheless. It’s very easy to get lost in abstraction and formalisms—to think you understand some subject, when you really have only memorized a few nebulous technical phrases to say about the subject. It is easy to hide confusion beneath a formulaic mask.

High-school debaters are fond of predicting imminent nuclear war unless their preferred policy passes. Their opponents often reply that mutually assured destruction will prevent a nuclear war from starting—neither side will use nuclear weapons, because if they do, the other side would use them too. In response, the people who are arguing that a nuclear war will start unless you pass their obscure change to farm policy or whatever generally say something like “a nuclear war could start—mutually assured destruction doesn’t check due to accidents and miscalculation.”

I said this line many times. And at some point, I realized that I hadn’t the foggiest clue what miscalculation even meant in this context. I had a fancy word to say about why mutually assured destruction wouldn’t stop a nuclear conflagration, but that’s all it was: a word. There was no deeper idea that I understood. All I had was thought-terminating jargon.

Thought-terminating jargon is where you have some fancy word to say about a subject which takes the place of deep understanding. Like thought-terminating cliches, it allows a person to avoid thinking deeply about a subject without realizing it. A slogan plays the stunt double of real thought.

One good example of this is the term “emergence,” which crops up frequently in explanations of how a mind emerges from matter. There are serious puzzles about how consciousness emerges from non-conscious matter. The physical world is exhaustively defined in terms of behavior. But consciousness isn’t about behavior. It’s about having inner experience, where there’s something it’s like to be you. It’s not clear how behavior alone produces this. Behavior may beget more complex behavior, but it’s not at all clear how it gets you qualia. This is a major reason why so many people think that consciousness is irreducible.

Now, maybe this whole line of thinking is confused. Perhaps there’s some subtle way to get experience from non-experience. Perhaps the entire way of thinking is a bit confused, and some feature of our conscious experience is illusory in a deep sense. But at the very least, you’ll have to say something about this puzzle. There’s a real worry here, considered decisive by a great many able philosophers, and you can’t just handwave it away.

Very commonly, however, people will simply declare that consciousness emerges, as if this is a total explanation. A emerges from B when A’s behavior arises from the behavior of B. Planets emerge from the laws of physics, for standard operations of the laws of physics naturally give rise to planets. To say that consciousness emerges from the physical isn’t any kind of explanation. It’s a thing you can slot in to replace your confusion—so that your puzzlement about getting experience from behavior is paved over by the phrase “it emerges.” As Eliezer Yudkowsky put it:

The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?

I name emergence or emergent phenomena—usually defined as the study of systems whose high-level behaviors arise or “emerge” from the interaction of many low-level elements. (Wikipedia: “The way complex systems and patterns arise out of a multiplicity of relatively simple interactions.”)

Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a market crash and saying “It’s not a quark!” Does that feel like an explanation? No? Then neither should saying “It’s an emergent phenomenon!”

Perhaps there is no group more prone to thought-terminating jargon than continental philosophers. Continental writing is famously nebulous. Instead of having a concrete and stateable point, there is often some hazy mishmash of words that gestures in the vicinity of some sentiment, but doesn’t quite state anything specific. For example, from Derrida:

The trace is not a presence but is rather the simulacrum of a presence that dislocates, displaces, and refers beyond itself. The trace has, properly speaking, no place, for effacement belongs to the very structure of the trace. . . . In this way the metaphysical text is understood; it is still readable, and remains read.

As best I can tell, “the trace” refers to the fact that a word’s meaning comes from it being different from other words. This claim—that for a word to mean something is for it to differ from different words that mean other things—is not very interesting. Yet because Derrida is speaking unclearly, this sounds deep, and he is able to follow it up with many other claims that have, at best, highly uncertain meaning. It is easy to fool yourself into thinking that some claim is profound when you aren’t quite sure what it says. If you make a point using words that are vague and contain many letters, it is harder to notice that it is trivial or implausible.

A while ago, Dylan Black wrote a piece titled A Measured Response to Bentham’s Bulldog. The article raised the measure problem in response to fine-tuning—worrying about whether the space of possible worlds was well-defined (see my response to the objection here). In it, Dylan complained that I didn’t specify the sigma algebra in my blog post over which I was calculating probabilities.

Now, the measure problem is a fairly standard objection to fine-tuning. There are a number of responses to it that have been given which are, in my view, decisive. What I mostly didn’t like about the article was that I felt like most of the readers were getting thought-terminating-formalism-sniped. Rather than getting a gears-level understanding of an objection, people simply got some hazy understanding of measure theory, and confident assurance that it totally dissolves the fine-tuning problem. I like how Scott Alexander put it:

(as a less legible objection, I’m nervous that work like this is taking something that’s obviously true - it’s really weird that these constants are in the tiny realm suitable for life - and merely raising enough objections that you’re not allowed to talk about it unless you have ten years of postgraduate math. I think in order to convince me, you would need to find a way to intuitively show me how this actually defuses the surprisingness rather than just lets you pounce on anyone who tries to discuss it rigorously.)

Speaking of Scott Alexander, is this just Eulering? Have I just reinvented Eulering under a new name? Getting Eulered is getting bullshitted by math. Here’s the canonical example Scott gives of Eulering:

Euler said, in a tone of absolute conviction: “Monsieur, (a+b^n)/n = x, therefore, God exists! What is your response to that?” and Diderot, “for whom algebra was like Chinese”, had no response. Thus was he publicly humiliated, all the Russian Christians got an excuse to believe what they had wanted to believe anyway, and Diderot left in a huff.

But what I’m talking about is a bit different. Rather than being when someone else bullshits you with math, it’s when you bullshit yourself with math (or other technical vocabulary). It’s when you get some hazy understanding of a technical concept, and use it as an excuse to shut your brain off. Whenever you feel the faint clouds of confusion forming on the horizon, you repeat the phrase “sigma algebra,” three times in front of a mirror, and your feeling of bafflement disappears.

In Politics and the English Language, Orwell discusses how easy it is to defend the indefensible through nebulous rhetoric. It’s easy to defend some misconduct if you don’t have to discuss specifically what the conduct is, and can instead simply say abstract things about it. I think thought-terminating formalisms are related to this—perhaps Orwell’s idea had a daughter out of wedlock with Eulering, and thought-terminating formalisms were the result.

In both cases, specific analysis is replaced by some hazy and vaporous outline of an idea. In Orwell’s words “A mass of Latin words falls upon the facts like soft snow, blurring the outlines and covering up all the details.” He elsewhere describes “This invasion of one’s mind by ready-made phrases (lay the foundations, achieve a radical transformation) can only be prevented if one is constantly on guard against them, and every such phrase anaesthetizes a portion of one’s brain.”

Humans engage in extremely motivated reasoning. We are always on the lookout for some respite from the other side’s argument—some consideration that will, once uttered, mean we no longer need to keep thinking about our opponent’s best argument. Abstractions and formalisms are particularly good for this, for they allow us to think confidently that the other side is wrong, without needing to understand exactly why. The more nebulous an idea is, the more formless its outline, the easier it is to delude yourself into thinking it eviscerates the other side’s best arguments. Vagueness has a chameleonic quality—a vague argument can take many forms, and it’s easy to talk yourself into thinking that the form it takes is whichever is most dialectically convenient.

Now, of course, there’s an opposite error. It’s easy to never update from evidence that you only have a hazy grasp of. If you do this, you’ll remain dogmatically locked into whichever side seems best according to the minuscule subset of the considerations that you can grasp easily. There’s no easy rule that can be followed to avoid both classes of error.

But it’s important to be on the lookout for these errors. Thought-terminating formalisms are, in my experience, relatively common. They cause many people to form false beliefs and dismiss obvious considerations—hidden in the fog of abstraction.