Beware Compelling Arguments

You can make a devastating case for anything

Edit 9/8 I have now realized the things I say have mostly already been said by Scott Alexander in his post epistemic learned helplessness.

Richard Hanania recently brought Dennis McCarthy (substack here) onto his podcast. McCarthy argues that Shakespeare wasn’t the original author of his plays, but instead that Shakespeare’s plays were mostly Shakespeare retrofitting earlier plays from Thomas North. Most of the evidence for this thesis comes from the fact that there are so many similarities between North’s writings and Shakespeare’s—one of the chapters in McCarthy’s book is titled “80 Shakespearean Passages Borrowed Nearly Verbatim from North.”

McCarthy seems like a smart guy—especially so because he subscribes to this substack!—and his case seems, on its face, devastating (it would be a big coincidence if Shakespeare and North used exactly the same phrases, often ones that were used by no one else before or since). This is not McCarthy’s entire case; he claims that there are lots of similarities between North’s life and themes that appear in Shakespeare’s plays.

Hanania concludes that “Either McCarthy has committed fraud, or the Shakespeare debate should have ended with the publication of his book.” This seems to be wildly overconfident. It reflects a fairly common error.

Now, McCarthy makes what appears to be an extremely powerful case, and it could very well be right. I’d love to see McCarthy debate someone with the mainstream view, and have, in fact, been trying to organize a debate like that (McCarthy is open, but I’ve been having trouble finding someone to represent the mainstream). But I think you should almost never be moved by a single case made for a position on a massively debated topic unless you are either a subject matter expert or have read the rebuttals. No matter how good it seems on its face, you shouldn’t be moved.

The problem is, it’s very easy to make a convincing-sounding case for anything.

I remember many years ago, reading Chomsky discuss some topic related to U.S. foreign policy. Chomsky’s claims were meticulously documented and footnoted, each backed up with what seemed like a wall of evidence. I concluded that Chomsky was demonstrably right—it seemed impossible for anyone to refute his conclusions.

Then I read a response someone wrote to him, arguing that Chomsky was completely nuts. That response was very convincing. Then, a fan of Chomsky wrote a response to the original criticism of Chomsky, and that was convincing as well. This went back and forth for several rounds, and by the end I had no idea what to think.

Richard Carrier is a crank. He thinks Jesus did not ever exist, not even as a real person in history, a position objected to by ~100% of historians for very good reason. But reading his case for his position, it sounds convincing at first. Only when you read his critics do you see how utterly loony it is. And the same thing is true on lots and lots of other subject; Ryan Dawson’s case for Israeli involvement in 9/11 sounds quite convincing (and was powerful enough to cause him to win a debate in about the most decisive fashion ever), but it turns out to be utterly devastated by a good debunking.

Examples of this abound—the Balko-Hughes disagreement is another example of a subject on which every progressive round of debunking seems decisive, until you read the one after it. Crucially, often a case will seem decisive unless its author practiced fraud, but only because it’s not intuitively obvious what a critic would say in response.

Imagine you were around in centuries before Darwin. The case for intelligent design would seem completely overwhelming—you might wonder how in the world one could go about refuting it. But that’s because the true solution is an unknown unknown not a known unknown.

Crucially, most subjects are like that. Most of the time, it’s not obvious to someone who isn’t a subject matter expert what form the right sort of rebuttals will take. Not only is it not obvious where the case actually does go wrong, it’s not even obvious where it might go wrong.

I’ve been mislead too many times in my life to be confident in pretty much any case for anything, until I’ve either read the rebuttals or become a subject matter expert. I’m not the only one—David Friedman has expressed the same idea. Even a prima facie devastating case isn’t something I think a person should update on too much! While McCarthy could very well be right, I certainly wouldn’t bet the farm on it until I’ve heard what his critics have to say.

I’m not intending to single out McCarthy here—I’ve used him as an example because his case seems particularly compelling and he seems particularly sharp. There are many people who make compelling cases for things that I don’t know how to refute. As one should be careful about drawing conclusions based on one study, one should be careful about drawing conclusions based on one case—no matter how devastating it appears.