New Paper Proves That Everyone Has to Say Something Weird About Cases Where There’s a Very Low Probability of Something Very Good

It isn't just a problem for utilitarians

Introduction

In an excellent new paper, Beckstead and Thomas show that everyone has to say weird things about cases where there is a very low probability of very high payoffs. People often think that this is just a problem for utilitarianism—but this is wrong, as Richard Chappell notes:

In an especially striking example of conflating utilitarianism with anything remotely approaching systematic thinking, popular substacker Erik Hoel recently characterized the Beckstead & Thomas paper on decision-theoretic paradoxes as addressing “how poorly utilitarianism does in extreme scenarios of low probability but high impact payoffs.” Compare this with the very first sentence of the paper’s abstract: “We show that every theory of the value of uncertain prospects must have one of three unpalatable properties.” Not utilitarianism. Every theory.

(Alas, when I tried to point this out in the comments section, after a brief back-and-forth in which Erik initially doubled down on the conflation, he abruptly decided to instead delete my comments explaining his mistake.)

The basic idea of the paper is as follows: suppose that one is going to get something good, for example, ten years of happy life. Someone offers them a deal: they’ll get ten times as many years of life and ten times as much pleasure over the course of each year in exchange for reducing the probability of the payouts from 100% to only 99.9999%. They of course take that deal—who wouldn’t? Then the person offers the deal again, where the payouts increase 100-fold and the probability is 99.9999% of getting the first deal. This is an improvement again. It seems like no matter how many times they do this, it keeps being an improvement, but if they take the deal enough times, they end up with a vanishingly low probability of a very high reward. But it doesn’t seem like the thing at the end—the very low probability of something very good—is better than the thing at the beginning, a very happy life. It does not, for example, seem like one should gamble away their life for a 1/100^100^100^100^100 chance of getting graham’s number utils (graham’s number is a really big number).

Thus, Beckstead and Thomas show that every view must be either reckless, meaning that it would support gambling away any good for any low probability of a sufficiently large good; timid, meaning that it would support passing up any arbitrarily large increase in reward at the cost of a vanishingly low decrease in the probability of the reward; or intransitive, meaning that it holds that when one prefers A to B and B to C, they don’t necessarily have to prefer A to C. This is for a simple reason: if each trade is better than the last one and transitivity is true, then the final gamble with a very low probability of very high payouts will be better than the starting guarantee.

Beckstead and Thomas don’t really talk about giving up transitivity in the paper, noting that there’s already extensive literature on it. I’d rather give up on all of ethics than give up transitivity—I basically have no stronger intuitions than my intuition of transitivity, and I think there are very strong arguments for accepting it. So I don’t think that just biting the bullet and rejecting transitivity is acceptable. It seems that most people share my intuition and agree that we should not reject transitivity.

So…recklessness or timidity. I think recklessness is the way to go, and here I’ll explain why. I think that while recklessness is pretty unintuitive, it’s way more plausible than timidity or intransitivity.

Timidity

Beckstead and Thomas note that one might initially motivate timidity by accepting bounded value. One might claim that as the amount of pleasure goes to infinity, the intrinsic value of that approaches a finite asymptote. Thus, there will be some threshold of pleasure where one shouldn’t take the deal—where an increase in pleasure by 1000000000000 at the cost of diminishing the probability of payouts by .000000000000000000000000000000000000000000001% is not worth it.

However, this is very implausible on the negative end. It is not plausible that being tortured for an extra 100 years tends towards have no marginal disvalue as one’s previous suffering tends towards infinity. Thus, defenders of this view would have to bite the bullet on the inverse argument—that a guarantee of googolplex years being tortured is less bad than a 1/googolplex chance of being tortured for graham’s number years (graham’s number is much, much, much larger than googolplex).

One might motivate timidity by arguing for discounting low risks. Discounting risks below some threshold is called Nicolausian discounting. But this is very implausible for two reasons. First, we can imagine a range of possibilities where each lies below the threshold, and Nicolausian discounting would mean all of them should be ignored. For example, suppose that some different bad things would happen depending the exact order of a deck of cards. Nicolausian discounting would say to ignore all risks, even though there’s a 100% chance of one of them occurring. Secondly, it implies a puzzling form of hypersensitivity—suppose one ignores risks below 10^-50. Well, this would mean that risks slightly above 10^-50 probability are treated as infinitely more significant than risks slightly below it. But this is bizarre—small changes in probability shouldn’t make differences this large.

Instead of Nicolausian discounting, we can adopt tail-risk discounting, which gives disproportionately low weighting to things that are unusually extreme in value—so high-risk high reward things would be discounted.

Now, one initial obvious problem with discounting is that it has to accept that there is some payoff such that increasing the payoff 1 trillion fold while only decreasing the probability slightly is not an improvement. But the view has other problems, arguably even more severe. The first problem is the following:

Consider, for the sake of concreteness, the prospect 𝑃of saving 1,000 lives with probability 0.1. A theory that is timid with respect to lives saved might say that 𝑃 is no worse than the prospect of saving even a vast number of lives with probability 0.099, only one percent smaller. We can imagine that the outcome of 𝑃 depends on the outcome of a raffle: if the golden ticket is drawn, then a mechanism will be activated that will save 1,000 lives. Now consider a prospect 𝑃1 that differs from 𝑃 in two ways: first, the mechanism is designed to save many more lives—let’s say 10,000; but second, there is a one-percent failure rate, and in the fail-state, 𝑃1 saves only 999 lives. So 𝑃1 almost certainly leads to ten times as many lives saved as in 𝑃, and at worst, in the highly unlikely fail-state, it will save only one life fewer. Intuitively, 𝑃1 is much better than𝑃. But now consider a prospect 𝑃2. In 𝑃2, the mechanism usually saves 100,000, but it only saves 998 in the fail-state. So, compared to 𝑃1, there are almost certainly 10 times as many lives saved, and at worst one fewer. 𝑃2 is clearly better than𝑃1, so also better than𝑃. But if we continue this sequence along, we get to𝑃1000, in which no lives are saved in the fail-state. Thus 𝑃1000 leads to a vast number of lives saved (namely,10^1003) with probability 0.099. By transitivity, 𝑃 1000 is better than 𝑃. And so, the argument concludes, our theory of value must not be timid in this particular way. Nor is there anything special about this case; mutatis mutandis, we have an argument against timidity in general

Second, timid views result in strange dependence on things that are going on in distant times and spaces. This is easy to see if it arises from bounded value—for example, suppose, for simplicity, that as the amount of utility goes to infinity, the value of that tends towards 1 quadrillion intrinsic value units. This would mean that if people in distant galaxies have lots of utility, they might bring us almost to the cap, meaning that billions of people here and now living great lives would be barely valuable at all. This is obviously false.

Now, suppose that one motivates timidity by tail-discounting, such that one discounts particularly extreme values. Well, particularly extreme values for the universe depends on things that are far away. To see this, consider the following payoff matrix.

Thus, A leads to n extra lives with probability p, B leads to n extra lives with probability p+q, and C leads to n+N lives with probability p. Suppose that p is a trillion times greater than q and N is a trillion times greater than n. The timid view has to accept that for some values, C will not be better than B. But now suppose that the n lives with probability p are in another galaxy and are not affected by the decision. If the stuff happening a galaxy away couldn’t matter, then this payoff would be equivalent to the following: A leads to nothing, B leads to n with probability q, and C leads to N with probability P. Thus, C leads to a trillion times greater chance of a trillion times more increase in value, but C is less valuable by virtue of unaffected stuff a galaxy away.

Now, one might object and say that what matters is the probability of the outcomes affected by some act. Thus, if an act does not affect the probability of the people a galaxy away living their good lives, then that doesn’t affect what gets discounted. But this leads to a problem (one not pointed out in the paper). This leads to the result that two gambles that produce identical payouts in all circumstances become differently valuable. For example, suppose in the previous case that A, B, and C each affect the people a galaxy away, but they lead to exactly the same outcomes as they would have in the earlier scenario—no matter whether A, B, or C happens, the people a galaxy away will exist with probability P. So no matter whether A, B, or C is taken, the people a galaxy away will exist with probability P. This view holds that, though this results in the same outcomes in all possible scenarios as in the original gamble where the people a galaxy away are unaffected, this radically changes decision-making. This is obviously wrong.

The third problem for these views is that they imply extreme risk aversion in very positive cases. This is easy to see with the bounded utility view—for example, suppose the bound of where utility stops mattering is at 10^10 years of happy life. This would mean that if given two deals, one of which gives you 10^10 years of happy life if heads and 1 hour of misery if tails, while the other gives you 10^80 years of happy life if heads and 1 hour and 1 second of misery if tails, the first deal is better. This is hard to believe.

However, the problem also arises for tail and Nicolausian discounting. On this view, there is some probability threshold such that the value cannot be outweighed by an increase in the value by a trillion-fold, at the cost of a slightly diminished probability. For example, if the threshold is .05, then the probability of .05 of 10,000 years of bliss would be more valuable than the probability of .04999999999999999999999 of 100^100^100 years of bliss. This is hard to believe.

The problem is even worse for negative discounting. If the threshold for negative discounting is .05, then being tortured for 10,000 years with probability of .05 is worse than being tortured for 1000000000000000000000000000000000000000000000000 years with probability of .049999999999999. This is obviously false.

The fourth and final problem is that timid views pay way too much attention to longshot bets in ways that are weird and objectionable. This is easy to see with tail discounting—suppose that the threshold is .00000000001%. This would mean that a .00000000001% probability of a sufficient payout is better than any sure bet very high payout. This is hard to believe, because it means that timid views imply a conclusion very similar to the reason people reject reckless views. If we adopt a bounded utility view, we get the following objectionable result

Lingering doubt: In Utopia, life is extremely good for everyone, society is extremely just, and so on. Their historians offer a reasonably well-documented history where life was similarly good. However, the historians cannot definitively rule out various unlikely conspiracy theories. Perhaps, for instance, some past generation ‘cooked the books’ in order to shield future generations from knowing the horrors of a past more like the world we live in today.

Against this background, let us evaluate two options: one would modestly benefit everyone alive if (as is all but certain) the past was a good one; the other would similarly benefit only a few people, and only if the conspiracy theories happened to be true.

On the bounded views, the second gamble would be better, because even though it gives lower payouts with lower probability, it only applies if we’re far away from the cap, so it has higher expected value. This is hard to believe.

I basically think that these comments comprise a totally knockdown proof. I’d rather give up on ethics than accept timidity—it implies so many conclusions that are so obviously false.

Recklessness

Beckstead and Thomas note that the initial intuition that one shouldn’t give up very good things for a vanishingly low probability of much better things is potentially subject to debunking. I’m inclined to agree. However, they argue that recklessness entails other things that are even more implausible.

For example, consider the Saint Petersburg gamble. This occurs when one flips a coin until they get tails. If they get tails on the first try, they will get 2 units of utility, second try they’ll get 4, third try 8, etc. This has infinite expected value—there is a 1/2 chance of 2 utility, a 1/4 chance of 4 utility, a 1/8 chance of 8 utility, etc. But despite this, it is guaranteed to be finite—one cannot get infinite coin flips. This means that it violates the following obvious principle:

Prospect-outcome dominance: If prospect 𝐴 is strictly better than each possible outcome of prospect 𝐵, then 𝐴 is strictly better than 𝐵

The gamble has infinite expected value, even though each result it could get is finite. Thus, Prospect-outcome dominance would entail that it is better than itself, which it is obviously not. Now, I’m inclined to reject the possibility of the Saint Petersburg gamble because I don’t think you can have concrete infinite amounts of things and I accept the B theory. This is a more complicated view, and I don’t want to discuss it much here—I’ll write a future article about it—but I think it allows us to avoid the paradox and to accept Prospect-outcome dominance. Huemer in his book Approaching Infinity gives a different way to accept Prospect-outcome dominance and rule out the Saint Petersburg paradox. So this view is not a challenge for any view that I accept. Beckstead and Thomas show that any view that is reckless that accepts the possibility of the Saint Petersburg scenario must reject the following principle:

Outcome-outcome dominance: If, no matter what, the outcome of 𝐴 would be at least as good as the outcome of 𝐵, then 𝐴 is at least as good as 𝐵.

But because I don’t think the Saint Petersburg gamble is possible, this isn’t a problem for my view.

Next, Beckstead and Thomas argue that believers in recklessness must be infinity obsessed in the sense that they value any small chance of an infinite payoff over any finite payoff. This means that they should dedicate their entire lives to slightly increasing the odds of infinite payoffs. This also means one should be tortured for 10 trillion years in exchange for a sufficient Saint Petersburg gamble, which is hard to believe. Now, I just think these scenarios are all impossible, so it’s not a threat to me, and more extreme finite scenarios get exponentially less likely, meaning that our decisions shouldn’t be dominated, in practical situations, by extreme long shots.

Next, they give a scenario where some civilization can either live bad lives while dedicating all their time to research on how to get infinitely good utopia—despite it having vanishingly low probability—or just have a very good utopia. It seems the very good utopia is better than the almost certainly drab world with a slight chance of infinity, but this view must deny such a conclusion.

Now, I deny the possibility of infinity. But I’d agree that there would be some finite probability of a sufficiently good outcome that would be worth sacrificing utopia for. This seems unintuitive to me. But I’m inclined to think that this principle is subject to the kinds of debunking arguments that Thomas and Beckstead are sympathetic to. And I think that, though this is unintuitive, we basically have a proof of it—based on just how untenable the other options are.

Conclusion plus one more argument for recklessness

Before I conclude, let me present one more argument for recklessness. The following principles are plausible.

1. For a gamble to affect the worthwhileness of another gamble, it must affect either the payouts of the gamble or the probabilities associated with it.

2. If one is taking a sequence of gambles none of which affect the desirability of the other gambles, they should take the best gamble at each stage of the sequence. (So, for example, if gamble A is better than B, and C is better than D, and one can choose A or B then C or D, they should take A and C).

3. An iterated sequence of reckless gambles is better than an iterated sequence of timid gambles.

But these require we accept that recklessness is better than timidity, because the sequence of gambles which don’t affect the desirability of each other that are each reckless turns out better than the sequence of timid gambles, which means each reckless gamble is better than each timid gamble. 1 and 2 are both very plausible, and three is obvious. To see this, suppose there are two gambles, one of which gives one a 1 in googolplex chance of graham’s number utils, and the other gives a guarantee of 1 quadrillion utils. Well, if these are iterated tree 3 times (1 quadrillion is basically zero compared to graham’s number which is basically zero compared to tree 3), then the person who takes the graham’s number gambles over and over again is almost guaranteed to get unfathomably higher payoffs.

So Beckstead and Thomas are right—there is a puzzle for everyone. Everyone has to say something utterly bizarre about these longshot gamble cases. But because everyone has to accept them, they’re not a good reason to reject utilitarianism. I think in order of plausibility, recklessness beats timidity which beats intransitivity. But this is a real puzzle and the way out is not obvious. I’d probably give up on ethics before I accepted intransitivity or timidity.