For A Deep Reflection

The most important decision that humanity will ever make

(Impact First is having a generator program, in London, designed to help early-career professionals. Consider applying!)

If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.

—Toby Ord, “The Precipice.”

Future humans will make profoundly important decisions. On Earth today, our options are largely limited by biology and sparse resources. But if we don’t go extinct, we will have the ability to venture out across the cosmos and transform it from a mostly barren void to an intergalactic bliss-filled utopia.

In his book “The Precipice,” Toby Ord proposes that before we do this we carry out a Long Reflection. Before we transform the universe on a galactic scale, he proposes a long and protracted collective planning process, where we carefully think through how it should be transformed. If our actions will determine the fate of incredible numbers of conscious organisms, across a sextillion star systems, across a trillion years, then it is of literally astronomical importance that we not screw things up.

Now, Ord has explained that the Long Reflection wasn’t intended as a super exact proposal. Getting progress to freeze for 1,000 years as we sit around in armchairs and reflect isn’t tenable. But still, he thinks we should aim for something in the vicinity of a Long Reflection. So do I. We shouldn’t permit major transformation of the universe to happen through the accidental and unplanned decisions of private actors. We should reflect before spreading across the cosmos and transforming it.

And the good news is that something resembling a Long Reflection wouldn’t take that long. As my colleagues at Forethought argue in Preparing for the Intelligence Explosion, once AI can automate cognitive labor, enormous amounts of cognitive labor can occur very quickly. Given falling costs of compute and improvements in AI capabilities, at some point AI-assisted reflection might be able to drive much more progress in a few years than humans could in 1,000. So a “Deep Reflection,” as the cool kids are now calling it, wouldn’t even take that long.

But what’s so important about this kind of reflection?

Humans make many terrible mistakes. If we’d ventured out to the stars a few hundred years ago, we might have brought slavery across the galaxy. Sixty years ago, we’d have brought segregation. Today, we might bring factory farming and wild animal suffering.

Even setting aside specific catastrophic risks, following the wrong moral theory leads to the loss of most available value. Suppose that the right view is hedonic utilitarianism (pleasure minus pain should be maximized), but we act on desire-satisfactionism (desire satisfaction should be maximized). We’d lose out on nearly all realizable value. Other examples can be even more extreme: the optimal world by the lights of totalism (one of the leading theories of population ethics) might be actively bad according to critical level views (another of the leading views).

Totalism says the value of a population is measured by its total welfare. Critical level views say that it’s determined by the welfare of everyone above some threshold minus the welfare of those below the threshold (so e.g. a person .5 above the threshold would add .5 to total value, and a person .5 below the threshold would subtract .5 from total value). So totalism might favor proliferating barely happy minds in enormous numbers. The totalist’s utopia is the critical level theorist’s nightmare.

One option is to simply settle the universe without any discernible plan in mind. But that is likely to be catastrophic. Most value could be jeopardized by ruthless competition to grab resources or individual civilizations unleashing cosmic catastrophes. Even if that doesn’t happen, unless we’re specifically optimizing for bringing about value, we’re likely to miss out on most value we can get. Arbitrarily technologically proficient civilizations get much more of what they optimize for than what they don’t.

Many of the best things to do in the future might be weird and counterintuitive. Perhaps the digital minds that can experience the most joy are weird. It’s hard to imagine a big constituency for bringing them about in large numbers unless we reflect carefully. Other routes towards the production of enormous good—like acausal trade—might be even weirder. Probably there are a lot of actions that are very valuable in non-obvious ways. Absent reflection, we likely won’t take them.

Put extremely concisely, the case for a Deep Reflection is that careful planning assisted by superintelligence allows civilizations to:

• Make good decisions that produce lots of value but are non-obvious.

• Avoid making serious moral errors akin to spreading slavery to the stars.

• Prevent most value from being lost to Darwinian competition.

It’s helpful to take an outside view here. Ask: would past societies have done terrible things if they’d settled space without a Deep Reflection? Of course! Why think we’re any different? You should be suspicious if you conclude that we are the first civilization to avoid major moral error. Past societies have thought the same about themselves. They were wrong. So are we.

Does my argument depend on moral realism? By no means! Even if you don’t think morality is objective, you should think that careful thinking can improve the quality of our decision-making. You don’t have to think that beauty is objective to think that the best way to maximize the universe’s beauty is to think carefully about how to do it.

Now, you might be skeptical of this kind of reflection. Sometimes you think hard about something and don’t get the right answer. But in general, I’d expect better decisions to be made if we reflect carefully than if we don’t. We should expect reflection to improve decision-making. If you didn’t think that, then you should favor reflecting less before making big decisions, because it’s unlikely we’re at precisely the optimal level of reflection. If we shouldn’t favor more, then we should probably favor less.

In addition, superintelligence is likely to improve our ability to reflect well. Compare Fable to the AIs of 2023. Then imagine another decade or more of that kind of progress. The end result would be AI that can make decisions very well. It’s unlikely that humans are the best decision-makers that there can be. Thus, it should be possible in principle to have significantly better decision-making than we have today. Given how important this decision will be, it’s worth waiting for an improved decision-maker before making extremely big decisions.

The Long Reflection often sounds slightly sci-fi-ish. But the core idea is very modest and even a bit trivial. Before making extremely big decisions of cosmic significance that will affect things for billions of years, we should think carefully. So long as you take morality seriously and think major moral mistakes could be cataclysmic, a Deep Reflection is vital.