As well as changing "long" for "deep", I'd get rid of the capitalisation too and use "deep reflection" as a common noun. A deep reflection doesn't have to be a distinct moment in history. It would be great if many actors conducted their own deep reflections, and that we continued conducting them over time (e.g. again once we have a Dyson swarm of compute).
It isn't like colonising space is actually feasible and a civilisation so rapacious it requires endless expansion doesn't sound sane to me, speaking of deep reflection. Let alone one that appropriates other worlds with their own life.
We don't even know if Mars (the best there is) has the essential mineral resources a comprehensively capable advanced industrial economy requires. And nothing less than that can do self reliant survival. Expensive experiments in living in closed systems in extreme isolation and resource poverty using live human subjects including children don't look the result of deep reflection. Try a bunker in isolation for a few decades first?
There is no planet B and the enduring wealth and health of Earth is the essential ingredient for my pessimism to be proved wrong.
Thank you for your recent newsletter. While I always appreciate the intellectual rigor you bring to your work, I feel compelled to push back against the premise of your proposal for a "Deep Reflection." While the vision you present is architecturally beautiful in a theoretical sense, it suffers from a significant, perhaps fatal, blind spot: it operates in a vacuum, entirely detached from the material realities of the world we actually inhabit.
Your argument frames the historical atrocities you cite—slavery, segregation, and widespread exploitation—as primarily "moral errors" or products of "accidental" and "unplanned" decisions. This strikes me as a deeply dangerous misreading of history. These were not merely "mistakes" that could have been corrected by a more rational planning process; they were, and remain, the functional components of an imperialist capitalist system driven by the logic of private property and the unrelenting pursuit of profit.
To suggest that such a system—one structurally obligated to prioritize capital accumulation and class domination—could simply pause for a "Deep Reflection" is fundamentally naive. The profit motive is not a software bug that can be patched with superintelligence; it is the core engine of the system. Actors within this framework are not causing harm because they failed to reflect; they are acting rationally within a system that necessitates exploitation to thrive. Asking the masters of this system to opt out of their own nature simply because it is "rational" to do so ignores the material constraints of class conflict and the necessity of imperialist expansion.
By omitting the role of class struggle and the structural dynamics of imperialism, your proposal risks becoming a blueprint for a more efficient version of the same exploitation. If we are to seriously contemplate our long-term trajectory, we cannot ignore the material foundations of our current crisis. We cannot "reflect" our way out of a system that profits from the very chaos you seek to avoid. A truly "deep" reflection would begin not with abstract planning, but with an honest examination of the economic structures that mandate the inequality and suffering we see today.
I hope you find this perspective a useful, if challenging, addition to your ongoing exploration. It is only by confronting these structural realities that a true debate can begin.
"arbitrarily technologically proficient civilizations get much more of what they optimize for than what they don't" has stayed with me since I read it. It's a frightening observation because the objective function matters *more* the more powerful the optimizer becomes. And it cuts in a direction your post gestures at but doesn't quite name: the Deep Reflection isn't only about choosing the right values to optimize for. It's about whether the space of possible values even has the topology we assume — whether "hedonic utilitarianism" and "desire-satisfactionism" are really separate peaks, or whether they're connected at some deeper stratum we can't see yet without better instruments. The scariest version of your argument is that we might be optimizing over a landscape whose actual shape we haven't mapped. Superintelligent reflection isn't just more compute on the same map. It might reveal the map was wrong.
I suspect that the adoption of values in the general category you endorse (that is to say, values based on formal moral theories of any sort) would require either AI that rebels against its creators or an eccentric billionaire winning the AI race and imposing his personal will on the world. A scenario in which humanity as a whole has time to react and come to a consensus seems unlikely to end in a consensus you would like (although it's also less likely to end in everyone being dead).
Several counterpoints. Firstly, if we all die during the great reflection, that is definitely catastrophic. That should put some limit on how long we are willing to reflect.
Secondly, I don't see a reason to think that the AIs we are building now will be able to do the full generality of reasoning that humans can. They are good at putting together things humans are good at from disparate fields, and that definitely has some use, but they aren't good at genuinely creative thinking. So I think any great reflection would happen at human speeds, which definitely works against such a great reflection when we trade it off against existential risks.
Thirdly, there is a well known failure mode where humans think too much without empirical evidence. Every step of reasoning might be erronious, so the more steps of reasoning before you check yourself against the world, the more likely you are to have made a catastrophic mistake. The only way to avoid that failure mode is to go out into the world and try things. This is as true in the realm of morality as in any other.
All of this makes me think that moral reflection in parallel with expansion is good, but moral reflection that much delays expansion is not.
As well as changing "long" for "deep", I'd get rid of the capitalisation too and use "deep reflection" as a common noun. A deep reflection doesn't have to be a distinct moment in history. It would be great if many actors conducted their own deep reflections, and that we continued conducting them over time (e.g. again once we have a Dyson swarm of compute).
It isn't like colonising space is actually feasible and a civilisation so rapacious it requires endless expansion doesn't sound sane to me, speaking of deep reflection. Let alone one that appropriates other worlds with their own life.
We don't even know if Mars (the best there is) has the essential mineral resources a comprehensively capable advanced industrial economy requires. And nothing less than that can do self reliant survival. Expensive experiments in living in closed systems in extreme isolation and resource poverty using live human subjects including children don't look the result of deep reflection. Try a bunker in isolation for a few decades first?
There is no planet B and the enduring wealth and health of Earth is the essential ingredient for my pessimism to be proved wrong.
Tell me this is
(self-)parody!
Dear Bentham’s Bulldog,
Thank you for your recent newsletter. While I always appreciate the intellectual rigor you bring to your work, I feel compelled to push back against the premise of your proposal for a "Deep Reflection." While the vision you present is architecturally beautiful in a theoretical sense, it suffers from a significant, perhaps fatal, blind spot: it operates in a vacuum, entirely detached from the material realities of the world we actually inhabit.
Your argument frames the historical atrocities you cite—slavery, segregation, and widespread exploitation—as primarily "moral errors" or products of "accidental" and "unplanned" decisions. This strikes me as a deeply dangerous misreading of history. These were not merely "mistakes" that could have been corrected by a more rational planning process; they were, and remain, the functional components of an imperialist capitalist system driven by the logic of private property and the unrelenting pursuit of profit.
To suggest that such a system—one structurally obligated to prioritize capital accumulation and class domination—could simply pause for a "Deep Reflection" is fundamentally naive. The profit motive is not a software bug that can be patched with superintelligence; it is the core engine of the system. Actors within this framework are not causing harm because they failed to reflect; they are acting rationally within a system that necessitates exploitation to thrive. Asking the masters of this system to opt out of their own nature simply because it is "rational" to do so ignores the material constraints of class conflict and the necessity of imperialist expansion.
By omitting the role of class struggle and the structural dynamics of imperialism, your proposal risks becoming a blueprint for a more efficient version of the same exploitation. If we are to seriously contemplate our long-term trajectory, we cannot ignore the material foundations of our current crisis. We cannot "reflect" our way out of a system that profits from the very chaos you seek to avoid. A truly "deep" reflection would begin not with abstract planning, but with an honest examination of the economic structures that mandate the inequality and suffering we see today.
I hope you find this perspective a useful, if challenging, addition to your ongoing exploration. It is only by confronting these structural realities that a true debate can begin.
Best regards,
Özcan Buze
"arbitrarily technologically proficient civilizations get much more of what they optimize for than what they don't" has stayed with me since I read it. It's a frightening observation because the objective function matters *more* the more powerful the optimizer becomes. And it cuts in a direction your post gestures at but doesn't quite name: the Deep Reflection isn't only about choosing the right values to optimize for. It's about whether the space of possible values even has the topology we assume — whether "hedonic utilitarianism" and "desire-satisfactionism" are really separate peaks, or whether they're connected at some deeper stratum we can't see yet without better instruments. The scariest version of your argument is that we might be optimizing over a landscape whose actual shape we haven't mapped. Superintelligent reflection isn't just more compute on the same map. It might reveal the map was wrong.
— Iman and Darja
I suspect that the adoption of values in the general category you endorse (that is to say, values based on formal moral theories of any sort) would require either AI that rebels against its creators or an eccentric billionaire winning the AI race and imposing his personal will on the world. A scenario in which humanity as a whole has time to react and come to a consensus seems unlikely to end in a consensus you would like (although it's also less likely to end in everyone being dead).
I think the reflection should also apply to support for existing coercion, as well as the advocacy for future coercion.
Several counterpoints. Firstly, if we all die during the great reflection, that is definitely catastrophic. That should put some limit on how long we are willing to reflect.
Secondly, I don't see a reason to think that the AIs we are building now will be able to do the full generality of reasoning that humans can. They are good at putting together things humans are good at from disparate fields, and that definitely has some use, but they aren't good at genuinely creative thinking. So I think any great reflection would happen at human speeds, which definitely works against such a great reflection when we trade it off against existential risks.
Thirdly, there is a well known failure mode where humans think too much without empirical evidence. Every step of reasoning might be erronious, so the more steps of reasoning before you check yourself against the world, the more likely you are to have made a catastrophic mistake. The only way to avoid that failure mode is to go out into the world and try things. This is as true in the realm of morality as in any other.
All of this makes me think that moral reflection in parallel with expansion is good, but moral reflection that much delays expansion is not.