It will require more than a long reflection. It will also require a long investigation, to check the facts are correct, and what values are actually right. I do not necessarily want the coordination of reflected human values to tile the universe, because I do not trust them. It will be difficult, and it may take millions of years to do right by itself, even with AI acceleration. These issues may be incredibly thorny.
It is worth mentioning that this is not where you should set your bar, or I am afraid you will be disappointed. Do not let the perfect be the enemy of the good. It may be much much much worse than this if we do not get the basics right, which we are not currently on track to do.
I agree that things could go a lot worse than failing to have a long reflection (we could all die, for example)! The reason I mention this as being a big deal is that I think most expected value is in the worlds where we roughly optimize with respect to space resources, and that the odds of doing that is below 50%. So the EV loss from failed optimization is greater even than existential risks (assuming your P(doom) isn't extremely high which mine is not).
I think failed optimization just results in so much lost EV that it's the top priority. I think bad futures from digital suffering are maybe #3 and doom is #2--especially when you take into account all the ways AI might kill us.
I guess it depends on your view of how good good can get, and how bad bad can get, and the respective probabilities. I think digital suffering is probably #1.
EDIT: But I don't act like it because I do not like thinking about it -> irrational I know but a guy's gotta live.
The danger isn’t only value lock-in. It’s temporal incoherence. Every civilization before us operated on short human cycles now we are creating systems that operate on infinite ones. If we don’t align on a long-term temporal governance model, we won’t be choosing the future. The future will choose itself.
It would seem to follow from your reasoning that EV calculations will depend on measuring this psychophysical harmony in some way. E.g., if pleasure is an important component of world-optimization, then most likely the *qualia* of pleasure have a quite different utility from the mere physical or structural-functional *correlates* of pleasure.
Yes it is the same AI as that, currently, though the one being spoken about in the post is a future AI. Though I agree! This is very worrying behaviour! I am worried about it!
What is totalitarian about wanting to make the universe a happier, more flourishing place, I ask out of sincere curiosity? That would not fit my definition of totalitarian.
Don’t worry, I am very difficult to offend. I agree lots of bad stuff has come of grand visions of wanting to change the world — but so has lots of good stuff: ending slavery, defeating the nazis, women’s rights, eliminating polio.
Just because there are people who use “let’s do good things” as a means to do bad things should not poison you to the idea of doing good things, there is so much in the world we can do to make things better!
It will require more than a long reflection. It will also require a long investigation, to check the facts are correct, and what values are actually right. I do not necessarily want the coordination of reflected human values to tile the universe, because I do not trust them. It will be difficult, and it may take millions of years to do right by itself, even with AI acceleration. These issues may be incredibly thorny.
It is worth mentioning that this is not where you should set your bar, or I am afraid you will be disappointed. Do not let the perfect be the enemy of the good. It may be much much much worse than this if we do not get the basics right, which we are not currently on track to do.
I agree that things could go a lot worse than failing to have a long reflection (we could all die, for example)! The reason I mention this as being a big deal is that I think most expected value is in the worlds where we roughly optimize with respect to space resources, and that the odds of doing that is below 50%. So the EV loss from failed optimization is greater even than existential risks (assuming your P(doom) isn't extremely high which mine is not).
I don't mean to be macabre, but things could be a lot worse than that. And I have some sympathy to the negative utilitarian view.
I agree!
So P(doom) and EV loss from failed optimization become two considerations among many re: the risks, possibly not the dominant ones.
I think failed optimization just results in so much lost EV that it's the top priority. I think bad futures from digital suffering are maybe #3 and doom is #2--especially when you take into account all the ways AI might kill us.
I guess it depends on your view of how good good can get, and how bad bad can get, and the respective probabilities. I think digital suffering is probably #1.
EDIT: But I don't act like it because I do not like thinking about it -> irrational I know but a guy's gotta live.
The danger isn’t only value lock-in. It’s temporal incoherence. Every civilization before us operated on short human cycles now we are creating systems that operate on infinite ones. If we don’t align on a long-term temporal governance model, we won’t be choosing the future. The future will choose itself.
You have written before (https://benthams.substack.com/p/arguments-for-god-tier-list) about the tremendous psychophysical harmony our world exhibits as well as (IIUC) its irreducibility by typical naturalistic explanations.
It would seem to follow from your reasoning that EV calculations will depend on measuring this psychophysical harmony in some way. E.g., if pleasure is an important component of world-optimization, then most likely the *qualia* of pleasure have a quite different utility from the mere physical or structural-functional *correlates* of pleasure.
What is to be done about this?
Is this the same AI that thinks most popes were Black women the that the Nazis were a multiracial multi-ethnic organization?
I asked ChatGPT this: "answer both of these questions in one word: were most popes black? Were the Nazis a multi-racial multi-ethnic organization?"
It responded:
"No.
No."
Not sure what your point was but seems like maybe you were misinformed?
I believe it was Gemini and it was very well covered in the news. I do not use AI but I did a regular Internet search and found this as the first of many results: https://www.vox.com/future-perfect/2024/2/28/24083814/google-gemini-ai-bias-ethics
Modern AIs do not do this anymore though, so to answer your question: No, it’s not the same AI.
Ok, is it the same AI that made a recruiter have a psychotic break with reality or is August 2025 not considered modern AI? https://www.nytimes.com/2025/08/08/technology/ai-chatbots-delusions-chatgpt.html
Yes it is the same AI as that, currently, though the one being spoken about in the post is a future AI. Though I agree! This is very worrying behaviour! I am worried about it!
Unless other suboptimal civilization starts spreading before us creating land of suffering in which case we should accelerate as fast as possible
What is totalitarian about wanting to make the universe a happier, more flourishing place, I ask out of sincere curiosity? That would not fit my definition of totalitarian.
Don’t worry, I am very difficult to offend. I agree lots of bad stuff has come of grand visions of wanting to change the world — but so has lots of good stuff: ending slavery, defeating the nazis, women’s rights, eliminating polio.
Just because there are people who use “let’s do good things” as a means to do bad things should not poison you to the idea of doing good things, there is so much in the world we can do to make things better!