I would not say that his arguments are consistently interesting or intelligent. I stopped following him when he started writing a bunch of woke stuff that was just as dumb and boring as any other woke stuff. But the particular arguments you are engaging with here I agree are interesting and intelligent, and I have some kind of engagement with them on my to-do list, so thank you for reminding me of that.
“You should assign non-trivial credence to humans reaching a state where the odds of extinction per century are approximately zero”
No you shouldn’t. This is absurd. Humans will never solve all problems everywhere forever. However good we get at, say, detecting and deflecting asteroids before they eradicate all of humanity, there will always be some non-zero likelihood of critical failure. The asteroid only needs to hit once. We need to have a perfect batting average forever to prevent it.
You’re also not considering the very high likelihood of a cataclysm not severe enough to cause extinction that nonetheless degrades or eliminates our extinction prevention capacity. If a nuclear war kills 90% of the human population the remaining 10% needs to return to a level where they can reliably bat away asteroids before the next asteroid approaches.
There are also some catastrophes that simply aren’t preventable. Eventually the sun will render the planet uninhabitable. If we can somehow manage to invent “solar rejuvenation” before that happens (we won’t), eventually there will be a gamma ray burst that sterilizes the galaxy. The extinction of the human race is inevitable.
There will never be a self sufficient human settlement anywhere but on Earth. We might have permanent bases on Mars or the Moon, but no one will ever move to there to raise a family. Look at Antarctica. It’s a veritable paradise compared to the next most inhabitable planet, and yet literally zero people in the entire history of humankind have ever tried to raise a child there. Barely anyone wants to live in Wyoming.
Well, first of all, there could be lots of non-biological people. Second of all, you're assuming current conditions. But it's plausible that with very advanced technology we might terraform other planets and make them nice and habitable. The long-term future of humanity--and whether it will span multiple planets--shouldn't be the kind of thing you're 99.99% sure about.
It’s not plausible that technology will ever be advanced enough that Mars won’t be a shithole compared to Earth. Given the choice between raising a family on Antarctica vs raising a family anywhere else on the planet, literally every human being who has ever lived—with ZERO exceptions out of 100 billion or so—has chosen not-Antarctica. It would take huge leaps in technology beyond our current ability to fathom to bring Mars up to the standard of living of Antarctica, but while we’re doing that, technology will also be improving the standard of living available on Earth. There will always be a vast difference between the quality of life available on Earth and the quality of life available on Mars. There will never be a society abundant enough in resources to terraform another planet that wouldn’t also establish a paradise beyond all imagining on Earth. There will never be sufficiently large subpopulation of people willing to sacrifice their own and their children’s standard of living to establish a self-sufficient settlement on another planet that could survive the extinction of the population on Earth.
I am 100% confident this will never ever happen. It is foolish to expect that it could ever happen, when there is no history of human beings on Earth willing to deliberately make their lives worse. Human colonization of Earth has always been about the pursuit of a *better* life for ourselves and our children. If you think there’s a reason why colonization of extraterrestrial planets would change this tendency, the onus is on you to establish it not on me to rebut it.
As for the robots, why would their continued existence be any consolation at all for the extinction the human race? It’s virtually certain that there is already intelligent life on other planets that would survive the extinction of humanity. Consciousness will probably never go extinct as long as there are stars in the sky. Why is it better that the human race is survived by robots than by aliens?
What do you mean "a gamma ray burst that sterilizes the galaxy"? Life has existed on Earth for like a quarter of the Universe's existence. Seems like if galactic-scale gamma-ray bursts are likely one would probably have already happened? Are you thinking of some sort of alien weapon?
Quasars would be incapable of this, black holes would be incapable of this, supernovae would be incapable of this. Where does the idea of a galactic-scale gamma ray burst come from?
I'm late to the party, but I have a question. You wrote:
"If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning reducing odds of existential risks by 1/10^20 creates 10^28 extra lives."
What about the scenarios that cover the rest of the 99.99%? I can also imagine the possibility that the future holds 10^52 *unhappy* people having a 0.01% chance (among many other pessimistic scenarios). Wouldn't that cancel out the expected value of the 10^28 extra (happy) lives?
I think you're wrong, or at least overconfident, on the declining probability argument. You say of the St Petersburg paradox that we're "supposed to stipulate that you have some good reason to think they’re telling the truth."
But you have to think seriously about what would actually count as a good reason to believe them. And I think it's pretty intuitive to that the prior probability you should have on them telling the truth should scale with the amount of resources they're implication promising. If someone offers to double or nothing a bet with one dollar on the line, I require almost no evidence to believe them. With a million dollars on the line, I'll ask to see bank statements or something. With a trillion dollars on the line, I won't believe them at all.
Now, I can imagine what good evidence to believe someone has access to a trillion dollars could look like, but a googol? A googolplex? I don't think a person can even count to Graham's number; I have no idea how I would even verify a supposed sum of money to be Graham's number dollars.
Another consideration: for any proposition with a natural number variable N, so that P(N) is the outcome "proposition P applies to N people/dollars/units of time/etc", it is necessary that the probability of P(N) occurring is monotonically non-increasing in N, since P(n) is a strict subset of P(N) for all n<=N. What's more, except in very unique circumstances (proof by induction or something like that), it's plainly irrational to have the conditional probability p(P(N+1)|P(N)) to be 1, so generically it'll be monotonically decreasing.
Now, how fast it decreases can be debatable, but as a simple example, if you think the conditional probably is bounded above by a fixed constant 1-c, where c basically is some epistemic humility parameter like, "surely the odds of this failing to continue can't be lower than c", then the probability of observing P(N) will be bounded above by the exponentially decaying (1-c)^N.
Now obviously this is too simple to take seriously, but I think it points to a real problem: the numbers for how many future people there might be are *modeled*: some calculation is done on the number of people at some average density who can fill some spacetime volume. But the probability is *not modeled*, it's vibecoded by saying, "but surely there's *some* non trivial probability". This method is totally unprincipled; the same vibes that make me suspicious if 10^-52 as a probability, also make me suspicious of 10^52 as a number of people. If I'm being told that I have to take the latter seriously on the basis of some calculation, I want to see a comparable computation for the conditional probability of humanity going to occupy spacetime volume V+dV, conditional on them occupying spacetime volume V.
Ultimately, both the outcomes and the probabilities should emerge from some unified model, so we can be confident that the two numbers are comparably justified, and so we can get a sense for how sensitive conclusions are to parameters, etc.
I'm happy to see engagement with this article, and I think you make interesting points.
One bigger-picture consideration that I think you are neglecting is that even if your arguments go through (which is plausible), the argument for longtermism/xrisk shifts significantly.
Originally, the claim is something like
There is really bad risky tech
There is a ton of people in the future
Risky tech will prevent these people from having (positive) lives
________________________________
Reduce tech risk
On the dialectic you sketch, the claim is something like
There is a lot of really bad risky tech
This tech, if wielded well, can reduce the risk of all other tech to zero
There is a small chance of a ton of people in the future
If we wield the tech well and get a ton of people in the future, thats great
_________________________________________
Reduce tech risk (and, presumably, make it powerful enough to eliminate all risk and start having kids)
I think the extra assumptions we need for your arguments against Thorstadt to go through are ones that make longtermism much less attractive to many people, including funders. They also make x-risk unattractive for people who disagree with p2 (i.e., people who do not believe in superintelligence).
I think people are aware that this makes longtermism much less attractive - I typically don't see x-risk work being motivated in this more assumption-heavy way. And, as Thorstad usefullly points out, there is virtually no serious e(v) calculus for longtermist intervention that does a decent job at accounting for these complexities. That's a shame, because EA at least originally seemed to be very dilligent about providing explicit, high-quality e(v) models instead of going by vibes and philosophical argument alone.
I've always taken the core Longtermist argument to be "the future might be very big and thus has lots of people in expectation." Thorstad's arguments might for emotional reasons affect how much people buy it, but as I explain, I don't actually think it weakens its force. And in any case, I was being *extremely* generous when I gave his arguments 4 OOMs of force.
The additional assumption of eliminating all xrisk seems to massively weaken its force (as you note).
It also seems highly relevant because this assumption extremely implausible for any x-risk that is not superintelligence - for example, theres no reason to believe biorisk reduction would lower extinction risk from other causes to 0.
I don't think dismissing it as emotional is very sensible.
Would you agree with the following?
This is a big extra assumption that is not reflected in existing e(v) models of xrisk and sriks; the fact that you can patch it does not make pointing out the extra assumption irrelevant; ideally, future models would account for/make explicit the extra assumption, as they'll be inaccurate otherwise.
FWIW the "mathematical" solution to the St Petersburg paradox is Kelly betting. Now I know it doesn't make sense to kelly-bet utilities (because a 10% chance of 100 utility is _just_ better than a 50% chance of 2 utility -- it's literally utility, kelly betting for utility itself doesn't make sense). But maybe you should define your utility to have _some_ dropoff past a certain point
This might be justifiable for Klaas Kraay esque reasons, i.e. if you have 10^googol minds, then probably some of those minds are extremely extremely similar, so maybe it's less valuable to insantiate 2 copies of extremely similar (possibly identical) people than to instantiate 1 person whose life is great in way X and 1 person whose life is great in a way very different to X. This leads to dropoff past a certain amount of utility, since the utility might be instantiated in virtually identical ways, possibly making it less valuable (on Kraay's account).
Bentham, you got a great shout-out from Will MacAskill at EA connect yesterday - not sure if you saw :)
I would not say that his arguments are consistently interesting or intelligent. I stopped following him when he started writing a bunch of woke stuff that was just as dumb and boring as any other woke stuff. But the particular arguments you are engaging with here I agree are interesting and intelligent, and I have some kind of engagement with them on my to-do list, so thank you for reminding me of that.
I do find Thorstad to sometimes be unreasonable and uncharitable, but he often makes interesting points that are worth responding to. https://slatestarcodex.com/2019/02/26/rule-genius-in-not-out/
“You should assign non-trivial credence to humans reaching a state where the odds of extinction per century are approximately zero”
No you shouldn’t. This is absurd. Humans will never solve all problems everywhere forever. However good we get at, say, detecting and deflecting asteroids before they eradicate all of humanity, there will always be some non-zero likelihood of critical failure. The asteroid only needs to hit once. We need to have a perfect batting average forever to prevent it.
You’re also not considering the very high likelihood of a cataclysm not severe enough to cause extinction that nonetheless degrades or eliminates our extinction prevention capacity. If a nuclear war kills 90% of the human population the remaining 10% needs to return to a level where they can reliably bat away asteroids before the next asteroid approaches.
There are also some catastrophes that simply aren’t preventable. Eventually the sun will render the planet uninhabitable. If we can somehow manage to invent “solar rejuvenation” before that happens (we won’t), eventually there will be a gamma ray burst that sterilizes the galaxy. The extinction of the human race is inevitable.
Well if we're multi planet then the odds of asteroids killing us are very low.
There will never be a self sufficient human settlement anywhere but on Earth. We might have permanent bases on Mars or the Moon, but no one will ever move to there to raise a family. Look at Antarctica. It’s a veritable paradise compared to the next most inhabitable planet, and yet literally zero people in the entire history of humankind have ever tried to raise a child there. Barely anyone wants to live in Wyoming.
Well, first of all, there could be lots of non-biological people. Second of all, you're assuming current conditions. But it's plausible that with very advanced technology we might terraform other planets and make them nice and habitable. The long-term future of humanity--and whether it will span multiple planets--shouldn't be the kind of thing you're 99.99% sure about.
It’s not plausible that technology will ever be advanced enough that Mars won’t be a shithole compared to Earth. Given the choice between raising a family on Antarctica vs raising a family anywhere else on the planet, literally every human being who has ever lived—with ZERO exceptions out of 100 billion or so—has chosen not-Antarctica. It would take huge leaps in technology beyond our current ability to fathom to bring Mars up to the standard of living of Antarctica, but while we’re doing that, technology will also be improving the standard of living available on Earth. There will always be a vast difference between the quality of life available on Earth and the quality of life available on Mars. There will never be a society abundant enough in resources to terraform another planet that wouldn’t also establish a paradise beyond all imagining on Earth. There will never be sufficiently large subpopulation of people willing to sacrifice their own and their children’s standard of living to establish a self-sufficient settlement on another planet that could survive the extinction of the population on Earth.
I am 100% confident this will never ever happen. It is foolish to expect that it could ever happen, when there is no history of human beings on Earth willing to deliberately make their lives worse. Human colonization of Earth has always been about the pursuit of a *better* life for ourselves and our children. If you think there’s a reason why colonization of extraterrestrial planets would change this tendency, the onus is on you to establish it not on me to rebut it.
As for the robots, why would their continued existence be any consolation at all for the extinction the human race? It’s virtually certain that there is already intelligent life on other planets that would survive the extinction of humanity. Consciousness will probably never go extinct as long as there are stars in the sky. Why is it better that the human race is survived by robots than by aliens?
Robots could be people and could live in a wide variety of environments in principle.
So what?
What do you mean "a gamma ray burst that sterilizes the galaxy"? Life has existed on Earth for like a quarter of the Universe's existence. Seems like if galactic-scale gamma-ray bursts are likely one would probably have already happened? Are you thinking of some sort of alien weapon?
Quasars would be incapable of this, black holes would be incapable of this, supernovae would be incapable of this. Where does the idea of a galactic-scale gamma ray burst come from?
I'm late to the party, but I have a question. You wrote:
"If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning reducing odds of existential risks by 1/10^20 creates 10^28 extra lives."
What about the scenarios that cover the rest of the 99.99%? I can also imagine the possibility that the future holds 10^52 *unhappy* people having a 0.01% chance (among many other pessimistic scenarios). Wouldn't that cancel out the expected value of the 10^28 extra (happy) lives?
I think you're wrong, or at least overconfident, on the declining probability argument. You say of the St Petersburg paradox that we're "supposed to stipulate that you have some good reason to think they’re telling the truth."
But you have to think seriously about what would actually count as a good reason to believe them. And I think it's pretty intuitive to that the prior probability you should have on them telling the truth should scale with the amount of resources they're implication promising. If someone offers to double or nothing a bet with one dollar on the line, I require almost no evidence to believe them. With a million dollars on the line, I'll ask to see bank statements or something. With a trillion dollars on the line, I won't believe them at all.
Now, I can imagine what good evidence to believe someone has access to a trillion dollars could look like, but a googol? A googolplex? I don't think a person can even count to Graham's number; I have no idea how I would even verify a supposed sum of money to be Graham's number dollars.
Another consideration: for any proposition with a natural number variable N, so that P(N) is the outcome "proposition P applies to N people/dollars/units of time/etc", it is necessary that the probability of P(N) occurring is monotonically non-increasing in N, since P(n) is a strict subset of P(N) for all n<=N. What's more, except in very unique circumstances (proof by induction or something like that), it's plainly irrational to have the conditional probability p(P(N+1)|P(N)) to be 1, so generically it'll be monotonically decreasing.
Now, how fast it decreases can be debatable, but as a simple example, if you think the conditional probably is bounded above by a fixed constant 1-c, where c basically is some epistemic humility parameter like, "surely the odds of this failing to continue can't be lower than c", then the probability of observing P(N) will be bounded above by the exponentially decaying (1-c)^N.
Now obviously this is too simple to take seriously, but I think it points to a real problem: the numbers for how many future people there might be are *modeled*: some calculation is done on the number of people at some average density who can fill some spacetime volume. But the probability is *not modeled*, it's vibecoded by saying, "but surely there's *some* non trivial probability". This method is totally unprincipled; the same vibes that make me suspicious if 10^-52 as a probability, also make me suspicious of 10^52 as a number of people. If I'm being told that I have to take the latter seriously on the basis of some calculation, I want to see a comparable computation for the conditional probability of humanity going to occupy spacetime volume V+dV, conditional on them occupying spacetime volume V.
Ultimately, both the outcomes and the probabilities should emerge from some unified model, so we can be confident that the two numbers are comparably justified, and so we can get a sense for how sensitive conclusions are to parameters, etc.
Crossposting from EA Forum
I'm happy to see engagement with this article, and I think you make interesting points.
One bigger-picture consideration that I think you are neglecting is that even if your arguments go through (which is plausible), the argument for longtermism/xrisk shifts significantly.
Originally, the claim is something like
There is really bad risky tech
There is a ton of people in the future
Risky tech will prevent these people from having (positive) lives
________________________________
Reduce tech risk
On the dialectic you sketch, the claim is something like
There is a lot of really bad risky tech
This tech, if wielded well, can reduce the risk of all other tech to zero
There is a small chance of a ton of people in the future
If we wield the tech well and get a ton of people in the future, thats great
_________________________________________
Reduce tech risk (and, presumably, make it powerful enough to eliminate all risk and start having kids)
I think the extra assumptions we need for your arguments against Thorstadt to go through are ones that make longtermism much less attractive to many people, including funders. They also make x-risk unattractive for people who disagree with p2 (i.e., people who do not believe in superintelligence).
I think people are aware that this makes longtermism much less attractive - I typically don't see x-risk work being motivated in this more assumption-heavy way. And, as Thorstad usefullly points out, there is virtually no serious e(v) calculus for longtermist intervention that does a decent job at accounting for these complexities. That's a shame, because EA at least originally seemed to be very dilligent about providing explicit, high-quality e(v) models instead of going by vibes and philosophical argument alone.
I've always taken the core Longtermist argument to be "the future might be very big and thus has lots of people in expectation." Thorstad's arguments might for emotional reasons affect how much people buy it, but as I explain, I don't actually think it weakens its force. And in any case, I was being *extremely* generous when I gave his arguments 4 OOMs of force.
The additional assumption of eliminating all xrisk seems to massively weaken its force (as you note).
It also seems highly relevant because this assumption extremely implausible for any x-risk that is not superintelligence - for example, theres no reason to believe biorisk reduction would lower extinction risk from other causes to 0.
I don't think dismissing it as emotional is very sensible.
Would you agree with the following?
This is a big extra assumption that is not reflected in existing e(v) models of xrisk and sriks; the fact that you can patch it does not make pointing out the extra assumption irrelevant; ideally, future models would account for/make explicit the extra assumption, as they'll be inaccurate otherwise.
Biorisk reduction will lower extinction from biorisk short term, and then AI might stop other ex risks.
Agree with the last bit but I don't think they affect things too much, and I don't think it's a big deal that simple models leave out this assumption.
I disagree they massively weaken its force.
FWIW the "mathematical" solution to the St Petersburg paradox is Kelly betting. Now I know it doesn't make sense to kelly-bet utilities (because a 10% chance of 100 utility is _just_ better than a 50% chance of 2 utility -- it's literally utility, kelly betting for utility itself doesn't make sense). But maybe you should define your utility to have _some_ dropoff past a certain point
This might be justifiable for Klaas Kraay esque reasons, i.e. if you have 10^googol minds, then probably some of those minds are extremely extremely similar, so maybe it's less valuable to insantiate 2 copies of extremely similar (possibly identical) people than to instantiate 1 person whose life is great in way X and 1 person whose life is great in a way very different to X. This leads to dropoff past a certain amount of utility, since the utility might be instantiated in virtually identical ways, possibly making it less valuable (on Kraay's account).