Discussion about this post

User's avatar
Noah Birnbaum's avatar

One thing that I think this post is missing is a notion of goodharts law in ethics - namely, if you optimize too hard on some proxy, even if that proxy is usually a good heuristic to get to the terminal end (i.e. some rule like rights is approximately right for the correct ethics most of the time), you will likely miss it.

This is important if you think that ethical theories come apart very hard as you optimize for them harder - I think this is very likely true. When you optimize really hard for a view like utilitarianism, you get something very different than optimizing for a view like person-affecting utilitarianism, for instance. This means that it really makes a difference if you get the exact ethical view correct or if you just nearly miss it.

In this future world that you imagine, whatever agents are there, it seems, will likely do a very good job of optimizing for whatever the best thing might actually be. This just seems true from the historical of technological progress - as we get better technology, we are able to optimize harder for our goals.

Given such a big space of possible ethics and good arguments for many different views, you might think that it’s pretty likely that we miss it by at least a bit - even if we get good AI that makes us better reasoners like you state.

Therefore, we should think it’s very likely that we have bad values. To hedge against this, maybe one might want to push off the optimization time.

Expand full comment
Oscar Delaney's avatar

Excellent post! I think the thing of some (hopefully non-trivial) fraction of people/capital/agents will want to promote The Good, while hopefully a near-zero fraction will want to specifically promote The Bad seems especailly important to me.

A couple of snippets I wrote last year are relevant:

https://forum.effectivealtruism.org/posts/JdDnPmZsSuBgh2AiX/space-settlers-will-favour-meat-alternatives about how it is unlikely factory farminig will persist in space, because people with more cosmopolitan values will go to space.

https://forum.effectivealtruism.org/posts/fosBQPAokSsn4fv77/cosmic-nimbys-and-the-repugnant-conclusion about how we might miss out on a lot of value by having too few people with really high welfare rather than far more people with marginally lower welfare (kind of an anti-repugnant-conclusion).

But yours is more systematic, these focused on just narrow pieces of the relevant idea space.

Expand full comment
5 more comments...

No posts