Discussion about this post

User's avatar
Noah Birnbaum's avatar

One thing that I think this post is missing is a notion of goodharts law in ethics - namely, if you optimize too hard on some proxy, even if that proxy is usually a good heuristic to get to the terminal end (i.e. some rule like rights is approximately right for the correct ethics most of the time), you will likely miss it.

This is important if you think that ethical theories come apart very hard as you optimize for them harder - I think this is very likely true. When you optimize really hard for a view like utilitarianism, you get something very different than optimizing for a view like person-affecting utilitarianism, for instance. This means that it really makes a difference if you get the exact ethical view correct or if you just nearly miss it.

In this future world that you imagine, whatever agents are there, it seems, will likely do a very good job of optimizing for whatever the best thing might actually be. This just seems true from the historical of technological progress - as we get better technology, we are able to optimize harder for our goals.

Given such a big space of possible ethics and good arguments for many different views, you might think that it’s pretty likely that we miss it by at least a bit - even if we get good AI that makes us better reasoners like you state.

Therefore, we should think it’s very likely that we have bad values. To hedge against this, maybe one might want to push off the optimization time.

Expand full comment
River's avatar

I'm skeptical of the idea that moral reflection (by humans or AIs) is likely to lead us closer to moral truth. We saw in the covid pandemic that bioethicists had markedly worse judgments about bioethics than random people off the street. Philosophy as a discipline hasn't had much to show for itself in centuries. Reflection as a method for improving ones moral judgments does not have a great track record, at least when taken to its extremes.

Expand full comment
6 more comments...

No posts