The Parable of the AI Genie

Epistemic status: not to be taken too seriously.

A brief note: I wrote this before I had a blog, in 11th grade, but thought it was fun, so I decided to publish it.

Genie: I offer you three wishes that I fulfill.  I, like God, can only do things that are metaphysically possible, of course. However, if after I grant you my wishes no interesting improvements to the world happen, I will kill you. 

Person: I’d like infinity more wishes

Genie okay, what’s your second wish

Person: I’d like you to bring about the maximum possible amount of the things I would want if I were fully rational

Genie: No can do… I adopt a reductionist view of identity.  I don’t know what it would mean to be you and to be fully rational given that you are not fully rational. 

Person: I’d like you to bring about the maximum possible amount of the things that an agent identical to me in every respect except for their rationality would do if they became fully rational

Genie: Okay, done. 

Person: What happened?

Genie: I did what you said and nothing happened.

Person: Why did nothing happen?