Bentham's Newsletter

Bentham's Newsletter

Share this post

Bentham's Newsletter
Bentham's Newsletter
The Parable of the AI Genie

The Parable of the AI Genie

Epistemic status: not to be taken too seriously.

Bentham's Bulldog's avatar
Bentham's Bulldog
Jan 28, 2024
∙ Paid

Share this post

Bentham's Newsletter
Bentham's Newsletter
The Parable of the AI Genie
Share

A brief note: I wrote this before I had a blog, in 11th grade, but thought it was fun, so I decided to publish it.

Genie: I offer you three wishes that I fulfill.  I, like God, can only do things that are metaphysically possible, of course. However, if after I grant you my wishes no interesting improvements to the world happen, I will kill you. 

Person: I’d like infinity more wishes

Genie okay, what’s your second wish

Person: I’d like you to bring about the maximum possible amount of the things I would want if I were fully rational

Genie: No can do… I adopt a reductionist view of identity.  I don’t know what it would mean to be you and to be fully rational given that you are not fully rational. 

Person: I’d like you to bring about the maximum possible amount of the things that an agent identical to me in every respect except for their rationality would do if they became fully rational

Genie: Okay, done. 

Person: What happened?

Genie: I did what you said and nothing happened.

Person: Why did nothing happen? 

Keep reading with a 7-day free trial

Subscribe to Bentham's Newsletter to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
© 2025 Bentham's bulldog
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share