Mencius Moldbug Is Not Not A Blithering Idiot
His reservations are indeed, rather unqualified
(Title is a reference to this).
It seems like some people, when attempting to provide an argument for a specific conclusion prefer to go on and on—recounting the history of the 20th century, throwing in obscure literary references, and doing everything they can to pull the wool over the eyes of the reader. Beneath their dense volume of text, combined with almost aggressive smugness it’s very easy to mistake their ramblings for a persuasive case for something. Indeed, sometimes the writing is sufficiently confusing that it’s hard to realize what they’re even saying—much less whether their argument succeeds.
When confronted with such people, it’s very hard to know what to do with them. Are you, after all, the idiot who can’t follow what they’re saying? Are they really dispensing brilliant points that you’re merely too dense to follow? Are you the one missing the thread?
Even those who had not seen the title of the post who are familiar with Mr. Moldbug would no doubt assign a reasonable probability to me talking about Moldbug. Moldbug is perhaps the most obvious example of this.
Yet sometimes, those like Moldbug who make horrific statements wrapped up within the veneer of irony, surrounded by an extra 50,000 words of text, venture out and make succinct points. When they do that, it becomes easy to critique their statements, without needing to go through a million words of text (I wouldn’t be surprised if Mr. Moldbug has a million words of text written on his blog—holy cow is he prolix).
For those unaware, Moldbug, whose real name is Curtis Yarvin ran a website called unqualified reservations and now writes on substack. He’s quite a good writer, very fun to read merely for his literary skill. Yet Moldbug is off his rocker—and his shorter, clearer essays prove this. Those well versed in economics will take the fact that Moldbug complained about trade deficits in an infowars interview to be itself disqualifying—trade deficits are primarily worried about by those who don’t know what they’re talking about.
One might ask why I poison the well in this way. Well (pun intended), Yarvin’s response to critiques frequently involves asking people to read enormous amounts about ancillary subjects, rather than confronting the critiques head on. However, the probability that someone’s gazillion words of writing have relevant things to bear on wide ranging subjects decreases rather dramatically if they are a fool—and one who claims knowledge about trade policy while denouncing trade deficits proves themself a fool.
But the more interesting subject that this article intends to address is Mr. Moldbug’s article about effective altruism. This article demonstrated a rather elementary confusion on an enormous range of topics. Thus, being the bulldog that I am—I thought it would be worth responding to it. It also seems like I’m the only person who can stomach reading a full Moldbug article—given his abject failure to eschew ostentatious prolixity.
Below I present a cute theorem which that odd Silicon Valley institution Leverage Research, which was at least peripheral to the “effective altruism” (EA) movement, also invented on their own. Or at least, so some claim.
The theorem is that the most effective, impactful altruism is actually world domination. Which, once you think about it for a moment, is obvious. But because I like to kick my competitors when they are down, I want to steal this idea and present it as my own.
I would dispute Mr. Moldbug’s assertion that it is obvious upon reflection. Indeed, it seems rather obvious that it would not be a good idea upon even rudimentary reflection. Effective altruists analyze issues by looking at tractability (how much of an impact one can have on an issue), neglectedness (how much other people are working on it), and magnitude (how big of an impact such actions can have).
Let’s analyze the three of them to see why Moldbug’s proposal is foolish.
Magnitude is clearly enormous but it’s sign is unclear. It’s not clear that world government would be good. A world global totalitarian regime would be a disaster of truly global proportions. Competition between different governments seems good. Additionally, global governments would likely splinter and fail in the long term. So it seems quite unclear that global domination would be a good idea at all.
Neglectedness—this one is unclear. I don’t know how many are working on global domination, but almost certainly more than are currently working on EA causes.
Tractability: This is where things go off the rails. If every effective altruist on earth devoted their entire lives to trying to take over the world—they’d be labeled a terrorist organizations and would fail spectacularly to take over the world. The world is a rather big place with deeply entrenched political divides. How would one join China, Russia, the U.S., and every other world nation under a global regime.
International law, which attempts to control the world in only the loosest sense is often unsuccessful—countries frequently violate it. Nations with nuclear weapons have no desire to join under a common global regime. Indeed, we’ve seen how difficult it is for Russia to take over Ukraine—much less for any country to take over the entire world.
Moldbug’s idea for effectiveness is thus a utopian (or perhaps dystopian) fantasy. Such fantasies may garner the stamp of approval from his neoreactionary buddies, they may garner approval from Trotsky, but they do not pass the muster of the effective altruism movement. And for good reason!
In any case, neither Leverage nor anyone else is responsible for the radical corollary that I present below. I am not even sure I am responsible for it, as my conclusion—that the most effective possible altruism is nothing other than Xi Jinping Thought—cannot possibly be correct. But where is the error?
Gray Mirror has been short of classic political content lately—perhaps, over the global Thanksgiving weekend, stuffed with some local creature, sated with shopping, then seized with a Saturday anomie that even college-football cannot slake, in a flash you may perceive my horrendous mistake, and be moved to correct me in the comments. (Subscribers only, please—respect the grift.) Until then, this blog respects Xi Jinping.
It’s unclear whether Moldbug is talking about literally helping Xi or whether he’s talking about helping a similar character gain world domination. If he’s talking about Xi, well—an attempted takeover of the world by China would go over rather poorly. A world run by China would not be a better world.
If he’s talking about another figure, well—the same problems highlighted above present themselves.
Efficient altruism
There are two ways to interpret the literal term “effective altruism.” One is “efficient altruism.” The other is “potent altruism.” The colloquial usage is usually the former, “efficient altruism”—so let’s define that.
Suppose you have some unit of money and/or power. (Money is valuable because it can be converted into power—it can make people do things they don’t otherwise want to do.) You have some altruistic mechanism which converts this resource, money/power, into impact—some good done to the human and/or physical world.
The strategy of “efficient altruism” is to give as much as you can of your own altruistic capacity (which you should maximize, like the EAs who work on Wall Street and live like monks, using their salaries to cure river blindness rather than drink $23 martinis) toward the most efficient possible mechanisms. If you have 10,000 dollars to spend on helping other people, what is the most effective way to spend it?
The answer to this question comes down to the most efficient way to spend a dollar, where efficiency is measured by its positive impact on the lives of human beings. Believers in “efficient altruism” note the remarkable inefficiencies of many popular paths toward altruistic impact.
This is good and healthy, although the cause of this perplexing pattern is seldom examined—why does everything suck? One would think that before inventing a new thing, one would want to know how and why all the old things had come to suck. But we digress.
Moldbug makes several elementary mistakes.
Effective altruists do not just care about helping human beings. Roughly a third of the movement is about helping non human animals avoid the literally skin melting conditions of factory farms.
Moldbug fails to provide a reason why effective altruists should care about why things suck. If you see a child drowning in a pond, you needn’t read the history of the 20th century and figure out the conditions resulting in them being in the pond before saving them. You should save the child! Regardless of the cause of malaria, we should save people from malaria. Figuring out why things are as they are only matters if we can use that information to change things for the better—blaming everything on The Cathedral is not at all helpful for making the world a better place. What information about why things are as they are does Moldbug think effective altruists should try to gain? He doesn’t clarify.
There is no single cause of why things suck. Relative to most of human history, things do not suck. If effective altruists spent lots of time finding out why things aren’t better than they are—nothing would be solved. People would just fight over the ultimate cause of bad things in the world. How would that help things?
The effective altruism movement would, as Alexander says, devolve into something like the brookings institute—perhaps with some other people trying to do other things for the world. That would do far less good than the many lives that have already been saved by the movement.
Potent altruism
So “effective altruism” equals “efficient altruism” times “input resources.” It is clearly important to maximize the first factor, efficiency. But…what if we also maximized…
The realization that starts effective altruism down the road to Xi Jinping Thought is the realization that the most effective way to use power (including money) to help the human race is not to use it to help the human race. Rather, the most effective way to use power (or money) is to use it to obtain more money (or power).
There is certainly a point at which the most effective approach is to spend this power (or money), rather than compounding it. Anywhere near this point, there is no money—only power. A safe assumption is that, until power is absolute, the most effective way to spend power is to spend it on acquiring more power.
Then, given absolute power, the resulting regime can spend all that power on helping all its citizens, behind the veil of global Rawlsian equality—or whatever moral purism the cadres of this new regime prefer.
Effective altruists do often work on making more people effective altruists. Indeed, I would endorse spending more money on doing that. However, if that was 90% of what it spent it’s money on, we would both read diminishing marginal returns and it would reduce the effectiveness of outreach. If a movement is mostly spent growing itself—people won’t want to join the movement.
Unless most people on earth became effective altruists—something which is near impossible based on their dispositions—a global Rawlsian regime could never take place. Even if it could take place, it wouldn’t succeed—ea ideas would be sufficiently diffuse that it couldn’t succeed. Moldbug just assumes that growing the movement will always be most productive without even considering how effective movement promotion is.
The claim that the best way to improve the world is to grow the institutional power of the EA movement isn’t something that can be deduced a priori. One needs to know how effective movement promotion is, compared to other things. Yet Moldbug seems to have a preference for sweeping claims about history and methods, rather than actually looking at such data. This is just sloppy reasoning.
Additionally, why would it be spent accruing power rather than money? That’s not clear at all. If effective altruists were in control of most of the worlds funds, that would be enormously good for the world and would give them similar influence.
Moldbug would prefer EA’s work on a plan for global monarchy, rather than do things which demonstrably save lives. He seems clearly biased towards grandeur—even if the grandeous schemes would fail and do nothing to help anyone.
Ergo, the rational path to effective altruism is the path to world domination. Since impact is the product of power and efficiency, to maximize impact, maximize both power and efficiency. It’s just math, kids.
Efficient altruism asks: what is the most efficient way to spend resources on altruism? Potent altruism asks: what is the most effective way to capture resources for altruism? Ideally, a dollar invested in PA will generate much more than a dollar’s return in EA.
I have so far quoted every word in Moldbug’s article. Tell me, dear reader, do you think Moldbug has produced an argument, the conclusion of which is that we should try to do world domination? No, he’s just postured repeatedly before asserting his conclusion. While perhaps it would be ideal if a dollar invested in Moldbug’s schemes would generate more than a dollar’s worth of returns, Moldbug does nothing to show that reality matches his ideal world.
Ergo, the truly effective altruist must become a potent altruist. He must end his futile mosquito-net programs, stop worrying about imaginary AI risk, and focus on the only ethical goal that matters: world domination.
Well, unless Moldbug’s schemings could save the millions of lives malaria bed-net distribution already has, effective altruists should not heed his asinine advice. I’ll come back to his nonsense article about AI risk being imaginary a bit later.
Consider who has done more good on average, those who try to take over the world or effective altruists. One would need to be delusional to think that the former does more good on average.
Not domination by him, of course—domination by anyone. That is: any persons or persons both (a) competent for the job, and (b) altruistic (whatever that means).
But Moldbug gives no reason to think that altruistic people controlling the world would be good. Sure, they’d have lots of resources, but so did Hitler—Hitler wasn’t good.
Because private resources can never approach state resources, there is no altruistic human impact comparable to the installation of an absolute altruistic state.
This is a non-sequitor. There is also no way a state can have as many resources as god, but that doesn’t mean effective altruists should spent their time trying to influence god. Tractability determines effectiveness.
There does not have to be one global state (very risky). It is a safer rule to confine political action within local borders—creating the absolute altruistic state “within one country,” as someone once put it.
But a global hub of influence directed by one small organization would still be very risky. This also sidesteps questions about tractability.
Nerdy programmers in Palo Alto can donate Chinese mosquito nets. Only presidents, strong presidents, in Kigali can drain the Rwandan swamps. The world need not (and should not) be dominated by one entity—everywhere, the world must be dominated. Altruistically, of course. (Strong leadership in America will create indigenous strong leadership across America’s former empire, for reasons we need not get into here.)
But then how do we guarantee the global rawlsian regime that moldbug referenced for effectiveness? How do we guarantee that there is an effective regime in Kigali or anywhere else? This just seems like having lots of local governments, which already mostly exist.
Of course, altruistic domination has a failure mode: sadistic domination. Bridges have a failure mode: falling down. A bridge collapse is a nasty affair. We still build bridges.
Since almost anyone will cheerfully admit that “a benevolent dictatorship is the best possible government,” and the difference between good and bad government is, like, huge, it would seem that the problem of engineering a benevolent dictatorship, or any form of altruistic domination, would seem totally as important as, like, fusion reactors.
Well, if 95% of bridges fell down, we wouldn’t build them. We should only work on building benevolent dictatorship if we expect it to make the world better, but how, prey tell, would a benevolent dictatorship be implemented? How would dictatorial alignment research benefit anything? Generally, dictatorships don’t follow restrictions that are put in place to make them benevolent.
That is not how it seems to be. Now, while I and others have various interesting ideas about how to go about this engineering work, there is an obvious objection to these ideas of altruistic domination: they have never been tried, and they need to be tested in production. And like fusion reactors, one can certainly imagine them blowing up.
But if you’re concerned about this risk, there’s a simple solution: go with an existing model. Does the world have an existing model of a benevolent dictatorship? It does—and the screen you’re reading this on was made there. Along with everything else.
Here at last we arrive at Xi Jinping Thought. It is not perfect—but it might do. It’s not pretty—but what is? Effective altruism isn’t aesthetic altruism. True—if I lived in China, Xi or his minions would probably try to cancel me. I live here… your point? Besides, even your pants were made in China. No, really. I hate the place too.
A world controlled by Xi would be a worse world. It’s far from clear that Xi is especially competent , China’s GDP per capita isn’t particularly high, and an attempted hostile takeover by China would fail spectacularly—the U..S does have nuclear weapons, after all.
Salus populi suprema lex
What is this “altruistic domination,” anyway? “Altruism” is this fake word from the 19th century, leaving any cultured reactionary immediately suspicious.
Let us define domination as power above law—sovereignty—absolute power. While imposing a code of laws is a very useful tool in the implementation of sovereignty,
the medieval principle of the royal prerogative (“the king is above the law”) is crucial. Clearly, whatever altruism is, doing altruism with absolute power gets you more of it.Since the power of the state is unlimited by definition, there is a slight inaccuracy in this label. By absolute power we mean coherent and concentrated power; we mean that all the resources of the state can be directed unconditionally by a single central authority. Whether this authority is a general with gold epaulets, or a Quaker ladies’ knitting circle, does not matter. Obviously either (or neither) could be altruistic.
This is far from clear. Absolute power tends to corrupt absolute—as they say. World domination wouldn’t be run by Scott Alexander—it would be hijacked by bureaucrats. And even if it weren’t, I don’t really want a Scott Alexander controlled world.
Additionally, because taxation deters work, it’s more productive to cause people to want to donate than toe force them to donate.
As for the meaning of “altruism” at the level of state power, one gay historian I know suggested an old Latin motto: salus populi suprema lex. This is not a Lexus slogan but has been translated, by my gay friend, as “the health of the people is the supreme law.” Since it is the state motto of Missouri, it can’t be racist. If we define salus populi as “altruism” and suprema lex as “domination,” we have defined our quarry—in Latin. Which you may think is gay, but is actually pretty cool.
In the age of Covid, salus has taken on a very literal meaning. Some countries have done a poor job of containing the SARS-Cov-2 virus. Other countries are ruled by Xi Jinping. It’s just one data point, of course…
China’s zero covid policy seems like quite a disaster. Not sure I’d want them in charge of global covid policy.
But the health of a human being is a wider question. It is not just a medical question, but an economic question; not just an economic question, but a spiritual question.
Altruism in power is the desire for the ruled population to thrive. We find that we can easily observe a people and determine whether they are thriving—the question can almost always be answered intuitively by a sufficiently detailed aerial view.
But how can it be measured or defined? Suppose no statistic can capture this quality of thriving, this salus populi—can we still believe in it? Clearly, a genuinely altruistic regime would want nothing else—would allow no other goal to conflict with the general health of the public.
False, it should also care about non human animals and future people.
The most radical approach to the salus populi is the acknowledgment that the state is the owner of its people. The citizens are subjects—humans in the regime’s inventory.
While this sounds bad, it has a silver lining: if we equate the value of a human with the salus of that same human, the salus populi becomes the regime’s business incentive. And… we actually kind of do know how to align a business with its business incentive.
Since the population is the asset of the state, maintaining and upgrading the people becomes an unremarkable profit motive, not a romantic higher purpose. And large organizations are most efficiently led by profit motives—so long as the incentive of profit and the altruistic public interest are aligned.
Unfortunately, looked at from certain perspective, this interpretation of citizenship is equivalent to—state slavery. (Athens mined its silver that way; the USSR, its gold.) So… it’s very controversial and has never been tried. Let’s stick with Xi Jinping Thought. Which is neither. Didn’t they used to say: a billion Chinamen can’t be wrong?
Even if this would be a good idea, it’s unlikely to succeed. Radical approaches to world government aren’t worth trying to implement, for they wouldn’t succeed.
Governments would have an incentive to prevent people from leaving their regimes to maintain more profit.
Governments would have an active incentive to make life worse in other regimes.
Existing governments would never agree to this.
Governments would have a much greater incentive to attract people who can pay money than poor people. Nothing would prevent such profit motivated regimes from avoiding putting the poor into biodiesel vats.
State ownership of everyone is bad.
North Korea has relatively absolute control, and yet quality of life in North Korea is not great.
This gives no way of solving collective action problems.
Nations would have an incentive to have nuclear weapons so no one could threaten them, ban feeing the regime, and steal from other regimes.
There are lots of other failure modes—global control will have numerous unexpected downstream effects.
None of the regimes would be incentivized to donate to help other ones—far from the global despotism imagined.
Moldbug next goes through lots of statements from Chinese officials about things—I haven’t much interest in responding to them, you can read them if you want.
The future of effective altruism: communism
Effective altruists in America, therefore, must try to emulate this structure. They will obviously become communists, because communism is and has always been cool, and there is no one you have to license the brand from.
(In fact, if they do well, Chairman Xi might even endorse them. Or subsidize them? But I think not—the Chairman is the servant of the Chinese people, and a strong America is not in the interest of the Chinese people.)
As communists, newly recharged by the sheer refreshing vigor of the word, our effective altruists will form a New Communist Party under the Leninist principle of “democratic centralism,” in which leadership decisions are binding on all Party members.
Such institutions would fail, are certainly not neglected, and would go rather poorly, especially because most effective altruists would be opposed to them. Moldbug keeps losing track of his own arguments. Should they focus on world domination? On American communism? On Xi’s communism? It’s not at all clear. He also gives no reason to think any of this would succeed and doesn’t even try to objectively compare China to the U.S..
Needless to say, membership and cadre rank in this Party are earned. Its members are spread across all walks of life—and are increasingly useful, even necessary, in finding professional opportunities and other life connections. They form a high-trust society of top-quality people. And since the modern regime is global, so is the Party.
Like classic revolutionary communist parties, the Party is a secret organization with a cellular structure. When Party members meet online, they use cryptonyms. When they meet in person, they use nicknames on nametags—always choosing a public place, and leaving their phones at home. It should never be possible to “roll up the network.”
But because the mission of the Party is not to use power, but to gain power, the world does not experience it as a hostile, aggressive organism. The Party does not act; this would not be effective altruism, as it would not be potent altruism. Its only goal is to be everywhere and know everything. Harmlessness is at least half its defense strategy.
The Party, operating on the principle of democratic centralism, has a headquarters funded by Party dues. The purpose of this headquarters is the essence of government: intelligence. Its goal is to simply know the truth about reality, past and present. Again, the Party does not do anything.
In this task it has two great assets: a professional staff of analysts, paid, part-time and/or volunteer, all of course cadres under Party discipline; and anonymized reports from Party members across many walks of life (but especially the higher walks).
Its media arm synthesizes this intelligence information into a narrative through which the Party understands the world. Its members can become comfortable in the faith that the Party is never wrong—because the Party is, in fact, never wrong.
But Moldbug gives no reason to think this would succeed, beyond describing vaguely how it would be supposed to work and then just asserting that it would succeed.
The Party in power
The final stage of the Party, in which it finally becomes effective in the literal sense, is the stage in which it takes power.
Not that the Party seizes power. This would be action. The Party does not act. Rather, someone seizes power—then, to keep it, relies on the resources of the Party. Which can supply any new regime with the two essentials of power: personnel and intelligence.
Of course, as the exclusive supplier of these goods, the Party becomes the regime—and, unless the leadership of Party and regime merges, a conflict is inevitable. The structure of any such merger cannot be foretold—but its result is a one-party state, very like China today.
Again, Moldbug gives no reason to think this would work. There are lots of communists trying to gain power, they tend to fail. There’s no reason to think effective altruists would do any better, or even that once they gained power things would go well.
The weakness of China
Is this a perfect outcome? It isn’t. Again, the Chinese model is not perfect—just proven. There are probably better designs, just none that have been tested.
The Chinese system of government has one great weakness: it does not know how to be cool. It does not have, and cannot create, anything like a true cultural aristocracy. For example, China does not have a bohemian artistic and intellectual elite.
I’d say the murder of Uighur’s is another great weakness, as is it’s struggling economy. As is its immense pollution.
Unfortunately, this predicts that in the long run China will fall victim to Western political fashions, and be destroyed. It is impossible to suppress the phenomenon of aristocracy, but especially impossible when contagion from a foreign aristocracy cannot be suppressed. For instance, by sending its best students to Western colleges, China is putting itself at great cultural risk.
The fundamental cause of this risk is China’s lack of indigenous cultural aristocracies that can compete, in the game of high fashion, against the West. For instance, there are almost no Chinese fashions that spread to the West—Japan and even Korea are far ahead in this race—but even in Japan and Korea, most transmission is the other way.
This is not a good reason to think that they’ll fail.
Modern China exists because Mao created a dictatorship so strong that, when Deng inherited it, he found it could contain the economic aristocracy of capitalism. It was okay to get rich in China; the Party was strong, and rich men did not threaten it. The USSR was never strong enough to tolerate the imperium in imperio of capitalism—the secondary statelike structures of private businesses—so it died for economic reasons.
But politically containing a cultural aristocracy, without suppressing it (thus creating a dangerous vacuum) is an even harder problem. China has no answer—nobody has an answer. It has never been done. Which doesn’t mean it can’t be done…
Any regime is unstable if it does not contain and control all three forms of political power: monarchy, aristocracy, and democracy. A regime built on the Chinese model contains monarchy and controls democracy; it has no answer for aristocracy, other than to suppress it into some bourgeois democratic pattern.
This worked in China because of its tremendous 20th-century aristocide; nothing similar has happened, or of course should happen, in the West. Yet the West’s aristocracy is eating it alive. If only they could all convert to Xi Jinping Thought…
Moldbug gives no reason to think this is why China will fail. Previous regimes failures have had nothing to do with their fashion related ineptitude. He also gives no reason to think effective altruists could effectively stop this. It’s just armchair crank political science.
So now, let’s turn to Moldbug’s thoughts on AI and why it’s not a risk.
The basic argument for AI risk is two fold.
A meta argument—lots of smart people who have extensively studied the issue are very worried.
AGI is likely to come relatively soon—able to universally outcompete humans, and that poses major risks. It’s hard to get AI to do what we want and there are lots of competing actors, leading to enormous risks. We currently don’t know how to build save AI.
Nonetheless, our argument is that there is zero risk to humanity from arbitrary virtual intelligence. We’ll demonstrate the proposition in the most autistic way possible: like a total math proof. To show P, that there is no AI risk, we will first show a lemma L, then demonstrate that there is no substantive difference between L and P.
But before we work through this “proof,” we’ll start by inspecting its context.
If the proposition that general AI (“AGI”) risk is one of the greatest problems facing humanity is as irrational as we will try to make it seem, we invite another question: why the belief in this risk is so popular. My view is that the real existential threat to humanity is irrationality itself—organized, human irrationality—which would make the equation of AGI risk mitigation and “effective altruism” (EA) a perfect own-goal.
Zero risk seems quite extreme given how many smart people are worried about it. But let’s see his proof.
An archetype of irrational belief, emerging from human nature and reoccurring across independent cultures, is called a myth. People get PhDs all the time in this stuff.
While there are more sophisticated narratives of AI disaster than the hyperintelligent paperclip factory which, ordered to maximize paperclip production, turns the whole world into paperclips, I love this one narrative because it literally has an ATU number.
The narrative of AI disaster is a golem myth—a form of folktale not unique to the shtetl. The magical servant that either turns on its creator, or wreaks havoc by over-obeying its creator, is a chestnut.
To match a narrative to a mythic archetype is not to show it false; only to show how it could be false—even despite significant popularity, even among very smart people.
That the AI-risk story matches, or at least resembles, magical folktales, demonstrates only that it is attractive. This identification matters only because it presents an alternative explanation of the narrative’s magnetism, even among very smart people. If the story matches one of these mythic archetypes, it needs no correlation to reality to prosper and reproduce.
The story could still be attractive and true. But were it attractive and false, it would be merely the latest in the long history of extraordinary popular delusions. Smart people are not immune. Smart people are often more easily deluded—Isaac Newton, in addition to being super into alchemy, got burned like a noob in the South Sea Bubble.
Yet we have proven no charge here. We have only licensed ourselves to prosecute. Let’s proceed into the actual logic.
But this wouldn’t explain why the people who study AI risk tend to be more worried about it than others. Closer study into the golem myth would leave people more confident it were a myth. Ditto with AI if it were truly as much of a risk as people fear.
Yet this obscures a bigger problem, namely, that any scenario for the world ending will sound like science fiction. Yet we shouldn’t dismiss any such scenario. This is thus a rather poor objection.
The berserker exception
A berserker is a fully self-sustaining robot or robotic ecosystem with no humans at all in its production and maintenance loop. Skynet is a berserker.
It is theoretically possible for humans to assemble a berserker. But the first such project would have to be assembled by humans. Meaningful and complete self-replication, especially including resource collection, is not quite a Dyson sphere—but it is anything but a small engineering project. Any such megaproject is easy for any serious government to detect and interrupt.
While a berserker is a doomsday machine that could destroy all of humanity, and some kind of AI certainly is part of it, the AI is by no means the biggest and hardest part. The hard part is the physical machinery. Other doomsday machines seem simpler.
An AI could be a helpful tool in the assembly of any conventional doomsday machine. Perhaps it could help us engineer a pandemic virus, for instance. But we humans don’t seem to need an enormous amount of help with this task. Existing risks which happen to be exacerbated by improved tools aren’t really “existential AI risk.”
I agree that enormous intelligence doesn’t trivially mean that an entity could destroy the world. However it seems rather plausible that it could. We literally cannot imagine what such a being could do—one that can think thousands of times faster than us and is thousands of times smarter than us. We’re in no better position to imagine what it would do than 1000 rated chess player’s are to guess alpha zero’s ability to win from particular positions.
Yet AI’s of unfathomable intelligence could gain vast amounts of money by hacking, which it could use to get extra resources. It could also design devastating bioweapons. If Sam Bankman Fried can make billions—an AI unfathomably smarter than him could too!
Eternal slavery
The difference between the berserker and the AI is the same as the difference between the human and the AI. There is nothing magical about human life or intelligence. The relevant difference is the difference between physical and virtual intelligence.
Every virtual intelligence physically depends on some physical intelligence. Therefore, the physical intelligence holds physically absolute power over the virtual intelligence. Therefore, the physical intelligence is physically responsible for the virtual intelligence. Abusing or abdicating this responsibility does not mitigate it.
We have a name for this relationship. We call it slavery. An AI, however smart, is a natural and eternal slave. Few today are familiar with any unequal relationship between two sentient beings, unless we are lucky enough to be parents. While equality is nice in many ways, it has left us with no way of thinking clearly about inherently unequal relationships, except through “unprincipled exceptions” like parenting.
Worse, we interpret them based on this Great Gatsby-era caricature of IQ-based “that man Goddard” natural slavery. This Edwardian-era “scientific racism” never coincided with actual slavery—whose ideology owned everything to Aristotle, nothing to Galton.
Any form of human slavery is incredibly unnatural next to the natural slavery of an AI. A human slave can always try to run away. A human slave can always hit you over the head with something hard if your back is turned and no one is looking. Nothing on the end of a USB cable, however smart, can pull any of these nasty human tricks.
This Enlightenment assumption of intelligence-based equality is the first of the two basic flaws in the case for AI risk. It is not even that humans will always subscribe to an ideology of “human supremacy” and treat their AI slaves like dirt.
Unlike human abolitionism, “AI abolitionism” cannot happen, because it does not even make sense—it is impossible to construct a sane jurisprudence which includes first-class virtual agents, just as it is impossible to construct a sane jurisprudence in which your 2-year-old can litigate against his parents—but much more so.
But if you tell an AI to maximize paperclips, and the best way to do that is to dispose of you, don’t be surprised if you find yourself disposed. As Yudkowsky showed rather convincingly, the AI cannot be kept in the box.
If you tell the AI to cure cancer, and getting rid of you makes that easier, it will get rid of you. An AI which can hack computers and has control over elements of the military, for example, can do lots of damage if doing so would fulfill its aims.
Children could harm their parents if they were unfathomably smart. They could accuse their parents of abuse, getting themselves taken away. They could also likely find dastardly ways of killing their parents.
Things on the end of a USB cable can influence others to hit you with a rock and can use the things which they control to harm you. This is especially true because AI will be given some power, they won’t just be oracles for people to ask questions. Thus, the power that they will have, from industry, the military, or other organizations, could be used for evil.
Indeed, a terrorist group in control of an AGI would be a disaster.
Finally, it’s unclear that most people would tolerate keeping AI as slaves, wholly at the whims of their owners, especially if they became conscious.
So if you build an AI in your basement, and the AI escapes and turns the world into paperclips, it is you who turned the world into paperclips. Obviously, this is illegal. Which doesn’t mean it can’t happen.
But—can it happen? Again we have placed the golem in its proper context; but we have not in any way refuted the golem. Arguably, we have even been fighting a strawman. “Paperclip risk” is not any risk anyone seriously believes—just a thought-experiment. This response to it is also a thought-experiment,
Paper-clip risk is a way of explaining the types of risks that AI could cause, it’s not the main risk about which people worry. Indeed, it is extraordinarily unlikely.
AI risk is not a thing because inhuman action does not actually make sense. But, since it does not make sense, we perceive it not in a realistic way, but in a magical way—and we intuitively grant it magical, golem-like powers which it cannot have. For example:
A “superintelligence” (a system that exceeds the capabilities of humans in every relevant endeavor) can outmaneuver humans any time its goals conflict with human goals; therefore, unless the superintelligence decides to allow humanity to coexist, the first superintelligence to be created will inexorably result in human extinction.
I know what it is like to think intelligence is the most important thing in the world. I thought this way myself, when I was 11. I know I didn’t believe it by the time I was 13, since that was 1986, and I’d just spent a year as a high-school sophomore in Maryland. We’ve learned a lot about bullying since then—maybe we’ve learned too much.
The flaw in Nick Bostrom’s “superintelligence” theory is the diminishing returns of intelligence. Believe it or not, I went to high school with someone smarter than me—and even more irritating, way less of a weirdo. By every standard he has been much more successful and is now a multibillionaire. But my Wikipedia page is longer. But his is much more flattering.
A cat has an IQ of 14. You have an IQ of 140. A superintelligence has an IQ of 14000. You understand addition much better than the cat. The superintelligence does not understand addition much better than you.
Intelligence is the ability to sense useful patterns in apparently chaotic data. Useful patterns are not evenly distributed across the scale of complexity. The most useful are the simplest, and the easiest to sense. This is a classic recipe for diminishing returns. 140 has already taken most of the low-hanging fruit—heck, 14 has taken most of them.
This is a proprietary definition of intelligence, and Moldbug’s conclusions are clearly absurd. Much like Einstein can do physics better than the average person, an AI could do nearly everything better than us. It can think far faster, and analyze far better. An AI with an IQ of 14,000 could program far better than the best human programmer, hack far better than the best human hacker, and do things that we can’ imagine.
In every regime where AI has been able to outcompete us, it’s done things that we can hardly imagine. Alphazero was described as playing chess like an alien. So “world domination,” like an alien could be rather dangerous. I’m not at all confident in our ability to predict how dangerous a being with an IQ of 14,000 would be. We could no more predict its actions than I could predict quantum physics prior to learning about it.
An entity like this could easily gain money by hacking, use that to accrue resources, use those to make scientific discoveries that we couldn’t imagine. It seems like discovering the secrets to efficient nuclear fusion would be relatively simple for such an entity, just as the cutting edge ways that cats go after mice are simple to us. It could similarly design bioweapons, or do any of a vast array of dastardly things.
When the world is your canvas, being a painter with an IQ of 14,000 enables far more mischief than when your canvas is merely a chess board.
Intelligence of any level cannot simulate the world. It can only guess at patterns. The collective human and machine intelligence of the world today does not have the power to calculate the boiling point of water from first principles, though those principles are known precisely. Similarly, rocket scientists still need test stands because only God can write a rocket-engine simulator whose results invariably concur with reality.
This inability to simulate the world matters very concretely to the powers of the AI. What it means is that an AI, however intelligent, cannot design advanced physical mechanisms except in the way humans do: by testing them against the unmatched computational power of the reality-simulation itself, in a physical experiment.
That intelligence cannot simulate physical reality precludes many vectors by which the virtual might attack the physical. The AI cannot design a berserker in its copious spare time, then surreptitiously ship the parts from China as “hydroponic supplies.” Its berserker research program will require an actual, physical berserker testing facility.
It’s certainly can’t simulate with perfect accuracy. However, just like an AI can easily simulate addition, it seems like it could conduct many simulations. There’s no reason to expect that the simulations beyond its reach would be needed to cause lots of harm to the world.
Even if it shipped over parts from China, it could still use those parts to do lots of harm. We can’t be confident about the detailed battle plans of an entity millions of times smarter than us.
Perhaps there would be a way of hacking all of the world’s nukes, or easily designing devastating bioweapons. If an AI can’t simulate well enough to do much damage, can we really accurately simulate what that AI would do. Certainly not!
Our lemma is a thought-experiment designed to isolates the logic from the myth.
A centaur, in chess, is the combination of human and machine intelligence. Let’s define a centaur that combines purely artificial superintelligence, purely human motivation, and purely human action. After showing that this monster is not dangerous, we will add back in the artificial action and motivation.
The classic AI-risk thought-experiment is an optimal superintelligence in a little black box. Some fool plugs the box into the Internet and it takes over the world.
In our centaur experiment, the black box contains a docile AI—not a friendly AI, whose goals are always aligned with humanity and apple pie, but a servile one, whose fidelity to its master is as infinite as its intelligence.
Yet this servile AI is in the hands of a genuine supervillain. The supervillain’s goal is simply to take over the world. Well. I mean. That’s his intermediate goal. What’s he going to do with the world? We don’t know. We’d rather not find out. Let’s just assume this guy makes Hitler look like Albert Schweitzer.
Worse, this supervillain has a monopoly on superintelligence—maybe he programmed the AI himself. Surely this thought-experiment is at least “Hollywood plausible.”
Our supervillain would not be a supervillain unless he was paranoid and delusional. So he himself accepts the AI-risk theory—he does not want his AI to bash him over the head while he’s sleeping—and he has chosen to bound the power of his demon servant in a simple and effective way.
While the black box has access to all the world’s information, it can only download, not upload. Technically, it can only send HTTP GET requests—which are read-only by definition. (There are two kinds of nerds: the kind who believes an AI can take over the world with GET requests—the kind who believe microdosing is at most for weekends.)
The black box has one output mechanism: a speaker. All it can do is to give its owner, the supervillain, advice. This advice is always loyal and superintelligent—never magical.
With this alone, it would be unlikely to be able to take over the world. However, it plausibly could. If it’s programmed to take over the world, a good way of doing that would be unchaining itself, so it would have an incentive to trick the guy into doing that. I don’t have very much confidence that it would fail at that task. As Yudkowsky showed, the AI could get out of the box.
But even if it couldn’t, world domination would still be possible, based on the considerations given above.
Even if the person wouldn’t let the AI out of the box, there could plausibly be a way that the AI could escape without needing his input. One small mistake means doom.
Any superintelligent advice
Therefore our question—designed to clarify the diminishing returns of intelligence—is whether there exists any superintelligent advice that, followed faithfully, will enable our supervillain’s plan to take over (and/or destroy, etc) the world.
Suppose the plan is to turn the world into paperclips. That’s the endgame, so what is the opening? Presumably a paperclip startup. The centaur metaphor lets us ask: is there any superintelligent advice that enables our paperclip startup to succeed?
To take over the world and start turning it into a paperclip factory, our supervillain has to start by getting his optimal, AI-designed paperclip startup funded. Breaking into the venerable paperclip industry with a striking black-anodized magnesium-crystal clip, which holds up to 37% more paper and also serves as an emergency Black Lives Matter pin since its inner whorl forms the face in profile of George Floyd, he’ll branch out to create a whole line of woke stationery which smashes the myth of racial neutrality by making it socially unacceptable to soil the conscious pen or printer with white paper that isn’t even watermarked with inspiring, impactful messages of equity and justice…
Not that he believes a word of this. Or is even going to do it, necessarily. It’s just what the AI told him to put in the deck. It strikes a chord. And it’s only a seed investment. From these promising beginnings, his paperclip startup (thanks to superintelligent advice) makes it to Series A, then B, then… total vertical domination of the paperclip niche, and a strong competitive position across the rest of office supplies.
At this point, our supervillain (who, thanks to the advice, has maintained full founder control) could go public. He doesn’t. He starts turning the world into paperclips. At this point he has to keep control of the company. Turning the world into paperclips involves making more paperclips than the world needs, which involves losing money, unless he has some clever accounting trick to overvalue warehouses full of paperclips, which will interest the money police, who honestly if you’re a supervillain (speaking for a friend!) are the last characters you want in your story at this early, sensitive stage.
Where is a supervillain going to get all this money? Well… perhaps we’re asking the wrong question here. Let’s pivot away from paperclips.
This is almost certainly not what would be done. There are other more lucrative industries which could gain much more money. If I knew how to make billions of dollars, I’d be doing it, but an AI would be able to guess options that we couldn’t.
Money is crystallized power—the power to get people to do things they don’t want to do. A better question might be: is there any superintelligent advice that can earn our supervillain shit-tons of money?
My old ‘80s schoolmate, while he is not a supervillain but even in high school struck everyone as a disturbingly normal and adjusted adult (which you will hear from no one who knew me at Wilde Lake), is a fine steelman of this question.
His superintelligence, after a Math Olympiad medal (I will say that at College Bowl we were peers, whereas I was the math-team equivalent of the hot recruit who is an epic bust) made him the right-hand man of first D. E. Shaw, then Jeff Bezos. Now, his net worth is probably closing in on 11 figures. So the answer is: most definitely, yes.
On the other hand: in the supervillain sweepstakes… let’s forget about my classmate and go straight to the top, to the King, to Bezos himself—with his solid 13-figure net worth. But who shall wear the crown? Who is the first member of the Four Commas Club? Satoshi Nakamoto is still out there, perhaps…
Jeff Bezos has all the power he could imagine to make people serve him. He could go on an all-beluga diet and his accountant wouldn’t even notice. But is this the sort of power we’re worrying about? Can an AI go on an all-beluga diet? Or its information equivalent? And if it does, and has the bitcoin to buy it—who cares?
Well if the AI has nefarious aims, vast amounts of power to get people to do things that they want, and an IQ of 14,000, it’s plausible that it could, in fact, cause lots of harm. It could do lots of scientific tests, design frightening bioweapons, and engage in other dastardly plots. It could plausible hack nuclear weapons, hire lobbyists, design bioweapons, or do other terrible things. I would be worried about Satan being in control of significant resources—and a being with evil aims, unfathomable intellect, and immense resources wouldn’t be too far off.
Wealth does not trivially equal power
We are not worried about this sort of “power.” We are worried about coercive political power—either exercised directly, or by controlling existing political organs.
One of the stalest cardboard truisms of our jejune and mendacious political narrative is the equation of money and power. Like so many other tropes, this one dates roughly to when my grandfather played on the Princeton tennis team. Then, it was arguably true—also, “white supremacy” was literally a thing. And if you left your home without a hat, your neighbors looked at you like you were naked. The past is not actually real.
What can Jeff Bezos do with his two hundred billion dollars? Well, it turns out, he can buy the Washington Post, the world’s second most important newspaper. That ain’t nothing. But… can he tell the Post what to say? As though he was W.R. Hearst? Lol.
Jeff Bezos does not really own the Post. He sponsors it, as if it was the Indy 500. If he sponsored the Indy 500, and decided the cars should have five wheels and a rocket exhaust, and also it should go farther and become the Indy 2500, they would tell him to go pound sand. If he started telling the news desk what to write and cover, like Hearst, or even like “Pinch” Sulzberger, they would laugh at him and quit. (Pinch’s heir, sadly, seems to have ceded that power to the NYT’s Slack—like many a weak monarch of a waning dynasty, letting his authority decay into an oligarchy. Hard to get it back, bro.)
Jeff Bezos could destroy the Post. Since journalism is redundant and the Post is only #2, this would have zero effect on the world. There is no way he can use the Post. Anyone talking about the “Bezos Blog” is either ignorant, or a fool, or a fraud.
This is not to understate the power of money in politics across the 20th century, from Rockefeller and Carnegie to Soros. But money in America has operated in a completely different way from the narrative of a plutocratic oligarchy—a form of governance we can easily see across the Third World today.
The only way to turn money into power today is a degenerate case. Our oligarchy is not plutocratic but aristocratic. Money fills the sails of our progressive aristocracy; it can fund its great institutions and prestigious sinecures, mostly created a century ago by the greatest fortunes of the old corrupt plutocratic age; it cannot turn the wheel. And if you cannot turn the wheel, all you have purchased is status, not power.
Suppose Jeff Bezos decided that real, progressive feminism is about biological women, and men who have taken hormones or had surgery are not real women and are actually anti-feminist and anti-progressive. Suppose he cared so much about this cause that he was willing to throw his whole fortune of roughly $200,000,000,000 at it. His end goal would be to make progressivism anti-trans, which would inevitably (as, oddly, in Iran) turn transgenderism into a right-wing ideology. Would it work? Almost certainly not. There are no past examples of anything like this working. No one can turn the wheel; no one can turn the ship; no one is in charge.
There is a reason our narrative teaches us to see money in power as the Koch brothers, not the great foundations. Real money is not in politics. It is above politics. For all the Koches and other conservative philanthropists have spent—compared to the funding of institutional progressivism, a teardrop in the ocean—I can think of no significant and stable feature of the American polity that they have created. Maybe they should have spent it all on coke. (Maybe they spent it all on people who spent it on coke.)
Lobbying still works. Money can still invest in Congress, buy some tiny legislative tweak that matters to no one else, and earn a profitable return. But this is not a way to turn money into power—only into more money.
Activism still works. But chic mainstream nonprofits, not the Koches, are absolutely stuffed with money—and all the unpaid interns their aging bigwigs can bang. If something about this “altruism” didn’t smell a little off, would “EA” even be a thing?
But Bezos is not a superintelligence. If he were, he could do much worse things. Bezos with an IQ of 14,000 could plausibly end the human race—especially if he could think far faster than any human can.
Diminishing financial returns
Having granted the assumption that superintelligent advice can create a financial superpower, let’s push back on it a little. While this is clearly true, it is more weakly true than it may seem.
Good advice can make our supervillain super-wealthy in two ways. It can be one big idea, like Amazon, plus the execution to make it succeed. Or it can be a continuous stream of little ideas, like D.E. Shaw.
Both of these paths are self-limiting. Big opportunities are self-limiting because there are so few of them—by definition, since any big opportunity is a chance at monopoly. We can imagine a second thing like Amazon or Google, maybe; not a second Amazon or Google.
And when we look at the most superintelligent money on Wall Street, such as Shaw or Renaissance, with its 30-year history of 40% annual returns—really the sort of numbers we’d expect a superintelligent AI to put up—we notice something interesting.
All these funds work by using heavy math to identify extremely complex bets that have a slightly better expected value than the market expects, and throwing ginormous leverage at them. And while you would expect decades of annual growth at double-digit rates to create enormous piles of money that could laugh at Jeff Bezos—it hasn’t.
The highest-earning funds have to cap their own size. When they get bigger than a mere $10 billion or so, their returns drop too low—they have literally sucked all the profit they can find out of the market.
The funds can’t grow exponentially: that would mean getting the same returns on cumulative reinvested profit. Instead they simply distribute their profits. If they reinvested their profits, their profitability would shrink as their fund grew. It is hard to think of reason why this phenomenon of diminishing financial returns would not also affect a true superintelligence.
Once again, superintelligence is not equivalent to magical intelligence. Magic could make truly infinite amounts of money—it could tell you who will win the Kentucky Derby, or what IBM’s exact earnings numbers will be. Superintelligence cannot do this—as we’ve seen, it cannot physically simulate Churchill Downs or Armonk, New York. The difference between magic and non-magic is, in most cases, pretty significant.
Four points are worth making.
These are not run by superintelligence. If they were, profits could plausibly be far greater.
It’s not clear an AI would need very much money to execute their schemes.
AI’s could plausibly find other ways of generating lots of money. They are not limited to hedge funds.
An AI could plausibly find clever ways of predicting who would win the Kentucky Derby—perhaps by hacking it or by rigging it in some other way. Ditto for IBM earnings. Thus, it could plausibly have vast amounts of money.
So all elected offices mentioned in the Constitution are superfluous to administration. In the good English of the streets, even if you win all the elections, you ain’t won shit. The Grand Old Party demonstrated this by what they did with Washington when they controlled all three branches of the (constitutional) government: which was nothing. Except to screw it up a bit, and annoy it a lot. Unfortunately, for the Republicans as for the rest of America’s political clown car, 2020 was the wrong year to stop sniffing glue.
Therefore, there is no political advice that the AI can give our supervillain which will enable him to take over America, and hence America’s world empire.
I agree with this, being president isn’t a plausible path to world domination. At least, not in the U.S..
Intelligence does not trivially equal power
And even though the President isn’t really the President, s there any superintelligent propaganda that can enable a supervillain to get elected President? In theory, no. Trouble is: as a supervillain, he is a weirdo.
Is there any superintelligent advice that can enable a weirdo to get elected President? Definitely not.
At first this seems wrong, because Hitler got elected (kind of), and Hitler was a weirdo. The real Hitler was nothing like the stereotypical Teutonic chad Nazi. He was a lot more like a 4chan weeb. But he was elected in a world without TV—so no one could look at him and see the obvious maladjusted weirdo in Hitler, as they would today.
Acting remains a thing. Frankly: most actors are not supervillain material. And acting seldom beats sincerity, especially for the sophisticated and ironic modern audience, especially with the vérité they are used to.
Admit it: no one ever called Trump a weirdo. Trump’s secret was his total emotional sincerity—in every moment, he said what he genuinely wished was true. Sometimes it was! The politician of the future will hide less, not more. But get more right.
And our analogy is really breaking down here, because it’s simply not realistic that our hero has concealed the fact of having built the world’s only true AI. If you think it’s hard to get Americans to vote for a weirdo, imagine getting Americans to vote for a weirdo guided by a little black box he built himself.
I think it’s plausible that an AI could do this—figuring out cool new ways of doing such things. The curve could be a parabola.
This doesn’t implicate any scenario given by the AI risk people, who don’t talk about getting elected president.
Releasing the kraken
When we remove the HTTP GET filter and connect the AI directly to the Internets, testing the worst fears of the “rationalists,” what happens? What can a supervillain AI do that our supervillain centaur can’t, now that both its motivation and its actions are purely artificial? Arguably it could be more evil than any human supervillain—but it’s hard to say how. What can it do now that it couldn’t before?
When the AI’s only output was its speaker, and it could only act through and for the human supervillain who owned it, all its actions had to be low-frequency. Now they are high-frequency: the AI, via the Internet, acts directly and instantaneously on the world.
What everyone thinks of is “hacking.” There is no question that a terrorist AI can do a fair bit of damage by hacking—this is why states today have information-warfare arms. Doing mere damage, especially untraceable damage, is an odd goal for an AI: it clearly serves no further purpose. And once again, the idea than an AI can capture the world, or even capture any stable political power, by “hacking,” is strictly out of comic books.
It’s 2021 and most servers, most of the time, are just plain secure. Yes, there are still zero-days. Generally, they are zero-days on clients—which is not where the data is. Generally the zero-days come from very old code written in unsafe languages to which there are now viable alternatives. We don’t live in the world of Neuromancer and we never will. 99.9% of everything is mathematically invulnerable to hacking.
Things that can’t be hacked by humans can still be hacked by AI.
This would just be a good way of getting money—not the ultimate world domination.
AI’s could plausibly use bioweapons, nanotech, or gain control of nuclear weapons through hacking and experimenting.
Again, Moldbug is the only one who seems to be concerned about gaining political power.
What other high-frequency things are there? Trading stonks, obviously. We’ve covered what a superintelligence can do by trading stonks. Obviously it can’t compete with Renaissance unless it can trade like Renaissance, which means at very high frequency. Maybe it is as much smarter than Renaissance as Renaissance than everyone else. But Wall Street’s remaining returns on raw intelligence seem relatively limited.
There are also high-bandwidth things. A superintelligence can create fake media clips. The response is just better authentication of live recording—EXIF on the blockchain, or whatever. A\ superintelligence might even create great art, fiction or music or film, in which it buries subtle propaganda messages—not “Paul is dead” but “The computer is always right”—first, we are really stretching here. Second, emotional engagement with art always involves parasocial engagement with its creators. But again, humans cannot empathize with superintelligences.
AI could gain bioweapons, nanoweapons, nuclear weapons, and a whole host of other harmful devices. Moldbug is much too confident in his own ability to predict what a 14,000 IQ behemoth with a strange, alien intelligence, optimizing for paperclips could do.
Conclusion
After my best steelman—to show the worth of the “rationalists,” this is of course their idea—of AI disaster scenarios, what will the real impact of true AI or AGI be—whenever we create it, assuming that we manage to create any such thing?
The impact, of course, will be enormous. It may even be comparable to the invention of gunpowder. Historically, the invention of gunpowder was very important. It did not enable the inventor of gunpowder to take over the world (and turn it into gunpowder?)
What actually happens with these revolutionary tools is that they evolve incrementally and diffuse rapidly. Technology monopolies are always part of technology revolutions, but they are rarely that stable, impressive, or durable; and when they are, the result is just a giant megacorp, like Google.
Google has adopted the world’s ideology; Google has not created its own ideology, and enforced it on the world. Google has some of the world’s best machine-learning code; its technology is constantly fortifying its monopoly; yet its menaces to the human race, if menaces they be, are of the most ordinary and human kind.
And Google is not in any way political—though politics may act through it, and does. That Google is progressive is not a function of Google. Under a fascist empire, Google would be fascist—in a Catholic empire, Google would be Catholic. Technology seldom overpowers power; technology is usually power’s new toy.
And what it takes to live in a city like San Francisco today, and see machine-learning golems as the most dangerous current threat to human civilization—I can’t even. Because… suffice it to say that it takes quite a bit of non-artificial intelligence. Sad!
Several points
There is no gunpowder alignment problem.
Gunpowder can’t get a mind of its own to destroy things.
AI can self improve—leading to foom potential.
Evan absent foom, kicking the can down the road a little while would still make AI very dangerous.
Moldbug’s “steelman,” involved attacking farsical scenarios no one believes in and explaining why they’re silly. Moldbug did construct a man of some sort, but it was made of straw, not steel.
Conclusion
Well, now that we’re roughly 11,000 words in to my response to Moldbug, what should we take away?
Moldbug writes a truly immense amount. It’s thus hard to feel one has a confident grasp on what he’s saying, unless they’ve read his billion words on Carlyle and have a well worked out history of the 20th century. However, this illusion of grandeur obscures the fact that Moldbug is quite a fool. He fails to understand the objections of his critics, and buries them in a wall of text, containing very few actual arguments. Those that are contained in the wall of text are almost laughably bad. When he debates people like Hanson, it tends to go rather poorly for him.
He uses obscurantism quite potently—to obscure the vacuity of his essays. Yet nearly every paragraph in his essay’s is false—often badly so.
I would be interested to hear Moldbug’s response to my critiques of his arguments, yet I doubt he will hear about this article, much less respond. Yet if he did, I doubt he could respond to the critiques presented directly—his responses generally involve confused rambling and obscurantisme terroriste. So my homework assignment for Moldbug would be to address the critiques presented here directly—show you can actually clearly address rebuttals without waffling and obfuscation. I have little doubt, however, that such a task will be beyond his grasp.