Ted Chiang Is Wrong About AI Consciousness
Plus some other stuff
Whether there might be digital minds is a topic of immense importance. Most expected future beings will be digital. This holds even if you’re pretty sure that AIs can’t be conscious, because if AIs are conscious, they could be incredibly numerous. If we attribute consciousness to them when they’re not conscious, we might pass bad policies—attempting to safeguard the welfare of a being who doesn’t have any welfare. If we deny that they’re conscious when they are, we might mistreat beings horribly in huge numbers.
This is a topic that merits careful thinking and precision. Pieces on the topic should acknowledge uncertainty and carefully distinguish between the question of whether current AIs are conscious and whether future AIs will be conscious. What a good piece on this topic looks like is a careful analysis of under what conditions, if any, AIs might be conscious. What a bad piece looks like is confident proclamations that AI won’t be conscious based on overconfident misunderstandings of philosophy of mind.
Sadly, the recent piece published by the Atlantic falls into the latter category.
It was written by the famed science fiction author Ted Chiang, who inspired the movie Arrival. Chiang is a brilliant writer of science fiction and many of his stories are provocative and interesting. But all too often, when brilliant people venture outside of their area of expertise, they make serious errors.
Now, my best guess is that current AIs are not conscious. This seems to be the conclusion of the best reports that have been done on the subject. I suspect that digital consciousness is possible in principle, and might even emerge in the relatively near future. This is in part because alternative views imply that if you switched out your neurons with functional duplicates, your conscious experience would gradually fade without you noticing. When many leading experts in the field got together to commission a report in 2023, their conclusion was:
Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.
If one simply canvasses the standard theories of consciousness, these generally imply that AI could be conscious in principle. One of the leading theories is the global workspace theory: it seems reasonably likely that in the not-too-distant future, AIs will have something relevantly like a global workspace. Even views of consciousness on which consciousness depends on substrate generally imply the possibility of digital consciousness in principle, so long as the Silicon entity is organized the right way.
At the very least, it seems hard to be confident in any theory that would rule out digital consciousness. We are very early in our understanding of consciousness. There are no knockdown arguments either way. So it seems we shouldn’t be extremely confident in either direction.
Ted Chiang disagrees.
Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?
No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work.
Set aside the oddly pessimistic framing “Generative AI is harmful enough when we understand it as a conventional technology.” If you are extremely confident in something, you should have good arguments for it or it should start with a very high prior. On priors alone, you shouldn’t be extremely confident either way on AI consciousness—if a thing can talk fluently and displays complex behavior, it would be dogmatic to start out extremely confident in it not being conscious.
So what is the amazingly powerful argument supposed to be that leads to him being near-certain?
First, Chiang notes that the mere fact that something produces detailed textual outputs doesn’t suffice to show that it’s conscious. If an LLM makes a story with Julius Caesar and Genghis Khan, nobody thinks that Caesar and Khan are themselves conscious. But the standard view isn’t that the chat window itself is conscious but that the underlying thing that produces the text is conscious.
Now, I don’t think the argument “LLMs produce language so they must have minds,” is knockdown. But it’s at least suggestive. If we went to an alien planet and encountered weird aliens with Silicon brains, we should have some uncertainty about whether they’re conscious. But if they start talking to us, we should think it’s more likely that they’re conscious. This isn’t proof on its own, and it’s mitigated somewhat by the fact that we specifically engineered AIs to talk, rather than language emerging naturally from other processes. But it’s not nothing, surely.
The fact that LLMs resemble autocomplete doesn’t tell us much about whether they have experience. Insofar as the computational processes going on underneath are of sufficient complexity, so that many leading theories imply they could give rise to consciousness, then that’s enough that we should think they might be conscious. If there was a complete simulation of the human brain scaled up in complexity—and this was used to output tokens—that latter fact wouldn’t give us much reason to deny it was conscious.
Next, Chiang notes that to produce a final output, the program is run dozens of times. Each time it’s run, it simply produces a single token. But what’s this supposed to imply? If an animal made one movement every second, so that computation of its brain only produced a single behavioral output, would this be reason to deny that it was conscious?
It seems that Chiang is attacking a very weak argument. He notes, rightly, that the mere presence of apparent conversation doesn’t suffice to establish digital consciousness. But that’s not why leading philosophers think AI might be conscious soon. Rather, they think this based on the kinds of functions that the AI performs to produce this text.
Chiang compares believing in LLM consciousness to thinking Microsoft Word is conscious. But this is a silly comparison. Microsoft Word doesn’t produce complex, rich, autonomous text, and it doesn’t have any features that any of the main theories of consciousness imply give rise to consciousness. No global workspace, no recurrent processing, no higher-order processing—nothing. This is like comparing a wall of neurons that activate in response to light to a human brain.
LLMs are different from Word with regard to the functions they perform. A random wall of neurons and a human brain are made out of the same stuff. One is conscious, the other isn’t. What differentiates them is the functions they perform—the pairing between inputs and outputs. Likewise, what differentiates LLMs from standard word-processors is the functions they perform.
Now, there is one pretty interesting argument in Chiang’s piece:
The neuroscientist Anil Seth has noted that no one claims that AlphaFold—the program developed by Google DeepMind to predict the folding of proteins—is conscious, even though its underlying architecture is in many ways similar to that of LLMs like ChatGPT and Claude.
I have three things to say in response:
Unlike AlphaFold, LLMs could soon exhibit internal architectural features that beget consciousness on leading theories.
It doesn’t actually strike me as that implausible that a suitably complex AlphaFold could be conscious. If AlphaFold has the internal features that give rise to consciousness on our leading theories at some future point, then it could well be conscious.
If you have two things with similar architecture, and one of them talks, you should think it’s more likely that one is conscious. As a point of comparison, the architecture of a bee’s brain is pretty similar to the architecture of a much simpler insect’s brain. But because bees exhibit more complex behavior—learning, tool use, etc—we think it’s likelier that they’re conscious than the simple arthropods with ~10,000 neurons.
For responses to Seth’s other arguments see here, here, and here. My basic takeaway from this discussion is that Seth’s arguments are interesting and not totally crazy, but nowhere near strong enough to establish any high degree of confidence in the non-existence of LLM consciousness.
Next, Chiang asserts that “without a body, a computer program could have no desires or emotions.” He gives literally no argument for the point about desires. Why does wanting one thing instead of another require a body? It’s standardly thought that human wanting depends on our brain, not our body.
The closest he gets to explaining why you need a body for emotions is by asserting “Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body.” But why think this? As bats can fly without feathers, why not think LLMs could, in principle, have emotions even without hormones.
Next Chiang expresses concern about LLMs engaged in moral reasoning. He claims that to reason well about morality requires having emotions. But insofar as LLMs can mirror human discussion of morality, their outputs can resemble those of humans with reflective emotions. In practice, LLMs are much better at moral philosophy than most people, who generally think ridiculous things about morality for no reason.
Chiang worries that consulting with AI on moral tasks will lead to our moral reasoning atrophying. He instead suggests one consult philosophers and people they know on moral dilemmas. But even if you deny that AI is doing True Moral Reasoning TM, why would consulting LLMs on moral questions lead to greater moral atrophy than consulting people? In both cases, we are getting advice from someone else, and using our best judgment to weigh it. The metaphysics is irrelevant.
Chiang’s theory makes empirical predictions. If it was right, then when you ask LLMs about moral dilemmas, they should give bad answers. In practice, this is just wrong—compare how reasonably Claude talks about meat-eating to how unreasonably most people do.
I find the suggestion that consulting AI on moral topics isn’t useful particularly bizarre. His view is that there’s no major upside to having a smart and reflective entity on hand whenever you have a moral dilemma, who has read the entire internet? Really? Often there aren’t people who could sensibly weigh in on a moral dilemma.
Suppose, for example, you want to learn about why some people don’t eat meat or are strong Longtermists. You’re not interested in doing a super deep dive—you just want to know basically. LLMs are good for this, and can give the standard responses to your objections, without you needing to read over a bunch of philosophy books.
Even if you deny that AIs are doing genuine moral reasoning, it’s useful to have something that produces outputs that resemble moral reasoning. I don’t always care whether my smart moral advisor really understands morality, or is just like the imagined character in Searle’s Chinese room.
Here is the big picture about Chiang’s piece: he implies that he’s well above 99.9% confident that current AIs aren’t conscious and broadly similar AIs won’t be either. But he never once discusses the reasons leading philosophers think LLMs might be conscious soon. He never mentions any leading theory of consciousness, nor the implications they might have for LLM consciousness. All he has are hackneyed analogies that dissolve in the face of serious attempts to ascertain under what conditions LLMs might be conscious. Ted Chiang is a brilliant guy—he should go back to writing brilliant science fiction rather than mediocre philosophy.
I think those reports tend to underrate the likelihood of consciousness. For example HOT theories make it relatively easy for AIs to be conscious now
Do you think there's any plausible mechanism by which LLMs or similar software could perceive time passing? That seems really implausible to me, and then I think it's analytic that conscious experience of any sort requires temporal experience. This isn't Chiang's argument of course.
I thought his initial argument was supposed to be a gesture toward the "roleplaying account" of LLMs (which GPT reminds me is due to McShanahan and has been discussed by Chalmers). According to GPT:
The basic distinction is:
The LLM is analogous to the actor.
The apparent conversational individual—“the Assistant,” Claude, Sydney, or whatever persona appears in the dialogue—is analogous to the character being portrayed.
Statements such as “I believe…,” “I remember…,” or “I want…” primarily describe the simulated character, not necessarily the underlying model.
On this picture, asking whether ChatGPT genuinely believes something may be like asking whether Laurence Olivier genuinely wants revenge because Hamlet says he does. The utterance is intelligible as part of the role, without implying that the actor possesses the character’s mental state. The attraction of the view is that it lets us use intentional language to interpret dialogue while resisting the inference that the underlying neural network has a stable self, beliefs, desires, or self-awareness.
There are several reasons behind the analogy:
The same model can instantiate radically different personas. It can speak as an assistant, a fictional villain, Socrates, or an anxious computer, depending on the prompt. That makes the model itself look less like one determinate personality and more like something capable of realizing many characters.
The apparent speaker is partly generated by the conversational context. The system prompt, conversation history, and user’s framing establish facts about what “I” refers to and what personality the speaker is supposed to exhibit.
The character can possess properties the model plainly lacks. A chatbot may say “I grew up in Paris” or “I am afraid of death.” Those claims can be true within the enacted persona while being false—or not even straightforwardly applicable—to the underlying computational system.
Chalmers describes the view almost exactly in your terms: the Assistant is treated as a fictional character simulated by the model, with ChatGPT playing the Assistant as Olivier plays Hamlet. But he then questions whether pretense is the only possible interpretation. His alternative is that the model might sometimes realize or constitute the conversational persona, rather than merely pretending to be it—rather as a person may genuinely occupy a social role instead of theatrically impersonating someone.
So there are really two separable claims:
Ontological distinction: the conversational persona is not simply identical to the bare LLM.
Fictionality claim: the persona is merely a fictional character rather than a real, transient agent constituted by the larger model-plus-context system.
The actor analogy strongly supports the first claim. It does not by itself establish the second. That is where the interesting philosophical dispute lies.