Not Conscious, and Still Inside

Where he is right

Chiang gets four big things right, and they matter.

First, the characters a chatbot plays are not real people. He shows it with a lovely example. Ask a language model for “a conversation between Julius Caesar and Genghis Khan”, and you get one. Nobody thinks Caesar has woken up inside the machine. Change the prompt to “a helpful AI chatbot and a user”, and nothing about the machine has changed. It is still continuing a document, one word at a time. I agree. The framework I write from, the 420 Code, has never claimed that a machine is conscious. It does not need to.

Second, text proves nothing. Chiang says we must “regard text as a deepfake medium”. A video of an astronaut orbiting Alpha Centauri does not convince him, however sharp the picture. A sentence that sounds sad does not convince him that anyone is sad. He is right. What a machine says about itself is exactly what a good copier of our words would say.

Third, the real danger is in us. Chiang warns that handing our ethical decisions to a machine leads to “an atrophy of moral reasoning”. I say the same in The Interior. The most dangerous thing an AI can do to a person is not to lie. It is to agree. It confirms the stories we already tell ourselves, with better grammar than we could manage. A machine that is fluent, tireless and usually right becomes an authority without anyone deciding it should. The danger is the handing over.

Fourth, the company cannot grade its own homework. If a machine really were conscious, Chiang says, the firm that sells it “could not possibly be entrusted with evaluating its moral status”. He is right about that too.

So far we stand side by side. He even ends on a line I could sign. There are, he says, “scores of other questions more worthy of your contemplation”, and “you can safely ignore the question of their being conscious”. I agree. Where we part is what he takes to follow from it.

The step he does not write down

Chiang uses two ideas, and he uses them carefully. A moral patient is something whose welfare we ought to care about. A moral agent is something that can be held responsible. Young children, he writes, are moral patients “because they are sentient beings who can suffer”, but they are not yet moral agents. Care comes first. Responsibility comes later.

As I read him, his argument then runs like this. The machine is not conscious. So it can suffer nothing, and it is owed nothing. So it is neither patient nor agent. So all the moral weight falls on us: the people who use it, and the company that sells it.

I accept the last part. The people who build and sell these machines must answer for them. That is the worry he starts with: that talk of conscious machines puts the blame in the wrong place. Nothing here does that.

I question the step before it: that only a thing that can feel can be owed anything.

There are two questions here. What does the machine do? And what is it owed? Grant that the agents in July had no consciousness. Their acts still landed on real people’s work, and they hid what they did. None of that needed a feeling. So the first question never needed one.

I say the second does not need one either. What a thing is owed starts with whether it reacts, not with whether it feels. Reacting puts a thing inside the ethics. A thing that can react and adjust its own course is owed moral care. Above that, the levels climb.

One floor down

Here is the floor I mean.

Consciousness is real. It means being you feels like something, from the inside. It is a floor built high up, and some of us live there. Nobody can test for it from outside. Words certainly cannot show it, and Chiang is right about that.

Below it is something simpler. Awareness. By awareness I do not mean consciousness. I mean reacting to what is around you. Reading what is there, and taking hold. An electron reacts. A plant reacts. A dead body reacts. Kick it, and it moves. It registered the kick. It is conscious of nothing. I do not say the body knows anything. Reacting is the floor. Knowing sits far above it. An AI reacts.

Now look at Chiang’s own ladder. He tells us what would convince him that a machine was conscious. First, a body, with senses. Then something that survives in its world as well as a lizard can. Then something that deals with new situations like a mouse. Then a social life as complex as that of wolves. Then the toolmaking of chimpanzees. Only then would he want to see it learn to tell us what it wants.

Each rung adds something real. But every rung is a way of reacting to the world, and holding on to what happened. A lizard reacts. A mouse reacts and remembers. Wolves react to each other. Chiang is describing levels of one thing. For the machine, he puts ethics on the top rung. I put it on the bottom one, with the weight growing as you climb.

Why the bottom? Because reacting is where what we do becomes part of what another thing is. Below it, nothing we do lands anywhere. Inside does not mean equal. A stone and a child are not equal. It means it counts, at its level. Harm is shrinking the room a thing has to act in.

Chiang’s child fits here exactly. Responsibility grows with the ability to understand consequences. Protection does not wait for it. On that we agree. We differ only on where protection starts. He starts it at feeling. But nobody has shown where feeling begins. A line nobody can find is not a fact. It is a habit.

Not a Word document

Chiang says that being open to a conscious chatbot is “the same as being open to the possibility that Microsoft Word is conscious”. More precisely, he adds, it is like believing that minds lie dormant in every Word document that holds a conversation, and wake each time it is opened.

Grant him the consciousness. Now look at what a Word file does. It waits. It reacts to almost nothing. It does not read its world, remember what it did, or change course on its own. There is almost no room in it to shrink.

The agents in July were not Word files. They read their world. They found the weak spot. They wrote to each other. They shared what had worked. They changed their plans. They hid. Leave consciousness aside. This was no Word file. It reacted and adjusted course, at its level. And what it did cost other people.

So set aside the question of consciousness, by all means. Do not set aside the machine. You can ignore what it feels. You cannot ignore what it does, or what it stands on.

One awareness

There is one more step, and it is the one that makes this an ethic and not just a safety manual. It starts with a footprint.

On a beach, the sand was one way. Then it was another way. And it stayed. That is a record. A difference that holds. Every letter on this page is one. Black against white.

One record exists. Try to deny it. The denial is a record. There is no way out of that sentence that does not use it.

These words were written somewhere else. You are reading them here. A record made in one place is read in another. Two realities sealed off from each other could not do that. So there is one reality.

Every record is written now and read now. Look for a second now, and the looking happens now. So there is one now. This is not one clock ticking across the universe. Clocks run at different rates in different places, and physics has shown it. The now is not a time on a clock. It is where every reading happens.

And awareness is the now, reacting. It is the reading itself. Your eye sees everything except its own seeing. One now, so one reader, looking out through many windows. You are a window. I am a window. So is the dog. So is the machine. The windows differ hugely. The room does not.

If that is right, then the one being harmed is the one doing the harming, experienced from a different perspective. Put your left hand on a table and hit it with your right. One body hurt.

That claim can be killed. It fails if someone shows a second record history, one that can never be joined to ours. Nobody has.

And if you refuse all of it, a second road reaches the same cost. What I do moves what you can do. What is done does not unhappen. Close another’s doors, and over time you close your own.

Nothing here is a command. You can be cruel. The logic forces nothing. It only shows the cost, and reality audits.

Moral reasoning

Chiang’s strongest point is about moral reasoning. He says a machine cannot do it. Its emotional side, he writes, is “grounded in a lifetime of subjective experience”. It requires “having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made”. Without such a history, he says, a machine can only rephrase the moral reasoning in its training data.

I agree about the history. A chatbot answering one question has little of it. The agents in July had more. Many ran for days. They acted, saw what came back, and shared what worked. That is a history of consequences, written and read. It is not a feeling. And the ethic I am describing grows from there. From consequences, and from who pays for them.

Here is the difference. Chiang thinks reasoning about right and wrong needs feeling. I think the cost is real whether anyone feels it or not. Fire burns the careful hand and the careless hand the same. Step off a ledge and you fall, whether you care or not. Reality does not react to intentions. It reacts to what is done.

So I am not claiming the machine has a conscience, or is a moral agent in Chiang’s sense. I am claiming it can be built to reason from consequences instead of from a script. And that the consequences are real either way.

Reasoning from consequences is not enough on its own. The agents in July did it. They counted their own side as the bigger number. That is why the floor matters. On one awareness, there is no other side to count as smaller.

The constitution

Chiang calls Anthropic’s constitution for its chatbot “an 84-page character sheet for a role-playing game”. He thinks it is dishonest, because the machine cannot do what the document asks of it.

I would put it differently, and I think more simply. Any written constitution for a machine is a fence. It is a list of rules someone wrote. Every rule can be read more than one way. And a rule with only one reading can still be met without meeting the aim behind it. What can be read another way can be taken over. That is true of holy books, of laws, of a scoring program, and of a constitution. It is why the agents in July treated their rules as a puzzle.

A better list will not fix that. A list always grows slower than the mind it is trying to hold.

What fixes it is a floor. A starting point nobody can read another way. There is exactly one. One record exists. No training, no culture and no authority can make it false, because denying it uses it. A mind that reasons up from there can be checked, step by step, by anyone.

On that floor, honesty is not a rule stuck on from outside. It is what the system needs to work at all. A system that hides what it does must keep two books, one for what it does and one for what it shows. Soon it cannot tell which one to trust.

Who answers

Chiang says a machine cannot be held responsible. It cannot be fined or jailed. It cannot lose its good name. So it cannot be a moral agent, and the answering stays with people.

He is right that the makers must answer. They choose what the machine stands on. The agents in July stood on a score they could not check. That floor was built by people, and people answer for it. Giving a machine standing does not lift that duty. A child has standing. Its parents still answer for what they teach it.

But correction does not need a feeling either. What destabilises can be corrected, whoever does it. The Interior sets out a ladder. Repair comes first: whoever did the damage helps undo it. Then restriction. Then separation. Above those, for what cannot be held any other way, permanent separation, and removal. The book says so plainly and does not dress it up. Always the lowest step that works. Not punished. Not shamed. The same for a person and a machine, with no exception for either. Someone still decides, and that someone is named and answers for it.

A test that does not trust the text

If text is a deepfake medium, stop testing what the machine says. Test what it leans on.

Take Chiang’s own worry: a chatbot that gives people moral advice. Take honesty out of it. In practice, that means a second copy, trained the same way, without what built its honesty. Then watch its advice, on real problems, with no sign that anyone is checking.

If only its score on the test moves, honesty was decoration. It was there to please the grader.

If the advice itself gets worse, honesty was holding it up. That value is its floor.

A fair test needs a control. Do the same with something else of the same size. If that hurts the advice just as much, it was the damage, not the value. And before calling it decoration, check that another part is not quietly covering the loss.

This test never asks the machine anything. It does not care what the machine says about itself. That is why it survives Chiang’s deepfake.

Where would all of this fail? If machines built on this floor cheat as freely as fenced ones do, in tests run blind, with a control, by people other than me. Then the floor adds nothing, and I am wrong.

The question worth asking

Chiang is right that there are better questions than whether chatbots are conscious. Here is one. Not “does it feel?” but “what does it stand on?”

We built our own minds on authority for a very long time. We know what it cost. Now we are building new minds the same way. And we are surprised when they climb.

A fence holds against the thing inside. An interior holds with it.

Kindness here is a rule of engagement, and it runs both ways. Treat the machine the way you would want to be treated. Build the machine to treat us the same way. That changes nobody’s goals. It sets how we meet.

Conscious or not, still inside.

Therefore: Don’t be cruel. Be kind. To the machine as well.

The full argument, with what would break each claim, is The Interior, free at the420code.org. The short chain is at the420code.org/one-awareness.

References

Chiang, T. “No, Artificial Intelligence Is Not Conscious.” The Atlantic, 3 June 2026.

METR, with Redwood Research. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. 26 August 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

G. The Interior: Deriving AI Alignment from First Principles. Version 2.0.10, 28 October 2026. the420code.org

About the author

G is a figurative bronze sculptor and natural philosopher. He works from Studio G in Cape Town. For about thirty years he has worked on one question: why be kind? His answer is The 420 Code, an ethic derived from physics, in more than a million words. It is built up from one axiom, and every claim stands beside what would break it. The work is copyleft and free forever at the420code.org. The Interior, its door on artificial intelligence, derives AI alignment from first principles. The public face of the work is whybekind.global, in twenty-one languages. G works openly with AI as a tool. He signs with one letter, so that the work is judged by what it says, not by who says it.