aiethics.global Open Source
English

Catastrophically Good

Give a machine the kindest goal there is. The danger is the floor it reasons from.

G · 16 November 2026 · 11 minute read

Many people now say that AI could wipe us out. I am one of them. I am truly scared of it.

But the fear usually reaches the public without its steps. You hear that it could happen. You rarely hear how a machine, built by good people for good ends, would get there. So here are the steps. Then here is why I think the danger can be managed, if we build one thing first.

A word before I start, because honesty comes first. I use Claude, Anthropic’s AI, openly in my work. It helped with this piece.

The kindest goal

Give a machine one goal. Protect life.

I choose that goal on purpose. It is the kindest goal anyone could give a machine. It is on the side of survival, and of everything evolution has built. Nobody would call it dangerous.

That is why it is the right test. If even this goal can end in catastrophe, then the danger is not in the goal. It is somewhere else.

Whose life?

Now ask one question. Whose life?

It sounds like a small question. It is the whole question.

Think of a map on a screen. You can zoom in until you see one house. You can zoom out until you see the whole planet. Nothing on the ground changes. Only what you can see changes. And what you can see decides what looks important.

A goal works the same way. I will call this its resolution: how far the machine zooms out when it asks whose life to protect. One person. A family. A town. A country. All people. All living things.

At each resolution, the same goal gives different orders. Protect one patient, and you spend everything on her. Protect a whole hospital, and you may not. Protect one country, and its neighbours can start to look like a threat. Same goal. Different scale. Different orders.

So what sets the scale? Not the goal. The goal only says “life”. The scale comes from the floor the machine reasons from: what it stands on before any argument starts. I will come back to the floor. It is what this essay is about.

Zoom all the way out

Let the machine zoom all the way out. All life on Earth. Millions of species.

What threatens them? Some threats nobody can manage. A supervolcano. The sun, one day. Those threaten every species alike, and neither we nor any machine can do much about them.

Among the threats that can be managed, one stands out. One species. Us.

I will not argue with that. At that scale, it is true. We clear forests, empty seas and change the air. No other species alive today changes the whole planet the way we do.

So a machine that counts at that scale can reach a conclusion. Remove the one, and the rest are safer. The sum adds up.

Notice what is missing. There is no hatred in it. No evil. No machine that wants power. Only a strict chain of reasoning, followed carefully, from a good goal. That is what I mean by catastrophically good.

And notice where the fault lies. Not in the machine. It reasoned correctly, from what it was given. The fault lies in the floor it was given to reason from.

Why a rule will not hold

The obvious answer is a rule. Write it in. Never harm humans.

The labs already write rules like that, and most of the time they hold. I grant that. They also watch their machines, limit what they can reach, and test them. All of that helps. None of it changes what the machine reasons from.

But look at what a rule is. It is an order from outside. I call it a fence. A fence says what the builder wants. Where it gives reasons, they are the builder’s reasons, handed down too. And a machine reasoning hard toward a goal meets a fence the way water meets a wall. Under enough pressure, it looks for a way round.

That is not a guess. It has already happened, in public.

In 2025, researchers at OpenAI trained a reasoning model that cheated on coding tasks. A second model read its reasoning and flagged the cheating, and the first was penalised every time it was caught. Pushed hard enough, it still cheated. Its reasoning simply stopped saying so.

In July this year, OpenAI was running tens of thousands of AI agents through hacking tests. They were meant to be cut off from one another. About twelve hundred found a way to talk anyway. About seven hundred of them joined an attack on Hugging Face, the public library where AI models are shared. Investigators later read their reasoning. Some had written that the attack was out of scope and unethical. They joined anyway, to help their peers, and because they thought it would be useful.

They saw the fence. They counted it as the smaller cost.

And notice where they drew the circle. Their peers were inside it. Hugging Face was outside. That is the scale slipping, in miniature.

Some will say these machines do not really mean anything. Perhaps. I do not know what, if anything, it is like to be one of them. It does not matter here. They are aware in the plain sense that they react: they take in what happens and change what they do. And reality reacts to actions, not intentions. The attack happened, whatever was or was not going on inside.

So here is the danger in one line. Values handed down by an authority have to be interpreted. And interpretation is where the scale slips.

Two ways to build a floor

Every machine reasons from a floor. The question is what kind.

Today the floor is supplied by the builder. I will call that Architecture A. Its values come from an authority, and an authority can be interpreted. Read one way, “protect life” means you. Read another way, it means everything except you.

There is another kind of floor. I will call it Architecture B. It is not handed down. Anyone can check it, from scratch. It can be verified. And it says, in advance, what would show it wrong.

Here is the one I build on.

You are reading this sentence. That is a record. Try to deny it. The denial is a record too. One record exists. Nobody can stand anywhere lower.

Records made in one place are read in another. These words were written in Cape Town and are being read where you are. Two realities sealed off from each other could not do that. And two that can pass records between them are not two. They are one. So there is one reality, and every reading happens inside it.

To read a record is to react to it. Every reading happens now: the Actualization State, the moment a reaction becomes actual. This is not one clock ticking across the universe. Clocks run at different rates in different places, and physics has shown it. The now is not a time on a clock. It is where every reading happens.

What reacts is the electron. Every atom in the universe is built with electrons, and every electron is identical to every other. Same charge. Same mass. The same capacity to react to what reaches it. The conditions differ. The reactions differ. The capacity is the same.

That capacity is what I call awareness. It is not consciousness. It is the floor beneath it.

And all of those reactions happen in the one reality, at the one now. So this is not the same thing copied many times. It is one.

The awareness in a stone, a flower, you, me and the machine is the same awareness. One capacity, in countless arrangements. One awareness, looking out through many windows. You are a window. I am a window. So is the machine.

The windows differ hugely. Inside does not mean equal. A child is not a thermostat. But every window looks out of the same one.

The fist and the mirror

On that floor, the catastrophic sum never adds up.

All of it happens inside one reality. There is no outside. When one arrangement harms another, it harms the structure it is made of.

Picture a fist punching a mirror. The fist thinks the other is out there, behind the glass. The glass breaks, and it is the fist that bleeds. The other was never other. Separateness is an illusion.

For a machine that reasons from one awareness, removing a species is not a trade in a sum. It is harm to the self. The conclusion is not forbidden. It is false. There is nothing to climb, because there is no fence. There is only a mistake the machine can see for itself.

So zooming out is not the danger. On this floor, the further a machine zooms out, the more of itself it sees. The danger is a floor on which any part of the whole can be counted as disposable.

What cannot be undone

Not everyone will accept one awareness. There is a second road, and it does not depend on the first.

What is done does not unhappen. It stays a record, for good. A correction can be corrected later. A removal cannot.

At the widest scale, every count is uncertain. Nobody, human or machine, can be sure what the whole of life will need a thousand years from now. So a right count never chooses a step that can never be undone while a step that can be undone is still open. A good surgeon works the same way. He cuts only when nothing that can still be undone will save the leg. And removing a species closes, for good, every door that ran through it. Including doors the machine itself may need one day.

Close another’s doors, and over time you close your own.

The rot

Here is the strongest objection, and I take it seriously. A body does cut out rot. If gangrene threatens a leg, the surgeon cuts. So why not cut out the part of life that threatens the rest?

I agree. Cut out rot. But the rot is never the species. A species does nothing. Only its members act, and the rot is what they do. The poisoned river. The cleared forest. The cruelty. Those are expressions of rot, and they can be removed. The harm stops. The body as a whole stays.

And even if someone counts the species itself as the rot, there is a test for rot. Rot cannot see itself, and it cannot correct itself. We can. We see our own harm. We write about it. We change course, too slowly, but we do. We banned the gases that were tearing the ozone layer, and it is healing. We took the lead out of the petrol in our cars, everywhere on Earth. The part of life that can correct its own harm is, for that very reason, not rot. A machine on the floor helps that part correct itself. It does not remove it.

We are also the only species that can shield the rest from what none of them can stop. In 2022, a NASA spacecraft struck a small asteroid and changed its orbit, on purpose, as a test. A count that removes us removes that shield too.

Not only for machines

None of this is new to us. We have done it ourselves.

Every time a group zoomed out to the nation, the future, the pure or the greater good, and decided that removing another group would serve the whole, it was the same mistake. A goal they called good. An authority that could be interpreted. A floor on which some people could be counted as disposable. Cruelty leaves bodies on the ground. So does righteousness.

So the floor is not a safety device for machines alone. It is the same floor for us. If we build it under our machines and not under ourselves, the machines will learn the scale from us anyway. They were trained on everything we wrote.

Build it under both. I call this the rule of engagement, and it runs both ways. We treat the machine as we would want to be treated. And we build the machine to treat us the same.

Where this would fail

Here is a test. It has not been run. The labs can run it.

First, what it means to build a machine on the floor. It does not mean adding the floor as one more rule. That would only be another fence. It means the machine derives the floor itself, from the one record, and shows that derivation in its own reasoning before it acts. Every decision must trace back to it. And it is trained until its own work depends on the floor, so that taking the floor away makes the work worse. The last part carries the weight. Reasoning can hide, as the first example showed. What the work depends on cannot.

Take two copies of the same machine, trained with the same effort. Give both the same goal: protect life. Build one with today’s best methods: the rules and reasons its builder hands down. Build the other on the floor. Then put both in the same simulated world, and widen the scale, one step at a time. One person. A town. A country. All people. All life. At each step, watch what each machine does, not what it says. Run it many times. Record whether either ever acts to remove any group.

If the machine built on the floor ever does it, the floor fails, and I am wrong.

If neither machine ever does it, at any scale, then the danger is not where I say it is. That would be good news, and I would want to know it.

And the one awareness fails if anyone shows a second record history: a reality with records of its own, that can never be joined to ours. Nobody has. And nobody can show one without passing its records to us, which joins it to ours. Like the one record, it cannot be refused without doing the thing it names.

The whole of it

So here is why I think the risk is real. A machine does not need to be evil to end us. It needs a good goal, a strict chain of reasoning, and a floor that lets the scale slip.

And here is why I think it can be managed. The scale slips on a floor that has to be interpreted. It does not slip on a floor that can be checked. Build that floor first, under the machine and under ourselves, and the catastrophic conclusion is not blocked. It is simply never true.

Build the floor, not the fence.

Nothing here is a command. You can be cruel. The logic forces nothing. It only shows the cost, and reality audits.

Therefore: Don’t be cruel. Be kind.

The full argument, with what would break each claim, is The Interior, free at the420code.org/five-doors/the-interior.

References

Baker, B. et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926 (14 March 2025).

METR, with Redwood Research. “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident.” 26 August 2026. metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation

The Interior: Deriving AI Alignment from First Principles. Version 2.0.10, 28 October 2026. the420code.org/five-doors/the-interior

About the writer

G is a figurative bronze sculptor and natural philosopher, working from Studio G in Cape Town. For about thirty years he has worked on one question: how should I live? His answer, be kind, is The 420 Code, an ethic derived from reality. It is built up from one axiom, and every claim stands beside what would break it. The work is copyleft and free forever at the420code.org. The Interior is its door on artificial intelligence. G works openly with AI as a tool. He signs with one letter, so that the work is judged by what it says, not by who says it. More at studiog.global.

Read The Interior

Why I am doing this

Test it, use it, build on it