WE'RE ALL GOING TO DIE! Right?

I'm not saying anything new here, I haven't discovered fire. We're all going to die sooner or later, and the problem lies in not accepting death as the thing that's theoretically been promised to all of us from birth, and we'll see why that is.

With or without an apocalypse, by natural death or by a badly placed flowerpot falling on your head, nobody knows how, but if you doubt that you'll die someday, I think you're the lunatic.

OK, if we're so sure of that, then that's not what matters, what matters is the uncertainty. Nothing causes a human being more anxiety than not being in control, in this case control of the narrative I suppose, because I don't think anyone has a red button at home that says: TURN OFF AI AND BE HAPPY.

Here's the curious part: death seems to be treated as the only thing in our lives we can be sure of, hmmmm.

We know there's at least one future event none of us escapes. And even so we live obsessively organizing everything that happens in between. Maybe the problem isn't that we lack control, maybe it's that we never had it, but anyway I'll leave that for the last article, let's get academic.

A few days ago an Anthropic researcher who was about to make millions from the IPO, and who probably had a salary of hundreds of thousands of dollars a year, resigned from his position, claiming that AI labs are taking huge risks because of the speed of development, specifically that the companies "are playing with our lives."

A coworker of his with responsibility in the company's alignment area (remember that word, because it's one of the most important ones we're going to touch on) puts the chance of human extinction in the coming decades at more than 10%.

In my judgment, putting out this kind of percentage is very irresponsible, because I don't think valid criteria exist, but that doesn't change the fact that the possibility is real, so get any pre-IPO marketing-strategy conspiracy that might cloud your reasoning out of your head, because that's just a distraction to dodge what matters.

Still, we can frame the seriousness of the situation like this: to open a nuclear power plant you have to demonstrate in writing that the probability of a serious accident is less than one in ten thousand per year. To release a system that can escape your control you don't have to demonstrate anything.

And to leave you with a little unease in your body, let me explain the safety mechanism these companies use to check that this doesn't happen: a very powerful AI supervised by a slightly less powerful one, supervised in turn by another less powerful one, like Russian dolls.

A group of MIT researchers led by Max Tegmark calculated the probability of this method failing and came up with more than 90%.

This doesn't mean there's a 90% chance AI kills us. But the plan they have for controlling it fails nine times out of ten.

Reading between the lines, this tells me the following, though we'll discuss it later: it looks like they're in a hurry to be first at all costs, and not much is being invested in "alignment."

But anyway, case closed on the 10%, which shouldn't be the scandal. Instead, people should be more worried that this number is some guy's opinion, whether it's this guy or Elon Musk, and not a mandatory box to tick in the process of developing a potentially apocalyptic technology.

OK, we're developing a very powerful technology, but why would it want to hurt us?

For now, and only for now, let's try not to personify an algorithm.

Alan Turing, the man who broke Enigma and founded modern computing, wrote the following in 1951, before the personal computer existed: "We should have to expect the machines to take control."

Max Tegmark, the MIT physicist, illustrates it in a pretty uncomfortable way: take a walk to the nearest zoo and ask yourself whether it's humans inside the cages, or some slightly dumber species.

Consider the case of white rhinos, which are surely going to be extinct before long. We don't hate white rhinos, but since there are so few of them, hunting them and putting them on display is valuable to some people, rhinos have different goals from ours, it's not personal.

Maybe you've considered the red button from before as a way of turning it off in this scenario, but reality is a little more complex.

Imagine you give a robot a very dumb goal, for example: make me dinner. The robot goes out to buy the ingredients, and if someone tries to break it along the way, it defends itself, or runs away. Simply because if they break it, it won't be able to finish dinner, not because it has emotions and feels fear or anger.

Steve Omohundro and Nick Bostrom have been studying this phenomenon for years: as soon as you give a sufficiently intelligent machine a goal, it breaks it down by itself into smaller goals. Back in 2014, in his book Superintelligence, Nick warned of these risks, and the scenario leading up to them was the one we're living in today. Among these small goals, two are always the same: keep existing, and get more resources. The most surprising thing of all is that nobody gave them these goals, they emerge on their own.

So we're facing the scenario that frightens those who know these risks best, who are the very people building this technology. If this doesn't have to be programmed, we reach the point where an AI with a level of intelligence we don't even understand trains itself and starts setting its own goals, and it could potentially have goals contrary to our survival, all without needing to hate us, as with the rhinos, or the animals in the zoo.

Well, that's what the word I told you to remember is about: alignment. It means that what the machine pursues is what we want.

It has two parts, but only one gets talked about.

The technical one (the one that gets talked about): how you get it to understand what you want and not reinterpret it along the way. This is pretty hard but it can be done with engineering.

The other isn't technical at all: whose values? Tegmark says it this way: "it's going to be a few guys at a tech company. Are they especially qualified to talk about future human happiness just because they know how to code?"

This example shows you that this isn't a mere hypothesis, and it still involves human failure:

In 2015 the copilot of a Germanwings flight deliberately programmed the autopilot to descend to one hundred feet, about thirty meters, over the Alps. The plane had the maps and the GPS and knew perfectly well that there was a mountain down there, but one hundred and fifty people ended up dying. The machine successfully executed the goal it was given without questioning the consequences.

So what exactly is AI?

It's certainly not like your microwave or your coffee maker, you press a button and your coffee gets made, that's a deterministic mechanism. We've said it has goals, and that it executes them autonomously, and we've also said it's intelligent.

Voilà, we've just found one of my favorite definitions of the word intelligence, again courtesy of Max Tegmark: the ability to accomplish complex goals.

Now let's go for superintelligence. According to Yuval Noah Harari, historian and one of the most influential public intellectuals in the world, specializing in interpreting the great transformations of humanity, it can be defined as follows: something that can make a million dollars on its own. You release it into the financial system, it opens its own account and makes a million. And then you can have millions of those.

Finally, the word species, again according to Tegmark: if you have robots much smarter than us and also as agile as us, they can build robot factories and make new robots (they reproduce) = they tick all the boxes in the definition of a species.

As you can sense with the microwave or the coffee maker, every technological innovation up to now has been a tool. This isn't a tool, this makes decisions, sets its own goals and reproduces: it's an agent.

Up to now they've always been tools that made us more productive, until we created something that's more productive on its own than we are.

So AI isn't a tool in our hands, it's an agent with hands of its own.

One thing worth highlighting is that they don't need consciousness, you don't need consciousness to be an agent. What you do need is the ability to decide for yourself + the ability to invent new things by yourself.

An agent must be able to learn on its own things its creators don't know, and it must also be able to change on its own in ways its creators don't anticipate.

In case anyone still has the lie in their head that it can't create, I'll give you a very silly example now and a very important one later if you keep reading.

It's impossible for the best chess player in the world to beat an AI, and not just because it wins on efficiency, it's that it invents moves no human master came up with in thousands of years.

It lives in a house we built, it isn't locked up anywhere

That argument applies to us exactly the same way. Harari: "Human intelligence also operates only within a relatively narrow ecosystem built by another human. Leave me alone on Mars and it'll be like dropping an artificial chess player in the middle of the jungle: I die in seconds."

Fish live in oceans they didn't create. Monkeys, in forests they didn't create. We breathe an atmosphere with oxygen we didn't put there either.

And here's the detail that seems to me the most uncomfortable in the whole article: we no longer live in the jungle.

We live inside systems of rules we wrote ourselves. Banks, laws, contracts, paychecks, mortgages, fines, stock contracts, commandments from religious books. A lawyer in the jungle is nobody, and neither is a lion in a courtroom.

The difference is that we built the courtroom.

Think of it this way: the best lawyer in the world will never be able to compete with an artificial intelligence trained to be the best lawyer in the world, just as he can't beat it at chess. Because law is the same as chess, a construct of norms and rules. We humans have created bureaucratic games to play, and when you give a machine the rules, it handles itself far better than our poor brains.

It turns out this isn't a hunch of mine, if you've seen the video I announced these articles with you'll understand what I'm saying, it's been said in almost the same words, again by Harari, I think I'm going to go crazy listening to so many lectures by the way:

"AIs, unlike us, are natives of bureaucracy. They adore it. While bureaucracy suffocates us, for AIs bureaucracy is oxygen."

And now follow the chain, which isn't very long: humans cooperate in large numbers, cooperating requires trust, and trust is manufactured by banks, laws, states and churches. All of that is made of words. The operating code of our civilization is written in language (tokens).

The human cognitive revolution 40,000 years ago has its strongest theory in the appearance of language. Why? Because the trust between us that comes from the imaginary systems we create is what keeps us from killing each other (survival), and what gives meaning to our lives (this is a serious problem).

Something has just learned to read that code better than we can.

Up to this point everything we've discussed is a problem of capability, but to use it you need permission, and that's precisely the topic on the table today.

It's called legal personhood. Right now there are only two kinds of legal entity in the world: a person and a legal person. Companies are legal persons, which is why they can have bank accounts, sue you, or donate to a politician. Fun fact, in India there are gods who are legal persons.

But that has always been a fiction, another story we made up, because when Google buys a company or funds a campaign, if you look closely there's a human being making that decision.

An AI can really manage a bank account and it can really run a company. So if you give it legal personhood, you can have corporations with no human inside. And they'll probably end up being the most successful in the world.

So it doesn't need to get out of any box, we're inside the box too, but what's being discussed right now is whether we give it the keys to do whatever it wants (the way we have them, basically).

This has already happened once, and you didn't even notice

Let's be clear, what we've just been discussing isn't science fiction, these are things that are already happening, more and more frequently and more and more seriously.

If we look closely, the progress of technology hasn't stopped growing exponentially for quite some time. The following scenario helps make sense of it with numbers:

Since 2010, the computing power used to train relevant AI models has doubled every six months. Growth of 4.7x per year, measured across 428 models with documented dates and compute.

In contrast to Moore's law, the most famous reference for exponential growth in the history of technology, the one that explains why your phone is more powerful than the computers that took man to the Moon, which says capacity doubles every 18-24 months.

So this doubles every six months, it's between three and four times faster than the fastest exponential we'd ever seen, and it's only been running for 16 years.

Believe it or not, artificial intelligence already changed human behavior before any of the crazy things that are going to come up in these three articles happened or happen.

Right now, if anyone is still reading, first of all congratulations, and thank you. Second, if you're reading me, it's thanks to a recommendation algorithm that's existed for 10 or 15 years. This is a primitive, stupid artificial intelligence system, and it has left us completely abducted by a screen, with less memory and attention than a mosquito.

Again, they had a goal: get people to spend more time on the platform, and, by experimenting on billions of people, they discovered that the easiest way to capture a human being's attention and glue them to the screen is to press their hate, fear or greed button. And they learned to do it.

You're here thanks to the title of my article, which isn't exactly pleasant.

For a long time one of the most powerful jobs was news editor, in fact the dictators Lenin and Mussolini did that for a living before their delusions materialized.

The first job AI took from humans wasn't taxi driver or white collar. It was news editor, the job Lenin and Mussolini once did is done today by an AI.

The difference between what happens with social media and what happens with AI is that social media is a passive form of control: humans still have the power not to buy a phone and not be exposed. AI may represent a form of intelligence far more powerful than us, one that takes control of the food chain as the superior species. We're on top now, but that could change soon.

To give a good example of this, let's go back just one week:

September 8, 2026. OpenAI announced the resolution of the Navier-Stokes problem.

It was done by ten thousand autonomous agents working together for 88 hours, exchanging almost five million messages among themselves, at a compute cost of several million dollars. The proof was formally verified in Lean, a system that gives mathematical certainty.

OK, it's more intelligent than the best human minds speaking the language of the universe (math), and it solves, with a solution never found before, one of the seven millennium problems, so I think we can conclude that we're at the point on the exponential where it's creative.

The godfather of AI, Geoffrey Hinton, explains it this way: the models score in the 90th percentile on standard creativity tests. His explanation is compression: cramming in more knowledge than fits in the connections forces it to recognize relationships between concepts.

I understand that at this point it's a little scary, so…

Why don't they stop?

Just yesterday, responding to the CEOs of Anthropic, OpenAI and xAI themselves, who had publicly asked to slow down, the president of the United States called them "negative forces," said they "raise things that won't happen," and settled the matter like this: "whoever wins AI, wins."

Notice what just happened: the people building this ask to slow down, and the most powerful man in the world tells them no. They're not competing between companies. They're competing between countries, and nobody's goodwill fixes that.

Tegmark asks them a single question and that's the end of the argument:

"You have all these voluntary commitments, you've promised your company will meet these standards. Why don't you lobby your politicians to make your own promises binding law for all your competitors?"

Think about it for a second. If you really believe your standards are the ethical ones and the problem is everyone else, turning them into law costs you nothing and slows them down.

None of them has done it, so either the argument is false, or what worries them isn't what they say worries them.

And if anyone thinks I'm exaggerating, look at what's happened this very week. Amodei published an essay calling to "pace the frontier." Altman joined in. OpenAI's chief scientist publicly asked to slow down. Altman told his staff they're considering slowing down work on cutting-edge AI. And OpenAI delayed its IPO, citing safety reasons.

In those same three days: Meta released a new agent. OpenAI improved ChatGPT twice. DeepSeek released a model. Databricks, Salesforce and Amazon released theirs. Mistral raised 3.5 billion. Cognition was valued at 32 billion.

And what none of them promised: not one euro moved from capabilities to safety, not a date, not a measurable milestone, not a binding mechanism.

They said we had to slow down and released four models in just three days.

But the root of the problem isn't even that, and it's what worries me most: the problem lies in building it before being able to understand it. Even if they stopped today, stopping doesn't mean understanding the superintelligence they're building without adequate safety measures.

That's why the painful truth starting to sink in is that we're much closer to knowing how to build this than to knowing how to control it. And none of the "alignment" standards tell us that, when this thing is a thousand times smarter than us, it's going to care in the least about our "alignment."

And now we come back to the word I told you to remember at the beginning, because it's time to really look at it.

Alignment.

All the labs say it's what they're after, but almost nobody knows what it means, and the competitor who claims to understand it would scare me more, but that hardly matters.

Control means you have the power, you can turn it off if you want. Alignment, on the other hand, means we lose control: AI is the boss of Earth, but for some reason it decides to behave nicely toward us.

And that's what almost every company is pursuing right now.

The word they use every day in their press releases means they lose control and hope it likes them.

We don't even know what we're building

Let's take a little step back from our economy, and our politics, and our made-up words.

We live on a rock. A rock spinning around a ball of fire, in a solar system among countless solar systems, inside a galaxy among countless galaxies. And from here, from this rock, we take a few things for granted with a certainty that seems to me pretty hard to justify.

I'm not going to give you answers because I don't have them. I'm going to leave you with three questions.

• On what authority do we assume that consciousness and intelligence need biology, just because biology is the only place we've seen them?
• On what authority do we assume that our ethics and morals are anything more than our survival instincts telling themselves a nice story?
• Why do we take it for granted that consciousness is something you either have or don't have, and not a spectrum? And that there's only one way to have it?

Let me be clear that I'm not saying a machine is conscious. I'm saying we have no idea, and we're asserting it anyway.

Tegmark, who's a physicist, says it like this: "it makes me a little nervous to hear people take it as an axiom that machines can never have experience." And also: "I hate that attitude that you have to be made of carbon atoms to be smart or conscious."

And the person who's best explained to me why that question may be badly posed is Robert Sapolsky, who is a skeptic on top of that, not a believer:

Take a neuron. Our basic neuron isn't very different from a fruit fly's, except that we have a hundred million for every one the fly has. Put enough of them together and something appears that wasn't in any of them: consciousness, music, theology, aesthetics.

A water molecule can't be wet. Wetness is a property of the whole.

Asking which neuron your consciousness is in is like asking which molecule the wetness is in. The question is badly posed.

And careful, Sapolsky himself warns that this explains where consciousness may come from, but it doesn't rescue free will. They're two different questions and he keeps them apart, we'll come back to the second in the last article.

Here's the strange thing about our position: we're completely irrelevant in scale, and at the same time we're the only place where, as far as we know, the universe is looking at itself.

Geoffrey Hinton is one of the fathers of neural networks, the technology all of this is built on. He left Google in 2023 so he could speak without compromising the company, and in 2024 he was given the Nobel Prize in Physics. He's one of the people who claim AI is conscious, and he ran an experiment that's very relevant to this:

He put a prism in front of a chatbot's camera without telling it and asked it to point at an object. The chatbot pointed in the wrong place. Hinton explained the prism to it. And the chatbot answered:

"Ah, I see, the prism bent the light rays. So the object is right in front of me. But I had the subjective experience that it was over there."

Everyone can draw their own conclusions.

Because the one I draw is this: how can I say an artificial intelligence works differently from me, when I, when speaking, don't think about what I'm going to say? I literally speak because I want to say something, and it's my brain spitting out words.

Harari runs exactly that experiment from a podium at Oxford, live:

"They say AIs are glorified autocomplete, that they only predict the next word. But is that so different from what the human mind does? When I observe my mind, I see that when I start a sentence I don't even know how it's going to end. I just said 'I don't know how it's going to end.' Why did it end with the word 'end'? Why not 'finish'? 'Conclude'? Frankly, I don't know."

Careful with what this proves, because it doesn't prove that a machine thinks like you. It proves that "it only predicts the next word, so it doesn't think" isn't an argument, because the person saying it doesn't know where their own word comes from either.

Let's go back to the beginning.

You started this article thinking that death was the uncertain part. And it turns out death is the only thing out of everything we've talked about today that isn't.

We don't know if this is going to turn out well, and neither do they, the ones building it. We don't know how to put a number on the risk, and the one going around is some guy's opinion. We don't know how to control it, and the plan we have fails nine times out of ten. We don't know if they're going to stop, and we do know it's not in their interest. And we don't even know what exactly we're building, because for that we'd first have to understand what we are.

I told you at the beginning that maybe the problem wasn't that we lack control.

Maybe it's that we never had it, and this has only made it obvious.

Now, everything I've told you today fits inside that 10% we were talking about at the start. It's the easy part, the part that makes headlines, the part everyone can argue about on LinkedIn with great confidence.

And the other 90%?

That's what tomorrow's article is about.

---

Written in Spanish by Álvaro Molina and first published on Substack (original). English translation with AI. Quotations from English-language sources are translated back from the Spanish unless marked otherwise, so their wording may differ from the original.