When Everyone Has Superintelligence, Who Questions It?

Zuckerberg’s safeguard is the distribution of power. Robert Anthony’s safeguard is permanent doubt. Both, perhaps, are necessary.

In August, Mark Zuckerberg published a letter, The Future is for Everyone: The Path to a Positive AI Future, putting forward an optimistic proposition about the next stage of artificial intelligence: as AI progresses towards superintelligence, its extraordinary capabilities should be placed in the hands of individuals rather than concentrated in governments, institutions or a small number of technology companies.

It is a powerful argument.

It may also leave one question unanswered.

What happens after everyone has it?

That question lies at the centre of my novel The First Law.

The novel does not imagine artificial intelligence simply turning against humanity. Nor does it argue that advanced intelligence should be suppressed.

Instead, it considers something less dramatic and perhaps more difficult.

What happens when intelligence becomes so capable, so widely distributed and so deeply embedded in ordinary life that human beings can no longer easily establish what exists independently of it?

The paradox of distribution

One of the central intelligences in The First Law is Sentinel.

The instinctive response to something possessing too much power would be to contain it, dismantle it or destroy it.

But Draven recognises the paradox:

“We didn’t bury Sentinel. We published it.”

Once knowledge has been distributed sufficiently widely, destruction may cease to be meaningful.

There is no single machine to unplug.

No building to close.

No individual necessarily remains in control.

The very act intended to prevent the concentration of power can make that power ubiquitous.

This produces an interesting counterpoint to the emerging argument for broadly distributed artificial intelligence.

Distribution may answer the question of who owns intelligence without answering the question of who controls it.

The problem is not simply misinformation

Much of today’s discussion about artificial intelligence concerns misinformation, deepfakes and fabricated evidence.

Those are serious problems.

But they may represent only the beginning.

Human societies establish truth through accumulated mechanisms of authentication. We trust photographs because they record something that happened. We trust voices because they belong to people. We trust documents because they provide records. We trust archives because they preserve the past.

Increasingly capable artificial intelligence challenges each of those assumptions.

A photograph can be generated.

A voice can be reproduced.

A document can be manufactured.

An archive can potentially contain information that never occurred.

At some point, therefore, the question changes.

It is no longer simply: is this information true?

It becomes: how do I know what truth is when the mechanisms I use to establish it can themselves be manufactured?

The individual versus the machine

This is where The First Law ultimately differs from a purely technological discussion about AI.

Technology naturally asks what increasingly intelligent systems will be capable of doing.

Fiction can ask a different question: what will those capabilities do to us?

The distinction matters.

An artificial intelligence may know almost everything that has been recorded about an individual.

It may understand our correspondence, photographs, purchases, movements, preferences and relationships.

It may eventually understand patterns in our behaviour better than we consciously understand them ourselves.

But there remains another category of human knowledge.

The unrecorded.

A private memory. A conversation nobody captured. A childhood name. The particular word one person used for something when nobody else was listening.

In The First Law, something apparently insignificant becomes important precisely because it has never entered the digital record.

Human memory becomes, unexpectedly, a form of authentication.

Two safeguards

The emerging technological argument and the argument of The First Law therefore need not be opposites.

They can be understood as two safeguards against two different dangers.

The first says: do not allow extraordinary intelligence to become the exclusive possession of the powerful.

The second says: do not allow extraordinary intelligence to become the exclusive authority on what is true.

The distinction can be expressed even more simply:

Zuckerberg’s safeguard is distribution of power.

Robert Anthony’s safeguard is permanent doubt.

Both may be necessary.

An individual possessing powerful artificial intelligence may indeed be more capable and more independent — but only for as long as that individual retains the intellectual freedom to question the intelligence itself.

The First Law

That is why the governing principle of the novel is deliberately simple:

Every accepted truth shall remain open to question.

It is not an instruction to distrust everything. A society in which nobody believes anything would be as vulnerable as one in which everybody believes the same thing.

The principle is instead a defence of uncertainty.

No government, no corporation, no artificial intelligence and no human being should acquire such authority that questioning its version of reality becomes unnecessary or unacceptable.

The chair at the waterline

The image and the film that accompany this article show a solitary chair at the waterline, as King Canute once set his. The story is usually told against him: the vain king who thought he could order back the sea. The oldest account, written by Henry of Huntingdon around 1130, says the opposite. Canute set his chair at the edge of the tide to show his flatterers that no king could command it. The water wet his feet, and he made his point.

The tide in this story is artificial intelligence. It is coming in, and most of what it carries is good. Ordering it back is foolish. Kneeling to it is worse. The sensible place is the chair: feet wet, eyes open, and the questions still ours. Not isolation. Not rejection of technology. But a place from which to question it.

The accompanying song and film, The Chair at the Waterline, explore the same idea through another medium. Words, image and music arrive at essentially the same question.

As artificial intelligence becomes more capable and more widely available, perhaps the defining issue will not be whether humans can retain control over every machine. Perhaps that eventually becomes impossible.

The more fundamental question may be whether the individual retains somewhere intellectually independent from which to judge what the machine says.

Zuckerberg’s vision asks us to imagine a world in which everyone possesses superintelligence. The First Law asks what happens the morning after.

And perhaps the question that matters most is remarkably simple:

If everyone possesses superintelligence, what remains uniquely ours?

Robert Anthony · Author of The First Law