Judgement Cannot Be Delegated · 1 of 12

It is easy to believe that artificial intelligence understands what we ask of it. Anyone who has written a message on their phone has, without knowing it, a tiny version of the same mechanism in their hand. The keyboard suggests the next word, and if you accept the suggestions one after another, without thinking, a whole sentence appears on the screen, well constructed, in our own tone, but perhaps saying nothing of what we wanted to say. It is an echo. A very probable echo.

José Antonio Bowen and C. Edward Watson, authors of one of the most widely read guides to teaching with artificial intelligence, start from a very similar point: search engines have spent years trying to guess the end of our sentences from what other people searched for before (1). Yet nobody has ever thought that Google understood them. With a language model the impression is different, because it answers questions we did not expect it to be able to tackle in impeccable prose, and then it seems that someone is speaking to us.

This experience sparks a certain initial enthusiasm and encourages many people to treat the machine with the rules of courtesy that hold between human beings, and of course with the same language and the same kind of dialogue. We greet it at the start, ‘good afternoon’, we thank it and we even say goodbye, ‘see you later’. And the astonishing thing is that it plays along, but… is this really the best way to have a useful conversation?

The educational conversation about artificial intelligence almost always stumbles over the same underlying problem: decisions are made about something that has not been understood in depth. This first article deals with the beginning of the journey.

What exactly does the machine do when it speaks?

The trick is in the bet

A language model does not search for information or consult a store of knowledge. It predicts the next word. That is the whole operation, repeated millions of times. The image of the bet describes what happens well. The model bets on the most probable word, writes it, and bets again, taking its previous bet as sound. It never asks itself whether it got it right. It simply carries on.

To be able to do this, the text is first turned into numbers, because a neural network works only with numbers. These minimal units are called tokens, and they represent words, parts of words, characters or punctuation marks. The model receives a sequence of tokens and calculates, across the whole available vocabulary, which one is most likely to come next.

This probabilistic mechanism can be observed from the outside. Bowen and Watson describe an internal setting that developers call temperature: when it is set to zero, the model always chooses the most probable word and its answers become predictable; when it is raised, it starts to select less probable words, and with them come both originality and nonsense. It is a single setting for both things. There is no dial for the interesting and another for the accurate, and that detail explains much of what we will see in the next article, ‘No Warning Light’.

This way of working does not, on its own, explain what we see on the screen. A word predictor has been on our phone keyboards for years and nobody mistakes it for someone to talk to. What separates one from the other is scale, and scale has two measures. The first is the text it is trained on: forty-five terabytes in the case of GPT-3, more than twelve trillion words. The second is the parameters, the internal variables that are adjusted during training. They do not count words; they count adjustments. GPT-2 had one and a half billion parameters; GPT-3 reached more than a hundred times that figure. The mechanism remained the same. The only thing that grew was the size.

In 2017 an architecture was also added, the transformer, which makes it possible to compare each token with all the others in the sequence at once, rather than one after another. More context, more nuance, more speed. No understanding.

Maps without territory

Si If the operation is so simple, why does it get so much right?

Dirk Wulff, of the Max Planck Institute for Human Development in Berlin, and Rui Mata, of the University of Basel, published a review in Trends in Cognitive Sciences in 2026 of what these systems can contribute to cognitive science. Their definition is sober: computational systems trained on text to learn the statistical regularities of language. And they add something decisive. As well as predicting, these systems turn text into numerical representations called embeddings, which place words and sentences in a space of many dimensions where proximity corresponds to similarity of meaning.

There lies the answer. What training produces is not knowledge of the world, but a map of proximities between the things people have said about the world. A very detailed map, built on an enormous portion of human language, in which ‘teacher’ sits close to ‘classroom’ and far from ‘turbine’. With a map like that you can cover a vast amount of territory without ever having set foot on any of it.

That map yields results that surprise even those who build it. Wulff and Mata point out that these models solve a wide variety of tasks without having been trained for each of them. The most debated case is a model called Centaur. Marcel Binz and his team presented it in 2025 in the journal Nature: they trained it on a very large collection of human behavioural data, drawn from experiments on memory, learning and decision-making. The model ended up predicting how people would decide in tasks it had never seen.

The result is remarkable, and Wulff and Mata themselves immediately qualify it in three ways. The model becomes unstable in the face of small variations in the wording of a task. Its internal representations remain opaque. And some researchers question at root whether any current model can teach us anything about the human mind.

That is the real state of the discussion, and it is worth passing it on to the teaching staff just as it is. The serious question is not whether the machine understands. It is what representation of the world a system that has only ever seen text can build, and how far that representation reaches when what needs explaining are processes that depend on the body, on perception and on action. Wulff and Mata leave it posed as an open question, not as a conclusion.

Why it convinces us

In 1950, Alan Turing asked whether machines can think and dismissed his own question as imprecise. He replaced it with a game: an interrogator converses in writing with two players and tries to work out which of the two is human. Bowen and Watson sum it up bluntly: what the test asks is ‘Can this chatbot make us believe we are interacting with a human?’. In 2022, a Google engineer was fired after publicly claiming that the company’s chatbot had become sentient. The machine did not need to think. It was enough that it seemed to.

What the machine does well, then, is sound like a person. This is not a side effect; it is precisely what it was trained to do: produce the most plausible continuation of a text written by humans. Bowen and Watson demonstrate this with a deliberately absurd request: a 200-word process for removing a peanut butter sandwich from a toaster, written in the style of the King James Bible, the 1611 English translation whose prose sounds like scripture. The model replied in that register, with the sandwich lodged in the depths of the appliance and the people dismayed at being unable to eat it. The authors immediately point out where the novelty lay: in the combination of the three pieces, the sandwich, the toaster and the biblical style, which only a person could have come up with. The machine answered more quickly than any of us would have, and that is true. The novelty had been supplied by the request, not by the response. The machine knew nothing about toasters or theology. It knew which words tend to follow which words when someone writes like that.

There remains a more serious objection, and it should not be dismissed. Alejandro Espeso-García, of the Catholic University of Murcia, argues in a 2025 article that these systems do not merely perform tasks: ‘they also emulate reasoning processes’. It is true that the most recent models chain intermediate steps together before answering, and that this allows them to solve multi-step problems and act with considerable autonomy.

But those steps are learned too. Wulff and Mata explain the procedure: the model is trained through reinforcement, rewarding the chains of steps that lead to a verifiable result. What happens inside those chains remains opaque even to those who build the models, and getting a great deal right does not guarantee explaining anything, as those same authors warn. Emulating a process and carrying it out are not the same thing.

There is one consequence of all this that deserves an article of its own: a system that calculates what is most probable produces falsehoods with the same fluency and the same assurance with which it produces truths.

What to do with this

In the classroom. The Finnish project Generation AI offers a tool called Little Language Machine, a browser-based environment in which students train their own language model, a real transformer much smaller than commercial ones. It runs locally and does not send data to external servers. Users watch the model move from random sequences of characters to increasingly coherent text, and can modify the training data, the size of the model and the randomness to observe what changes. The tool has been in open beta since 2025 and its release is expected in autumn 2026, so it is worth trying it out before taking it into class.

In the classroom. Repeat the toaster experiment with a request of your own, outlandish and very specific, and ask students to locate exactly where the creativity lay.

For the leadership team. A ten-minute script, for the teaching staff or for a meeting with families, with the three ideas from this post: the machine predicts, the map is not the territory, fluency convinces. With one caution. In an international Delphi study published in 2026 and led by Jonas Hallström, professor of technology education at Linköping University, the expert panel clearly rejected the idea that learning about artificial intelligence is mainly about using text-based generative applications such as ChatGPT: that statement obtained a mean of 2.037 out of 5. Literacy is not simply a matter of opening accounts and teaching people to write prompts.

To conclude

That same study holds that ‘technical understanding is a prerequisite for being critical and ethical’ with regard to artificial intelligence. That is why this series begins with the machine and not with school policies. It should be added, for the sake of accuracy, that the panel did not reach consensus that the black box has to be opened completely: that statement was discarded for lack of agreement among the experts. The disagreement is about how much technical depth is needed, not about whether any is needed. Understanding the mechanism is not an argument against the technology. It is the condition for being able to discuss it.

Food for thought. If tomorrow the machine were always right, would anything we have seen here change? And a second, more uncomfortable question: of the decisions we have taken this school year about artificial intelligence, which ones depended on believing that it really thought?