Judgement Cannot Be Delegated · 3 of 12

It is often said that the machine, at least, is impartial. It has no prejudices because it has no opinions, it has no interests because it is not paid, and that is why in more than one staffroom it is welcomed as the only neutral participant in the conversation. Nobody programmed that prejudice. And it is there all the same.

This is a reasonable idea, and it was the one held by the students in the first year of Bachillerato (Year 12) who took part in the experience documented by Miquel Àngel Fuentes Arjona, a secondary teacher at a state school in Catalonia and a trainer in teachers’ digital competence, reported in the ODITE Report 2026. At the start, when asked about bias, several answered that AI ‘was neutral because it is a machine’, one that it was completely objective, and two admitted that they did not know what the word bias meant. By the end of the sequence they were talking about discrimination, about stereotypes and about the origin of the data and the decisions of those who design it (4).

Someone chose what went in

It is worth going back for a moment to the mechanism of the previous article and giving it one more turn. A language model sorts probabilities for the next word from an enormous set of texts. But that set did not fall from the sky. The AI Literacy Framework published by the OECD and the European Commission in 2026 puts it with a precision that deserves to be read slowly: AI systems are trained to identify patterns among data elements that humans have selected, categorised and prioritised (1).

Three verbs, three decisions. To select is to decide what goes in and what stays out. To categorise is to decide which things count as the same thing. To prioritise is to decide what weighs more. None of the three is the decision to discriminate against anyone, and the result discriminates all the same. That is why the same framework holds that ‘bias inherently exists in AI systems’, which can also reflect societal biases present in the training data or in the design of the algorithms, and that humans can worsen or mitigate them, accidentally or deliberately, during design, development, testing and use (1).

The AI literacy framework that Kelly Mills and her colleagues published in 2024 at Digital Promise, a US non-profit organisation devoted to educational equity, gathers examples that students themselves spot and that go well beyond the cliché. Facial recognition systems that are more accurate with white faces than with others. Systems that do not recognise children’s voices. Voice assistants with a female voice by default, reinforcing a gender norm that nobody ever wrote down anywhere (2).

There is a finer mechanism, and it is the one worth understanding before signing a contract with a supplier. A model that allocates opportunities may use neither race, nor religion, nor gender, nor age, and discriminate in exactly the same way on the basis of the postcode and the family’s level of education, which work as substitutes for the prohibited variables (2).

It looks as though the solution would be to collect more data, doesn’t it? Well, it turns out that it is not. The same text warns that attempts to correct under-representation by gathering more information about already vulnerable communities have caused added harm, and raise serious problems of consent and surveillance (2).

The language it was thought in

The imbalance is not only one of categories. It is also geographical and linguistic.

Jasper Roe, of the School of Education at Durham University, with Leon Furze and Mike Perkins, argues in an article published in 2026 in Pedagogies: An International Journal that the absence of locally produced systems leads to a reliance on models from large corporations trained on Western-centric and English-language databases. To describe them, the authors take up the idea of ‘culture machines’, systems that tend to gravitate towards dominant cultural and linguistic generalities (3). Although it is worth saying what kind of work this is, because it marks how far it reaches: it is a conceptual analysis, with no data of its own, proposing a framework for thinking. It is not an empirical study.

Translated into a classroom, the effect becomes recognisable. Carles Sierra, research professor at the CSIC, director of the Artificial Intelligence Research Institute and former president of the European Association for Artificial Intelligence, illustrates it in the ODITE Report 2026 with a case any teacher can imagine: a system that recognised only Castilian accents could discriminate against students with other pronunciations. And he adds one that gives even more food for thought in a school, that of confirmation bias: a system that favours the predictable patterns already present in the data penalises or renders invisible students’ original, creative or minority answers, confirming only what was already in the record (4).

There is a second layer, rather less commented on. Bias does not come only from what went in, but also from what is adjusted afterwards. Roe, Furze and Perkins point out that distortions can be introduced not from the training data but by the developers themselves, who monitor and adjust the model’s output once it has been trained (3). The AI Literacy Framework confirms this from another angle: part of the training may include reinforcement learning, in which the system improves through trial and error guided by ratings and rewards (1). Someone decides which answer deserves a reward.

The corpus is the world that fitted in the data. The adjustment is the hand that afterwards trims the edges.

Two students, the same tool

At this point, the question stops being technical and becomes ours. If all the students in the school have the same tool open on the same screen, are they in the same situation?

Roe, Furze and Perkins answer that they are not, and the argument is the most useful one in their whole article. They distinguish between material access and epistemic access. The first consists in having the device, the connection and the licence. The second consists in knowing the cultural assumptions built into the tool, knowing that they are there and being able to question them. Those without the second are left, in the terms of Paulo Freire that the authors recover, as passive vessels for knowledge they are in no position to transform (3).

And the gap does not only separate countries, it separates desks. The same authors note that those who cannot pay for subscription models turn to the free tools, of potentially lower quality (3). In an ordinary classroom that means that two students handing in the same piece of work have not used the same machine, nor have they had the same capacity to argue with what the machine told them.

Hence the international consensus points where it points. The Delphi study coordinated by Jonas Hallström, of Linköping University, closed three rounds of consultation with agreement on only eight of twenty-three statements. Two of those eight are to the point here: that students need to learn about the limitations of these systems, for example those due to bias, and that critical thinking is the most important skill for examining bias, skewed data and deep fakes (5).

What there was no agreement on is just as interesting. The statement proposing that students need to open up the black box of AI did not reach consensus (5). Nobody is asking a fifteen-year-old girl to audit a model. What is being asked is that she should know that there are human decisions inside and be able to ask about them, because, as one of the panellists recalled, without a basic understanding of how bias is produced the system remains, for students, a magical black box in relation to which they have no opportunity for agency (5).

What to do with this

In the classroom, in primary. Olga Armengol Pastallé, a digital environment teacher at the Escola Sant Nicolau in Sabadell, documents in the ODITE Report 2026 the project ‘Can we detect insults with AI?’, developed with 52 students in Year 5 of primary school and two teachers working in co-teaching. The students first built a wall display with racist, sexist and homophobic words and insults, collected in class and at home with the families’ help, and then used them to train a classifier in Learning ML, programming the interface in Scratch 3.0 to observe the accuracy percentages. One student summed it up better than any definition: AI learns like us, ‘if we give it bad data, it gets things wrong too’ (4). The assessment was formative, with rubrics and a learning journal, not a controlled design.

In the classroom, in secondary. The Finnish project Generation AI, funded by the Research Council of Finland and awarded the European Digital Skills Award in 2026, offers free of charge the Teachable Machine application, which is not Google’s tool of the same name. It runs in the browser, locally, without collecting personal data, and allows an image classifier to be trained with the examples the group itself chooses. The activity consists in training it with a deliberately unbalanced set and checking whom it recognises least well. In this way, as the project itself puts it, AI is presented as a system designed and shaped by people, and not as an opaque black box (6).

For the leadership team. Two criteria before adopting any tool. The first is stated bluntly by Digital Promise: systems whose inner workings cannot be examined should not be used in high-stakes decisions, and it is worth asking which decisions those are in a school (2). The second is the question that is almost never put to the supplier: does it work equally well with the students we actually have (4)?

For the teaching staff. The systematic review by Iliana Guaranda-Arias and her colleagues at the State University of Milagro, in Ecuador, places the identification of bias among the contents that teacher training should include, alongside case analysis and source verification, with the caveat that the predominance of descriptive designs makes it impossible to establish causality (7).

To conclude

AI bias is not a moral failing of the machine or the wickedness of whoever programmed it. It is the accumulated result of reasonable decisions taken separately: what data are collected, how they are grouped and what is rewarded when the model is adjusted. That is why it is not corrected by demanding good intentions. It is corrected by teaching people to ask what is missing, who does not appear and what language this was thought in. It is a competence, and competences are taught.

Food for thought. Which of our students are today in a position to notice what the tool has left out, and which are not? And if that difference follows the line of what each of them has at home, which part of it can the school make up for this year?

Glossary

• Algorithmic bias. A systematic distortion in the results of a computer system that favours or harms particular groups, without anyone having explicitly arranged for it.
• Training data. The set of texts, images or records from which a system learns patterns. What is not in them does not exist for the system.
• Representativeness. The degree to which the training data reflect the real diversity of the people and situations the system is going to operate on.
• Proxy variable. A piece of data that does not name a protected characteristic but works in its place. A postcode can stand in for social background without mentioning it.
• Black box. A system whose inner workings cannot be examined from outside, so that the inputs and the outputs are known, but not the path between them.
  Epistemic access. As opposed to material access, which consists in having the tool, it is the possibility of understanding the assumptions the tool builds in and submitting them to discussion.

References

1. OECD and European Commission (2026). Cómo preparar a los alumnos para la era de la IA. Marco de Alfabetización en IA [Empowering learners for the age of AI. AI Literacy Framework].
2. Mills, K., et al. (2024). AI Literacy: A Framework to Understand, Evaluate, and Use Emerging Technology. Digital Promise.
3. Roe, J., Furze, L. and Perkins, M. (2026). Digital plastic: a metaphorical framework for Critical AI Literacy in the multiliteracies era. Pedagogies: An International Journal, 21(2), 185–199.
4. Informe ODITE 2026. Claves para una nueva educación [ODITE Report 2026. Keys for a new education]. Espiral, Educación y Tecnología. Chapters by Carles Sierra, Miquel Àngel Fuentes Arjona and Olga Armengol Pastallé.
5. Hallström, J., Mannila, L., Nordlöf, C., Heintz, F., Sperling, K. and Stenliden, L. (2026). AI literacy for K–12 education: an international Delphi study. Interactive Learning Environments.
6. Generation AI (2023–2026). Research project funded by the Research Council of Finland. Teachable Machine application page. https://www.gen-ai.fi/en/tools/tm
7. Guaranda-Arias, I., et al. (2026). Alfabetización ética en inteligencia artificial y pensamiento crítico docente: revisión sistemática [Ethical literacy in artificial intelligence and teachers’ critical thinking: a systematic review]. Innova Science Journal, 4(3).