The AI you did not decide to buy
A retail chain kept its job applications in an HR system. The system received a new feature called recommended candidates, and the HR team took to it at once. It looked like a convenient filter, a quicker way through a pile of applications, and that is how they used it.
It was not a filter. It assessed the candidates and put them in order, and the order came from a model the supplier had built.
Meanwhile the company's own record of its systems went on describing the thing as a database of applications. That description had been true when somebody wrote it. It stopped being true when the feature arrived, and nothing in the company registered the change, because nothing in the company was set up to register it.
Nobody in that company sat down and decided to use AI to assess candidates. The feature arrived, it was useful, and it was used.
About the cases in this book
That chain is the first of several cases in this book, so before going further, what they are.
They come from our own work with companies on this subject. The companies were small and medium-sized: between twenty and two hundred and fifty people, in Bulgaria and in other member states of the EU. A software company, a retail chain, a logistics business. What they had in common was a small HR team, little time for checking anything, and a reasonable wish to stop doing the same repetitive task by hand.
Names and circumstances in every case are changed. Where a case involved an error, we say what was seen and what was done, and we do not present a supposed cause as a proven one. A case is there to show you what to look for in your own company, not to tell you what happened inside somebody else's.
Where this book says what the law requires, it names the provision, so you can read it yourself. Where it says what we think, it says so. Those are our views, formed from the work, and you are free to weigh them differently.
This book is about one subject: the AI an employer uses on candidates and staff. It does not cover data protection law beyond the few GDPR provisions it names, such as Article 22 on decisions based solely on automated processing, and it does not cover the labour law of individual member states. Within that subject it starts from the beginning and assumes no earlier reading about the AI Act.
How the feature gets in
A feature reaches an HR system by an update to software you already have, by a module added on top, or by a purchase somebody makes. Only the purchase is a moment when somebody in the company knowingly chooses to use AI on people.
An HR platform sold as a subscription changes on the supplier's timetable, not yours. Most releases fix bugs or tidy a form. Now and then one adds a tab that ranks applicants or a rota that fills itself in, and it may be on before anyone in your company has looked at it. So the useful question is who, if anyone, is in the room when that happens.
In the smaller companies we worked with, the decision came from the owner, who wanted hiring to go faster. In the larger ones, the initiative came from HR, and IT checked the integration and the security.
Look at what each of those people was checking. The owner was checking speed. IT was checking whether the tool connected to the other systems and whether it was safe to connect. Nobody in either arrangement was checking what the tool would decide about a person, because nobody had been asked to.
And in one of the cases, there was no decision to check. The supplier switched on a new AI feature while updating an HR system the company already had. There was no separate internal decision about using it, before or after.
Why the record still says something else
If you go looking for your company's description of its HR systems, you will probably find one. It will be accurate about the day it was written.
When we asked the companies we worked with what documents they had, the answer was much the same each time. A contract with the supplier. A general privacy policy. Short instructions for using the system.
What none of them had was a description of the AI feature itself: what it was for, and where its use should stop. Nor did they have any record of a check made before the feature started working on real candidates or real staff.
This is not carelessness. The documents describe what was bought, and what was bought was a database of applications, or a payroll system, or a rota tool. The feature that now ranks people was never bought. It was delivered. No process in a small company is built to produce a document for something that was delivered rather than bought.
So the gap between the record and the system is the normal state, not a sign that your company is badly run. It is still a gap, and it is the first thing this book asks you to close.
Not only hiring
The retail chain's feature was about candidates. The same arrival happens in the systems that manage the people already working for you.
The tools we saw in the companies we worked with make a short list. Ranking CVs. Evaluating how staff perform. Allocating shifts. Beside those, a chatbot that answers candidates' questions, and a generator that writes job adverts. The rest of this part keeps coming back to these five, so they are listed once, here.
The ranking, the evaluation and the allocation share something the chatbot and the advert generator do not. Their output lands on a person: whether that person is read, how they are rated, which hours they work. A performance module that starts suggesting ratings, or a scheduling system that begins to propose the rota by itself, comes in the same way as the recommended candidates tab. It is just as easy to miss, because the screen it appears on was already there.
So when you look, do not stop at the recruitment system. The time and attendance system, the scheduling tool and the performance review platform are each a place where a feature can arrive.
When an HR feature counts as AI
Not every new tab in an HR system is AI in the sense of the regulation, and the difference decides whether anything in this book applies to it. Article 3(1) defines the term. In its own words, an AI system is 'a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments'The EU AI Act defines an AI system as a machine-based system designed to operate with varying levels of autonomy, which may show adaptiveness after deployment and which, for explicit or implicit objectives, infers from the input it receives how to generate outputs such as predictions, content, recommendations or decisions that can influence physical or virtual environments.verified, checked 2026-10-12How this was checked.
Most of that sentence describes almost any modern software. The verb that separates the cases is infers. An AI system works out, from what it is given, how to produce what it gives back. Nobody wrote down the route from a CV to a score; the system arrived at it.
The Commission's guidelines on the definition are not binding, but they are the best reading available. They place outside it systems that only follow predefined, explicit instructions, with no learning, reasoning or modelling, and call this basic data processing.The EU AI Act defines an AI system as a machine-based system designed to operate with varying levels of autonomy, which may show adaptiveness after deployment and which, for explicit or implicit objectives, infers from the input it receives how to generate outputs such as predictions, content, recommendations or decisions that can influence physical or virtual environments.verified, checked 2026-10-12How this was checked
Inside an HR system, that line usually runs between two kinds of feature.
The first kind does what somebody in your company told it to. Show only applicants who hold a forklift licence. Sort applications by the day they arrived. Return every CV that contains the word Excel. Put anyone who has not completed probation on the early shift. Each is a rule a person set, and anyone who can open the settings can read it. If you ask why candidate A sits above candidate B, the answer is a sentence: she applied on Monday and he applied on Wednesday.
The second kind produces an order nobody in your company wrote. A match percentage beside each applicant. A score out of a hundred for each employee's quarter. A proposed rota that differs from last month's for reasons nobody typed in. Ask why candidate A sits above candidate B and the honest answer is that the system weighed things, and nobody here can say how. That is inference, and the feature is very likely an AI system.
Two cases cause most of the confusion in HR.
A rule base can be large, and size does not settle the question in either direction. Recital 12, as the guidelines quote it, counts logic- and knowledge-based approaches that infer from encoded knowledge among the techniques that make a system an AI systemThe EU AI Act defines an AI system as a machine-based system designed to operate with varying levels of autonomy, which may show adaptiveness after deployment and which, for explicit or implicit objectives, infers from the input it receives how to generate outputs such as predictions, content, recommendations or decisions that can influence physical or virtual environments.verified, checked 2026-10-12How this was checked. A screening module whose knockout questions your recruiters wrote, where every rejection can be traced to one of them, sits on the rules side. An engine that reasons across its rules to reach a verdict nobody wrote down may not.
A model can be hidden inside something that looks like a rule. A field called 'suitability' that fills itself in, or a rota preference that the system adjusts by itself over time, may be the output of a model even though the screen shows it as one more column. The appearance of the screen does not settle it. What produces the value does.
When you cannot tell, do not guess. Question 2 of the supplier letter, below, asks the supplier whether each feature follows criteria you configured or learns from data, and the answer usually comes back quickly.
The words on the screen
Suppliers rarely label a feature as AI on the screen where your staff use it. The word appears on the website and in the sales deck. Inside the product, the feature has an ordinary name.
These are the names worth stopping at:
recommended, suggested, or top candidates
match, fit, or compatibility, especially with a percentage
a score, a rating, or stars next to a person's name
a shortlist the system proposes
a rota, schedule or allocation that fills itself in
performance insights, risk flags, or anything that marks a person for attention
None of these proves a model is at work. A star rating can be a manager's own rating typed in last month. But each one is a place where something may be ranking, scoring or recommending people, and each one is worth a question.
The other place to look is the release notes. They are easy to skip, and they may be the only written evidence of when a feature arrived. Where they are published, somebody should be reading the HR system's notes, even quickly, and stopping at any of the words above.
What the companies did not know
A feature arriving without a decision was not the only gap. When we asked the companies in these cases what they knew about the tools they already used, the same answers kept coming back.
They had not worked out what the tool really did. There is a large difference between a tool that puts information in order, a tool that recommends a decision, and a tool that in practice decides who moves forward. The retail chain thought it had a filter. What it had was a tool that assessed people. Whether that assessment only advised HR, or in practice settled who went further, was a question nobody had asked.
They assumed that an established supplier had taken care of the questions. A large, well-known product feels safer than a small one, and in many ways it is. But the supplier does not know what your HR team does with the output, how many applications it receives, or who reads what. Those facts are yours.
And nobody could say who in the company was responsible when the tool got something wrong. Not legally, which comes later in this book. Simply in practice: if a candidate complained, whose desk would the letter land on, and what would they be able to find out?
You can test your own company against this in an afternoon. For each HR system, ask the person who uses it most:
What does it show you first, and what do you do with it?
If it ranks or recommends, who decided to switch that on?
If somebody asked why they were ranked low, who here could answer?
The last question is the one that tends to end the conversation. Write down that it did.
What the demonstration does not show
Where a company does go out and buy, the sales conversation is the moment the questions could be asked, and it is built to answer different ones.
In another of the cases, the supplier promised more objective selection and less administrative work. Both are claims about outcomes, and the company had no way of checking either.
The documentation that came with the product did not explain how the score was produced. It did not say for which kinds of position the system had been tested, or under what conditions its results were unreliable. The client was given a demonstration of the interface. It was not given the limitations.
A demonstration shows what your staff will see on the screen when everything works. It says nothing about how the tool arrives at an order, where it has been tried, or where it goes wrong. A polished demonstration can even make those questions harder to ask, because the screen looks finished, and finished things feel tested.
Noticing this does not need a lawyer. It needs somebody to put the questions before the contract is signed or the feature is switched on, and to keep the answers.
The supplier letter
This book asks you to put questions to your supplier at several points. Rather than send a new email each time, send one letter per system, keep a copy, and add to it. We call it the supplier letter, and the rest of the book refers to it by that name. It becomes one of the forms that come with the book.
The version you need at this stage is short:
Which features of this system rank, score, recommend or allocate people, and since which version or date has each been switched on for us?
For each of them, does it follow criteria we configured, or does it learn from data?
What does each take into account? Does any input describe how a person has behaved, or their personal traits or characteristics?
What is the intended purpose of each feature, as you state it? If you publish instructions for use, please send them.
For which roles and conditions was it tested, and when should its results not be relied on?
Does any feature infer the emotions, mood or intentions of anyone, or put people into categories from their face, voice or body?
Who at your company answers questions about these features, and by when can we expect an answer?
Keep each answer next to the question, with its date. If an answer does not come, record the date you asked and that nothing came back. An unanswered question you can show you asked is a different position from one you never asked.
Later chapters add to this letter, and the chapter on what to ask the vendor turns it into the full questionnaire.
Whose tool it is
Everything in this book assumes that the AI your company uses on people was built by somebody else. In the law's terms, the company that built it and puts it on the market is the provider, and your company, using it in its own hiring and management, is the deployer. The two carry different duties, and most of this book is about the deployer's.
That assumption can stop being true without anyone noticing. Under Article 25(1), a deployer is treated as the provider of a high-risk system, with the provider's obligations, if it puts its own name or trademark on one already on the market, makes a substantial modification to one that leaves it high-risk, or changes the intended purpose of a system that was not high-risk so that it becomes high-riskUnder the EU AI Act, a deployer, distributor, importer or other third party is treated as the provider of a high-risk AI system, and carries the provider's obligations under Article 16, if it puts its own name or trademark on a high-risk system already on the market, makes a substantial modification to such a system that leaves it high-risk, or modifies the intended purpose of a system that was not high-risk in a way that makes it high-risk.verified, checked 2026-10-08How this was checked. For an employer the last of these is the easy one to walk into: a general chat assistant, set up by HR to score every application against a job description, has been given a purpose in point 4 of Annex III that its maker never stated. Part 3 of this book takes the question up properly. For now, if you built or trained the thing yourself, note it beside the feature, because it changes which duties are yours.
Doing it this week
Start with the system your HR team opens every day. Usually it is the applicant tracking system or the HR platform that holds staff records. Ask the person who uses it to show you the screen, and sit with them for fifteen minutes.
You are looking for any place where the system ranks, scores, recommends or allocates a person. For each, note:
What it does, in the words your HR person uses, and what they do with the result.
When it arrived, and whether it came by update, by an added module, or by purchase.
Who decided to use it. If the answer is nobody, write nobody.
Then compare what you found with whatever your company's own record says about that system. If the record calls it a database of applications and the screen shows a ranking, you have found the same gap the retail chain had. Send the supplier letter for that system the same week.
Do not try to fix the gap yet, or to decide what the law makes of it. The point of this week is to see it clearly and to have it written down with a date.
You may find nothing, and that is a result worth writing down too. Or you may find that the system you thought of as a filing cabinet has been putting people in order for some time, and that nobody can say since when.
The form for this
Open: Register of AI uses in HR, one row per feature, not per system.
The fields that take the thinking: what the feature does with a person, and who decided to use it. The second is often a blank, and the blank is the finding.
A filled line: System: applicant tracking. Feature: recommended candidates, arrived with an update. What it does: scores applicants and puts them in order. Decided by: no internal decision on record. Our own record still calls the system a database of applications.
Read again when: the supplier releases an update, offers a new module, or somebody in HR mentions that the screen looks different.