The EU has moved to control machine learning. What does this latest law mean for data scientists?

The EU AI Act just passed the European Parliament. You may think, “I’m not within the EU, whatever,” but trust me, this is definitely more vital to data scientists and individuals all over the world than you may think. The EU AI Act is a significant move to control and manage the usage of certain machine learning models within the EU or that affect EU residents, and it accommodates some strict rules and serious penalties for violation.

This law has loads of discussion about risk, and this implies risk to the health, safety, and fundamental rights of EU residents. It’s not only the chance of some form of theoretical AI apocalypse, it’s concerning the everyday risk that real people’s lives are made worse in a roundabout way by the model you’re constructing or the product you’re selling. When you’re aware of many debates about AI ethics today, this could sound familiar. Embedded discrimination and violation of individuals’s rights, in addition to harm to people’s health and safety, are serious issues facing the present crop of AI products and corporations, and this law is the EU’s first effort to guard people.

Regular readers know that I all the time want “AI” to be well defined, and am annoyed when it’s too vague. On this case, the Act defines “AI” as follows:

A machine-based system designed to operate with various levels of autonomy that will exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers from the input it receives, how one can generate outputs similar to predictions, content, recommendations or decisions that may influence physical or virtual environments.

So, what does this really mean? My interpretation is that machine learning models that produce outputs which can be used to influence the world (especially people’s physical or digital conditions) fall under this definition. It doesn’t must adapt live or retrain mechanically, although if it does that’s covered.

But if you happen to’re constructing ML models which can be used to do things like…

• choose people’s risk levels, similar to credit risk, rule or lawbreaking risk, etc
• determine what content people online are shown in a feed, or in ads
• differentiate prices shown to different people for a similar products
• recommend the very best treatment, care, or services for people
• recommend whether people take certain actions or not

These will all be covered by this law, in case your model effects anyone who’s a citizen of the EU — and that’s simply to name just a few examples.

All AI isn’t the identical, nevertheless, and the law acknowledges that. Certain applications of AI are going to be banned entirely, and others subjected to much higher scrutiny and transparency requirements.

Unacceptable Risk AI Systems

These sorts of systems are actually called “Unacceptable Risk AI Systems” and are simply not allowed. This a part of the law goes into effect first, six months from now.

• Behavioral manipulation or deceptive techniques to get people to do things they might otherwise not
• Targeting people on account of things like age or disability to alter their behavior and/or exploit them
• Biometric categorization systems, to try to categorise people in line with highly sensitive traits
• Personality characteristic assessments resulting in social scoring or differential treatment
• “Real-time” biometric identification for law enforcement outside of a select set of use cases (targeted seek for missing or abducted individuals, imminent threat to life or safety/terrorism, or prosecution of a selected crime)
• Predictive policing (predicting that folks are going to commit crime in the longer term)
• Broad facial recognition/biometric scanning or data scraping
• Emotion inferring systems in education or work and not using a medical or safety purpose

This implies, for instance, you may’t construct (or be forced to undergo) a screening that is supposed to find out whether you’re “comfortable” enough to get a retail job. Facial recognition is being restricted to only select, targeted, specific situations. (Clearview AI is certainly an example of that.) Predictive policing, something I worked on in academia early in my profession and now very much regret, is out.

The “biometric categorization” point refers to models that group people using dangerous or sensitive traits like political, religious, philosophical beliefs, sexual orientation, race, and so forth. Using AI to attempt to label people in line with these categories is understandably banned under the law.

High Risk AI Systems

This list, however, covers systems that are usually not banned, but highly scrutinized. There are specific rules and regulations that may cover all these systems, that are described below.

• AI in medical devices
• AI in vehicles
• AI in emotion-recognition systems
• AI in policing

That is excluding those specific use cases described above. So, emotion-recognition systems could be allowed, but not within the workplace or in education. AI in medical devices and in vehicles are called out as having serious risks or potential risks for health and safety, rightly so, and have to be pursued only with great care.

Other

The opposite two categories that remain are “Low Risk AI Systems” and “General Purpose AI Models”. General Purpose models are things like GPT-4, or Claude, or Gemini — systems which have very broad use cases and are frequently employed inside other downstream products. So, GPT-4 by itself isn’t in a high risk or banned category, however the ways you may embed them to be used is proscribed by the opposite rules described here. You possibly can’t use GPT-4 for predictive policing, but GPT-4 will be used for low risk cases.

So, let’s say you’re working on a high risk AI application, and you desire to follow all the principles and get approval to do it. Easy methods to begin?

For High Risk AI Systems, you’re going to be answerable for the next:

• Maintain and ensure data quality: The info you’re using in your model is your responsibility, so you might want to curate it rigorously.
• Provide documentation and traceability: Where did you get your data, and may you prove it? Are you able to show your work as to any changes or edits that were made?
• Provide transparency: If the general public is using your model (consider a chatbot) or a model is an element of your product, you have got to inform the users that that is the case. No pretending the model is just an actual person on the client service hotline or chat system. This is definitely going to use to all models, even the low risk ones.
• Use human oversight: Just saying “the model says…” isn’t going to chop it. Human beings are going to be answerable for what the outcomes of the model say and most significantly, how the outcomes are used.
• Protect cybersecurity and robustness: You want to take care to make your model protected against cyberattacks, breaches, and unintentional privacy violations. Your model screwing up on account of code bugs or hacked via vulnerabilities you didn’t fix goes to be on you.
• Comply with impact assessments: When you’re constructing a high risk model, you might want to do a rigorous assessment of what the impact could possibly be (even if you happen to don’t mean to) on the health, safety, and rights of users or the general public.
• For public entities, registration in a public EU database: This registry is being created as a part of the brand new law, and filing requirements will apply to “public authorities, agencies, or bodies” — so mainly governmental institutions, not private businesses.

Testing

One other thing the law makes note of is that if you happen to’re working on constructing a high risk AI solution, you might want to have a solution to test it to make sure you’re following the rules, so there are allowances for testing on regular people when you get informed consent. Those of us from the social sciences will find this beautiful familiar — it’s lots like getting institutional review board approval to run a study.

Effectiveness

The law has a staggered implementation:

• In 6 months, the prohibitions on unacceptable risk AI take effect
• In 12 months, general purpose AI governance takes effect
• In 24 months, all of the remaining rules within the law take effect

Note: The law doesn’t cover purely personal, non-professional activities, unless they fall into the prohibited types listed earlier, so your tiny open source side project isn’t prone to be a risk.

So, what happens if your organization fails to follow the law, and an EU citizen is affected? There are explicit penalties within the law.

When you do one in all the prohibited types of AI described above:

• Fines of as much as 35 million Euro or, if you happen to’re a business, 7% of your global revenue from the last 12 months (whichever is higher)

Other violation not included within the prohibited set:

• Fines of as much as 15 million Euro or, if you happen to’re a business, 3% of your global revenue from the last 12 months (whichever is higher)

Lying to authorities about any of these items:

• Fines of as much as 7.5 million Euro or, if you happen to’re a business, 1% of your global revenue from the last 12 months (whichever is higher)

Note: For small and medium size businesses, including startups, then the tremendous is whichever of the numbers is lower, not higher.

When you’re constructing models and products using AI under the definition within the Act, you need to at the start familiarize yourself with the law and what it’s requiring. Even if you happen to aren’t affecting EU residents today, that is prone to have a significant impact on the sphere and try to be aware of it.

Then, be careful for potential violations in your personal business or organization. You might have a while to search out and treatment issues, however the banned types of AI take effect first. In large businesses, you’re likely going to have a legal team, but don’t assume they will deal with all this for you. You’re the expert on machine learning, and so that you’re an important a part of how the business can detect and avoid violations. You should utilize the Compliance Checker tool on the EU AI Act website to provide help to.

There are numerous types of AI in use today at businesses and organizations that are usually not allowed under this latest law. I discussed Clearview AI above, in addition to predictive policing. Emotional testing can also be a really real thing that folks are subjected to during job interview processes (I invite you to google “emotional testing for jobs” and see the onslaught of corporations offering to sell this service), in addition to high volume facial or other biometric collection. It’s going to be extremely interesting and vital for all of us to follow this and see how enforcement goes, once the law takes full effect.