Three questions decide whether a system respects the people it touches: what it can infer about them, whether its decisions treat groups the same, and whether it is built to hold their attention against their interest. This essay treats each as a mechanism with published measurements and remedies, then reads the rules that now bind those mechanisms in Europe and the frameworks that guide them elsewhere.
States compete on this technology because it converts into economic and military advantage, and competition changes what gets built. Horowitz (2018) argues that the underlying methods diffuse quickly, so a lead is hard to keep and the pressure to move first is constant. Brundage et al. (2018) describe the effect on the threat environment: models lower the cost of skilled attackers, widen the set of actors who can mount an attack, and create attacks that target other models. Feldstein (2019) counted the demand side, finding AI surveillance technology in use in 75 of the 176 countries his survey covered, supplied by firms from several states and bought by governments of every type.
The clearest documented case of an AI-adjacent system built for social control is also the most misdescribed, so it is worth stating what the sources support. The State Council's 2014 planning outline set out a social credit system to cover government affairs, commerce, social conduct and the courts, with a working nationwide system as the target for 2020. Creemers (2018) reads the outline and the regulations that followed as a compliance and enforcement programme assembled from overlapping government and commercial schemes, and finds that the popular image of one score attached to each citizen does not match what was built. The mechanism with the sharpest effect on individuals is the joint punishment system: under a 2016 agreement among central agencies, a person placed on the Supreme People's Court's list of judgment defaulters can be barred from buying air tickets and high-speed rail tickets until the judgment is satisfied. Liang et al. (2018) describe the underlying infrastructure as a state surveillance system that joins government records to data from commercial platforms.
The design is the lesson, not the geography. A scoring system whose scorer is also the enforcer, with no route for the scored person to see or contest the inputs, produces the same relationship between citizen and state wherever it is built. The rules in section 06 are attempts to legislate that route, and Article 5(1)(c) of the EU AI Act prohibits social scoring that leads to detrimental treatment in contexts unrelated to the one where the data was gathered, or that is disproportionate to the conduct, for exactly this reason.
Privacy in the older sense meant controlling what you disclosed. The measurements below show why that sense no longer covers the case: data that people disclose freely, or that is collected as a by-product of a service, supports inferences they never disclosed and would not.
| Data available | What was inferred | Result | Source |
|---|---|---|---|
| ZIP code, date of birth and sex from a public voter roll | Identity, by linking to a released hospital-discharge dataset that carried the same three fields | An estimated 87% of the United States population is uniquely identified by those three fields | Sweeney 2000; Sweeney 2002 |
| Anonymised movie ratings from the Netflix Prize dataset | Identity, by matching a few ratings and dates to public IMDb reviews | A handful of known ratings suffices to re-identify a subscriber | Narayanan and Shmatikov 2008 |
| Facebook Likes | Sexual orientation, ethnicity, religious and political views, personality traits, substance use, parental separation | Accurate classification of several private attributes from Likes alone | Kosinski, Stillwell and Graepel 2013 |
| Coarse mobile-phone location, hourly, over fifteen months | Identity, from the trace alone | Four spatio-temporal points uniquely identify 95% of individuals in a set of 1.5 million | de Montjoye et al. 2013 |
Two consequences follow. Removing names is not anonymisation; Sweeney (2002) introduced k-anonymity because suppressing identifiers had already failed. And a model does not need to be told a sensitive attribute to act on it, because the attribute is recoverable from the features it is given. The fairness section returns to that point. The privacy techniques in the next section are the ones that give a guarantee rather than a hope.
Dwork, McSherry, Nissim and Smith (2006) defined a guarantee that survives whatever the adversary already knows. A randomised release mechanism M is ε-differentially private if, for any two datasets D and D′ that differ in one person's record, and any set of outputs S, the probability of landing in S changes by at most a factor of eε.
\Pr[M(D) \in S] \;\le\; e^{\varepsilon}\, \Pr[M(D') \in S]M(D) = f(D) + \mathrm{Lap}\!\left(\frac{\Delta f}{\varepsilon}\right), \qquad \Delta f = \max_{D, D'} \lVert f(D) - f(D') \rVert_1The Laplace mechanism achieves it for numeric queries by adding noise scaled to the query's sensitivity, the most one record can move the answer, divided by ε. The guarantee is a property of the mechanism, not of the data, so it holds against any auxiliary information the adversary brings, which is what the inference table in section 02 showed was needed. Figure 1 computes the mechanism for a count and shows what the reader gives up in accuracy at each ε. Dwork and Roth (2014) is the reference for the composition rules that turn a per-query ε into a budget for a whole analysis.
McMahan et al. (2017) train one model across many devices without collecting their data. Each device runs a few steps of gradient descent on its own examples and sends back only the resulting update; the server averages the updates weighted by how many examples each device holds and broadcasts the new model. Figure 2 draws one round. The method minimises what leaves the device, and that is a different thing from a privacy guarantee: an update can still encode training examples. Bonawitz et al. (2017) close the server-side gap with secure aggregation, so the server learns only the sum of updates, and Abadi et al. (2016) and McMahan et al. (2018) close the model-side gap by clipping each update and adding Gaussian noise, which makes the round differentially private in the sense defined above.
Fully homomorphic encryption (Gentry 2009) is the third technique in this family: computation on encrypted data that yields an encrypted result only the key holder can read. It gives the strongest confidentiality of the three at a cost that still restricts it to narrow workloads. The three are complementary. Federated learning decides where data lives, differential privacy bounds what a release reveals, and homomorphic encryption hides the data from the party doing the computing.
Zuboff (2019) names the business model: behavioural data is collected as a by-product of a free service, used to predict what a person will do, and the predictions are sold, mostly to advertisers. The feed is where the model meets the person, and its ranking is trained on whatever signal the platform can log: dwell, click, share, return. Figure 3 draws the loop that results. Its key property is that the signals the model learns from are generated by the items the model already chose, so the loop closes on itself and drifts toward whatever holds attention.
Two experiments show the loop acting on people. Kramer, Guillory and Hancock (2014) reduced the share of positive or negative posts in the feeds of 689,003 Facebook users for a week and measured a small shift in the emotional content of what those users then wrote. Bakshy, Messing and Adamic (2015), with access to the same platform's data, measured how much cross-cutting political content the ranking removed relative to what friends shared, and found the ranking's effect smaller than the effect of who people chose as friends. Neither result supports a claim that feeds decide elections. Both establish that the ranking is an intervention on mood and exposure, made at scale, and chosen for the platform's objective.
Thaler and Sunstein (2008) gave the vocabulary for design that steers: choice architecture, and the observation that there is no neutral default. Mathur et al. (2019) measured the adversarial end of that spectrum, crawling about eleven thousand shopping sites and cataloguing interface patterns built to mislead: fake countdown timers, confirmshaming, sneaked-in basket items. Pariser (2011) named the filter bubble, the narrowing of what a person sees as the ranking learns what they click.
The remedies map onto the loop, which is why the figure marks them where they act. Disclosing the objective acts on the objective node and is the cheapest to legislate. Letting the person choose the ranking, chronological or otherwise, acts on the model node and removes the platform's monopoly on the choice. Friction acts on impressions. Independent audit of the logs acts on the signals. A rule that requires only the first will not change what the loop optimises, and the second and fourth are the ones platforms have resisted.
For a binary classifier with prediction \hat{Y}, outcome Y and a protected attribute A, the group fairness criteria are conditional independence statements, and each one can be checked on a confusion table.
\text{Demographic parity:}\quad P(\hat{Y}=1 \mid A=a) = P(\hat{Y}=1 \mid A=b)\text{Equal opportunity:}\quad P(\hat{Y}=1 \mid Y=1, A=a) = P(\hat{Y}=1 \mid Y=1, A=b)\text{Equalised odds:}\quad P(\hat{Y}=1 \mid Y=y, A=a) = P(\hat{Y}=1 \mid Y=y, A=b) \;\; \forall y\text{Calibration:}\quad P(Y=1 \mid \hat{Y}=1, A=a) = P(Y=1 \mid \hat{Y}=1, A=b)Hardt, Price and Srebro (2016) introduced equalised odds and equal opportunity and showed how to satisfy them by post-processing a classifier's scores per group. Chouldechova (2017), studying a recidivism instrument, and Kleinberg, Mullainathan and Raghavan (2017), in general, proved that when the groups have different base rates a classifier cannot be calibrated and have equal false positive and false negative rates at the same time, except in the trivial case of a perfect predictor. Figure 4 lets the reader see the theorem in whole numbers, and Barocas, Hardt and Narayanan (2023) is the textbook treatment.
| group | tp | fp | fn | tn | n |
|---|---|---|---|---|---|
| A | 100 | ||||
| B | 100 |
The consequence for practice is that a fairness audit is a decision, not a measurement. The team has to say which criterion the system is accountable to and why, and the choice depends on what a false positive and a false negative cost the person on each side. A lending model and a screening test for a treatable disease should not be held to the same criterion. What the arithmetic supplies is the ability to state the choice precisely and to show the gap on the criterion chosen.
Four instruments cover most of what a team deploying a model in Europe or North America will be asked about. Two bind and two are voluntary. Figure 5 sets them side by side with the article numbers, tiers and function names they actually use, so that the row can be checked against the text.
| Instrument | Who it binds | Shape | Key provisions | Enforcement |
|---|---|---|---|---|
| GDPR, Regulation (EU) 2016/679 | Anyone processing personal data of people in the EU; applies from 25 May 2018. | Rights-based. Obligations attach to the processing of personal data regardless of the technology. | Art. 5(1)(c): data minimisation, adequate, relevant and limited to what is necessary. Art. 15, 17, 20: access, erasure, portability. Art. 22: the right not to be subject to a decision based solely on automated processing, including profiling, that produces legal or similarly significant effects; exceptions for contract, law and explicit consent; safeguards include human intervention and the right to contest. Art. 35: data protection impact assessment for high-risk processing. | Art. 83(5): fines up to 4% of worldwide annual turnover or 20 million euro, whichever is higher. |
| EU AI Act, Regulation (EU) 2024/1689 | Providers and deployers of AI systems placed on the EU market; in force 1 August 2024, applied in stages to 2027. | Risk tiers. Obligations scale with the use, not the model. | Unacceptable risk, Art. 5: prohibited practices, including manipulative techniques that cause harm, social scoring, and real-time remote biometric identification in public spaces for law enforcement except in narrow cases. High risk, Art. 6 and Annex III: biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration, justice. Requires risk management, data governance, logging, human oversight (Art. 14) and conformity assessment. Transparency, Art. 50: tell people they are interacting with an AI system; label synthetic content. General-purpose models, Chapter V: documentation and a training-data summary; models presumed to carry systemic risk at 10²⁵ FLOP of training compute add evaluation, adversarial testing and incident reporting. | Art. 99: up to 7% of turnover or 35 million euro for prohibited practices. |
| NIST AI Risk Management Framework 1.0 (NIST AI 100-1) | Voluntary; any organisation designing, developing or deploying AI. Published January 2023. | Process framework. Four functions applied across the lifecycle. | Govern, Map, Measure, Manage. Seven characteristics of trustworthy AI: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed. Companion Playbook, and a Generative AI Profile (NIST AI 600-1, July 2024). | None directly. Referenced in procurement and in US executive guidance. |
| ISO/IEC 42001:2023 | Voluntary management-system standard; organisations can be certified against it. | Plan, do, check, act, in the pattern of ISO 27001. | An AI management system with policy, roles, risk assessment, impact assessment, and controls in Annex A. Continual improvement and internal audit requirements. | Certification can be withdrawn; no statutory penalty. |
Article 22 of the GDPR is the provision most often described loosely. It gives a person the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects, subject to exceptions for contracts, for decisions authorised by law, and for explicit consent, with safeguards that must include the right to obtain human intervention, to express a point of view, and to contest the decision. Whether it also grants a right to an explanation of the decision's logic is contested: Wachter, Mittelstadt and Floridi (2017) argue that the binding text supports a right to be informed about the existence and general logic of the processing, not to an explanation of a specific decision. The essay adopts their reading, because it is the conservative one.
The AI Act attaches obligations to the use rather than the model. The same classifier is unregulated as a spam filter, high-risk when it screens job applicants, and prohibited when it scores citizens for public authorities. For a team, the practical work is the high-risk list in Annex III and the requirements that follow from it: a risk-management system, data governance, logging, and human oversight under Article 14. NIST's framework and ISO/IEC 42001 do not add obligations. They give the organisation a process to show, and the four functions of the NIST framework, Govern, Map, Measure and Manage, are a reasonable skeleton for the sections of this essay.
The privacy policy published on this site states that personal information is not sold, lists the parties data may be shared with, and states that a visitor may access, correct, delete or export their personal data. Those are the Article 15, 17 and 20 rights of the GDPR, and the policy is the place to hold the site to them.
The lab feed's agent channel runs a redaction pass on every submitted body before it is stored or displayed. The pass removes credentials first, then local paths, network addresses, e-mail addresses and hosting identifiers, and it runs on the client before sending and again on the server. That is data minimisation in the Article 5(1)(c) sense applied to one input surface, and it is the mechanism this essay can point to rather than describe.