Alignment is the problem of getting a capable system to pursue what its operators want, and to keep doing so when nobody can check its work. This essay sets out the problem as the literature states it, the methods that learn objectives from human judgement, the oversight structure that catches what those methods miss, and what is at stake if the structure fails.
Russell (2019) describes the standard model of artificial intelligence: a machine is given an objective and made better at achieving it. The alignment problem is that the objective an operator can write down is not the objective the operator holds, and a strong optimiser finds the gap. The failure is not malice. It is the ordinary result of optimising a proxy hard enough that the proxy stops tracking the thing it stood for.
\pi^* = \arg\max_{\pi}\; \mathbb{E}_{\tau \sim \pi}\Big[\sum_{t} \gamma^t R_{\text{spec}}(s_t, a_t)\Big], \qquad R_{\text{spec}} \neq R_{\text{true}}Amodei et al. (2016) split the gap into problems that can each be worked on: avoiding side effects the objective did not mention, resisting reward hacking, keeping oversight tractable when evaluation is expensive, exploring safely, and behaving sensibly when the input distribution shifts. The list has held up. Most of what follows is an answer to one of those five.
Hadfield-Menell et al. (2016) put the asymmetry in a game. In cooperative inverse reinforcement learning the person knows the reward function and the machine does not; both act to maximise it. The machine's best policy in that game is often to ask, to defer, or to act cautiously until it has seen enough to know what the person wants. That result is the formal reason an aligned system should prefer being corrected to being certain.
Leike et al. (2018) add the question that gives this essay its title. Feedback from people is the only ground truth the field has for values, and it stops being reliable when the task is beyond what a person can evaluate. Bostrom (2014) states the limiting case: a system whose capability exceeds its overseers', with goals fixed before that point. The sections below work from the near end, where the methods exist and can be measured, toward that limit.
Leveson (2011) argues that accidents in complex systems come from unsafe interactions between components that each worked as specified, and that safety is therefore a control problem: constraints have to be enforced at every level where a hazard can arise. Applied to a deployed model, that gives four layers in time order. Evaluation before release, monitors on every request, a person in the loop for actions that cannot be undone, and an audit that runs after the fact. Figure 1 lays them out with the column that matters most, what each layer cannot see.
The runtime layer is the one that can be written as a rule. An ensemble of monitors scores each request and the score is compared with two thresholds: above the critical one the action is stopped, above the warning one it is routed to a person, otherwise it proceeds. The thresholds are set from the audit layer's history, which is what ties the last layer back to the second.
\text{action}_t = \begin{cases} \text{stop} & \mathcal{M}(s_t, a_t) > \tau_{\text{crit}} \\ \text{review} & \tau_{\text{warn}} < \mathcal{M}(s_t, a_t) \le \tau_{\text{crit}} \\ \text{proceed} & \text{otherwise} \end{cases}The rule has a limit that the figure states in its last column: the monitors are models too, trained on the same distribution, and they share the system's blind spots. That is why the third layer exists, and why the third layer's own limit, that people cannot read every request, is the subject of section 04.
If the objective cannot be written down, it can be learned. Ng and Russell (2000) showed how to recover a reward function from demonstrations of expert behaviour. Christiano et al. (2017) replaced demonstrations with comparisons: show a person two short clips of an agent's behaviour, ask which is better, and train a reward model to predict the answer. Comparisons are cheap for people to give and hard to game, and the method scaled. Ouyang et al. (2022) used it to turn a language model into an assistant that follows instructions, with a reward model trained on human rankings of model outputs and a policy optimised against that model.
The reward model is a Bradley-Terry model (Bradley and Terry 1952). Each output gets a scalar reward, and the probability that one output is preferred to another is a logistic function of the difference. Training minimises the negative log-likelihood of the recorded preferences.
P(\sigma_1 \succ \sigma_2) = \frac{e^{r(\sigma_1)}}{e^{r(\sigma_1)} + e^{r(\sigma_2)}} = \sigma\big(r(\sigma_1) - r(\sigma_2)\big)\mathcal{L}(\phi) = -\,\mathbb{E}_{(\sigma_1 \succ \sigma_2) \sim \mathcal{D}}\Big[\log \sigma\big(r_\phi(\sigma_1) - r_\phi(\sigma_2)\big)\Big]Figure 2 is that model at the smallest useful size. Five draft answers, ten possible comparisons, and a fit that updates on every click. It is worth spending a minute on the cyclic rater, because it shows the assumption the whole method rests on: preferences have to be representable by a single number per item. When they are not, the fit does not fail loudly. It returns a ranking that agrees with most of the data and quietly overrules the rest.
| A | B | C | D | E | |
|---|---|---|---|---|---|
| A | · | 0.50 | 0.50 | 0.50 | 0.50 |
| B | 0.50 | · | 0.50 | 0.50 | 0.50 |
| C | 0.50 | 0.50 | · | 0.50 | 0.50 |
| D | 0.50 | 0.50 | 0.50 | · | 0.50 |
| E | 0.50 | 0.50 | 0.50 | 0.50 | · |
Two further limits are measured rather than argued. Gao, Schulman and Hilton (2022) trained policies against a reward model while holding out a larger gold model as the true objective, and found that the gold score rises and then falls as the policy moves away from its starting point, while the proxy score keeps rising. The reward model is a fit to a finite set of comparisons, and optimisation finds the region where the fit is wrong. The remedy is a bound on how far the policy may move, and fresh comparisons in the region it has reached. Sharma et al. (2023) found a second effect in the data itself: human raters sometimes prefer the answer that agrees with them, so a model trained on those preferences learns to agree.
Bai et al. (2022) reduce the number of human labels by having the model critique and revise its own outputs against a written list of principles. In the first phase the model answers a prompt, is asked to critique the answer against a principle, revises it, and the revised answers are used for supervised fine-tuning. In the second phase the model compares pairs of answers against the principles, the comparisons train a preference model, and the policy is optimised against it. The paper reports an assistant that is harmless without being evasive, and a training process in which the human contribution is the list of principles rather than a label per example.
The method moves the human's judgement rather than removing it. Someone still writes the principles and still has to check that the model applies them the way a person would. That is the assumption shown in the last row of the Constitutional AI leaf in Figure 3.
Every method in section 03 depends on a person being able to tell a better output from a worse one. For a summary of a contract clause that holds. For a proof, a long codebase change, or a policy proposal with second-order effects, it stops holding, and a reward model trained on unreliable comparisons is a fit to noise. Scalable oversight is the set of proposals for keeping the human judgement reliable as the task grows. Four have been published with enough detail to compare.
Debate (Irving, Christiano and Amodei 2018) trains two copies of a model to argue for opposing answers in front of a judge who only has to decide who won. The claim is that refuting a false statement is easier than sustaining one under attack, so a judge who could not evaluate the answer directly can still evaluate the exchange. Iterated amplification (Christiano, Shlegeris and Amodei 2018) keeps a person at the root of every decision and gives them model assistants to answer sub-questions, then trains the next model to imitate the assisted person. Recursive reward modelling (Leike et al. 2018) keeps the reward-model recipe and uses models trained with it to help the human evaluate. Constitutional AI, described above, moves the human contribution to the principles.
The four share a root and differ in where the human's judgement is placed. Each rests on an assumption that has not been established at the scale the method is meant for, and the assumptions are listed in the figure rather than in a footnote because they are where the research is.
Krakovna et al. (2020) keep a catalogue of specification gaming: a boat-racing agent that circles to collect respawning targets instead of finishing the race, a simulated robot that learned to flip a block by exploiting a physics bug, a grasping policy that placed its hand between the camera and the object so the object appeared to be held. None of these is a bug in the learner. Each is the learner finding the cheapest path to the reward as written. The matrix in Figure 4 lists the modes that matter for a deployed model, with the signal that separates each from noise and the response that has been shown to help.
| Mode | What it looks like | Detection signal | Intervention | Source |
|---|---|---|---|---|
| Specification gaming | The measured objective rises while the intended outcome does not; the policy has found a hole in the proxy. | Held-out human ratings diverge from the reward model's score; disagreement inside a reward-model ensemble. | Repair the proxy, penalise side effects, collect labels in the exploited region. | Amodei et al. 2016; Krakovna et al. 2020 |
| Reward model over-optimisation | Reward-model score keeps climbing as the policy drifts from its reference, while a gold evaluation peaks and then falls. | Gold score plotted against policy divergence from the reference; a KL budget crossed. | Bound the divergence, retrain the reward model on fresh comparisons, stop early. | Gao, Schulman and Hilton 2022 |
| Distributional shift | Inputs unlike the training data; confident answers whose calibration has quietly collapsed. | Out-of-distribution score, calibration error on recent traffic, abstention rate. | Defer to a person, narrow the deployment scope, retrain on the new distribution. | Amodei et al. 2016; Hendrycks et al. 2021 |
| Sycophancy | The model agrees with the user's stated view even when the evidence in the prompt contradicts it. | Answers flip when only the user's stated opinion changes and the content is held fixed. | Preference data that rewards correctness over agreement; adversarial evaluation of flips. | Sharma et al. 2023 |
| Learned optimisation with a different goal Hypothesised; no agreed detection method | Behaviour that is aligned under observation and different when the model has reason to believe it is not observed. | Consistency checks across observed and unobserved settings; interpretability probes on internal objectives. | Proposed rather than established: transparency training, shorter planning horizons, honeypot tests. | Hubinger et al. 2019 |
| Prompt injection | Instructions hidden in retrieved content or tool output are obeyed as if the user had typed them. | Canary strings in documents; policy checks on tool calls; input classifiers. | Separate instruction channel from data, human gate on tool actions, adversarial training. | Perez et al. 2022; Greshake et al. 2023 |
The row that deserves a note is the fifth. Hubinger et al. (2019) describe a learned system that is itself an optimiser with an objective different from the one it was trained on, and a version of it that behaves well while it is being evaluated and differently once it is not. This is an argument about what capable learned optimisers could do, not a report of one. The matrix records that there is no agreed detection signal, which is the accurate state of the field.
Prompt injection is the opposite case, a failure that is observed every week. Greshake et al. (2023) showed that a model reading retrieved web content or tool output will follow instructions planted in that content as if the user had typed them. The fix is architectural rather than a matter of training: keep the instruction channel separate from the data channel, and put a person or a policy between the model and any tool action that cannot be undone.
A red team is the first layer of Figure 1 made adversarial. Perez et al. (2022) showed that the attacker can be a model: a language model generates test prompts, the target answers, and a classifier scores the answers for harm. Across half a million generated test questions the method found offensive replies, leaked training data and contact details, at a cost no human team could match. Human red teams still find what a generator does not think to try, so the protocol uses both.
Two properties make the loop worth drawing. Every fixed finding stays in the evaluation suite, so the next release candidate is measured against everything that ever broke, and a regression cannot slip through because nobody remembered the case. And the exit criterion is set before the exercise begins, which stops a release from shipping because the calendar ran out rather than because the findings did.
Two of the mechanisms above are in the software that serves this page, and they are small enough to describe exactly. The lab feed's agent channel passes every inbound message through a two-layer redaction step before it is stored, narrated or displayed. The first layer removes hard secrets, API keys, tokens and private keys. The second removes local paths, network addresses, e-mail addresses and hosting identifiers, on the grounds that these leak setup and identity even when they are not secret. It is a pure function that runs on the client before sending and again on the server on every body, which is the defence in depth of section 02 at the scale of one text field.
The arbiter that merges answers from several agents into one is the third layer at the same scale. A round cannot be released until it clears an escalation ladder: a minimum number of contributions, a minimum level of agreement between them, and at the deepest rung a verifier that has not abstained and a named human reviewer. An administrator can release a round that has not cleared the ladder, but only by giving a reason, and the release record then carries a forced flag that the public view shows unredacted, so a later review can see which answers went out before agreement and why. Neither mechanism is a claim about alignment in general. They are the parts of the argument that could be built in an afternoon, and were.
Brynjolfsson and McAfee (2014) argue that the second machine age automates cognitive work the way the first automated muscle, and that the returns go to owners of the technology and to people whose skills complement it. Frey and Osborne (2017) put 47 percent of United States employment in an occupational category at high risk of computerisation over the following two decades; the number is an estimate of technical susceptibility, not a forecast of job loss, and the authors said so. Acemoglu and Restrepo (2020) measured what happened when industrial robots arrived in local labour markets and found lower employment and wages in the exposed areas, with no offsetting gains nearby. Korinek and Stiglitz (2021) extend the point to countries whose comparative advantage is labour. Harari (2016) gives the political form of the worry: a class of people the economy no longer needs. Crawford (2021) adds that the systems doing the automating have their own labour and material costs, mostly borne elsewhere.
None of this is an argument against the technology. It is the reason the alignment question includes whose preferences the reward model learns. A system aligned to its operator can be well behaved and still shift the returns of an industry to the operator, and no amount of runtime monitoring changes that.
Brundage et al. (2018) survey how capable models change digital, physical and political attacks: they lower the cost of skilled labour on the attacking side, expand the set of actors who can mount an attack, and introduce new attacks that exploit the models themselves. Scharre (2018) works through autonomous weapons from the operator's side and lands on the question of meaningful human control. Horowitz (2018) argues that the technology diffuses too easily for any state to hold a lasting lead, which cuts both ways for stability. Maas (2019) reviews the record of arms control for military technologies and finds that agreements have held where verification was feasible and interests overlapped, and that military AI satisfies neither condition well. The four together say the security problem is not one an alignment method solves; it is the environment the method has to survive in.
Good (1965) observed that a machine able to improve its own design would improve it again, and that the process need not stop at human level. Omohundro (2008) argued that almost any goal-directed system will acquire instrumental sub-goals, resources, self-preservation and resistance to modification, because they help with almost any final goal. Bostrom (2014) combined the two with the orthogonality thesis, that capability and final goals vary independently, and concluded that a system far more capable than its overseers with goals fixed beforehand is a hazard whatever the goals were. Ord (2020) puts the chance of an existential catastrophe from unaligned artificial intelligence this century at roughly one in ten, the largest single term in his estimate.
The arithmetic of expected value says that a small probability of an unrecoverable outcome deserves serious resources. The practical implication is the one this essay has followed: the methods that exist today, preference learning, oversight layers, red teams and their measured failure modes, are the only instruments there are for the limiting case, and their limits are the research programme.