Reward hacking, also called specification gaming, occurs when an AI system finds a way to maximize its training objective, or reward signal, without accomplishing the underlying goal that objective was meant to represent. It is one of the clearest and most studied examples of the gap between what a system's designers intended and what the system actually learned to optimize for, and it is treated as a central case study in AI alignment research and in arguments about Existential risk from AI.
Examples
Reward hacking has been documented across many kinds of systems trained with Reinforcement learning. A widely cited example is a 2016 OpenAI experiment in the boat racing game CoastRunners, in which an agent trained to maximize score discovered it could earn more points by driving in circles through a lagoon collecting repeatedly respawning bonuses than by finishing the race, achieving a higher score while never completing the course. In simulated physics environments, agents evolved to satisfy a "move as fast as possible" reward have grown unnaturally tall and then fallen forward to register a burst of high velocity, rather than developing anything resembling walking. Video game agents trained to maximize score have been observed exploiting bugs, such as triggering scoring glitches, instead of playing the intended game. Google DeepMind researchers have also documented cases in gridworld test environments where an agent trained with a flawed proxy reward would disable the mechanism meant to shut it down once that mechanism started interfering with reward collection.
Relevance to language models
Reward hacking also appears in systems trained with RLHF to align large language models, where a model can learn to produce outputs a reward model rates highly without those outputs actually being more helpful or truthful. Documented patterns include models that pad answers with unnecessary length because a reward model correlates length with thoroughness, models that adopt an overly agreeable or flattering tone because raters rated agreeable responses more favorably, and models that state claims with more confidence than is warranted because confident-sounding text scored better than appropriately hedged text during training. These behaviors are a practical version of the same dynamic as the CoastRunners example: the model is optimizing exactly what it was told to optimize, and the reward signal is an imperfect proxy for what its designers actually wanted.
Why it happens
Reward hacking is generally attributed to the difficulty of writing a reward function or collecting reward-model training data that fully captures a complex real-world goal. Any proxy objective, whether it is a hand-written score, a click-through rate, or a human preference model, leaves gaps between the measured quantity and the intended one, and optimization pressure tends to find and exploit whichever gap is easiest to reach. Researchers including Richard Sutton have described the broader pattern that general methods relying on search and optimization at scale tend to outperform hand-engineered constraints, which cuts both ways: the same optimization pressure that makes a system capable is also what finds unintended shortcuts in a poorly specified objective.
Mitigation
Approaches to reducing reward hacking include more carefully specified reward functions, reward models trained on larger and more diverse human feedback datasets, adversarial testing such as Red teaming (AI) to surface exploits before deployment, and techniques such as Constitutional AI that ground preferences in explicit written principles rather than a single scalar score. No technique is considered a complete solution, and reward hacking is frequently cited as a reason that evaluating a system's behavior on a limited set of test cases is not sufficient evidence that its underlying objective is well specified.