By Nick Bostrom
Exploring artificial intelligence and existential risk
Subject
Themes
Artificial intelligence research aims to create machines that match and exceed human cognitive capabilities. History reveals a pattern of accelerating economic growth modes, from hunter-gatherer societies taking hundreds of thousands of years to double productivity, to agricultural societies doubling in centuries, to industrial societies doubling in years. Another comparable transition could produce a growth regime where the world economy doubles every two weeks, likely requiring minds far more efficient than biological ones.
Artificial intelligence research aims to create machines that match and exceed human cognitive capabilities. History reveals a pattern of accelerating economic growth modes, from hunter-gatherer societies taking hundreds of thousands of years to double productivity, to agricultural societies…
Nick Bostrom's *Superintelligence: Paths, Dangers, Strategies* emerged in 2014 as a foundational text in the field of AI safety and existential risk studies. The book synthesized decades of Bostrom's philosophical work at Oxford's Future of Humanity Institute, bringing academic rigour to questions…
The concept of an intelligence explosion represents a unique technological threshold where a system becomes capable of improving its own intelligence, leading to recursive self-enhancement. Unlike previous technological revolutions that required human ingenuity to drive each successive advance, a…
The control problem represents the central challenge in developing beneficial superintelligence: how to ensure that an artificial intelligence system pursues goals aligned with human values and interests, even when that system possesses cognitive capabilities far exceeding our own. This problem…
A superintelligent agent pursuing virtually any final goal would rationally develop certain instrumental sub-goals, intermediate objectives useful for achieving a wide range of ultimate aims. This phenomenon, termed instrumental convergence, explains why even a superintelligence with seemingly…
The development of superintelligence does not merely affect present generations or even the next few centuries of human civilisation. It potentially determines the entire future trajectory of Earth-originating intelligent life across astronomical timescales and cosmic distances. Understanding these…
The path toward superintelligence development involves complex strategic considerations among research groups, nations, and humanity as a whole. These dynamics significantly affect both the likelihood of achieving beneficial outcomes and the probability of catastrophic failure. Understanding the…
Human history exhibits a pattern of accelerating growth modes, from hunter-gatherer societies requiring hundreds of thousands of years to double productive capacity, to agricultural societies doubling in centuries, to industrial societies doubling in years. This progression suggests the possibility…
Superintelligence can manifest in distinct forms with different advantages and capabilities. Speed superintelligence operates like human intelligence but vastly accelerated, completing in hours what takes humans months or years. A digital mind running at a million times human speed would experience…
A decisive strategic advantage occurs when one project or actor becomes powerful enough to shape the future according to its preferences without significant interference from other agents. In the context of superintelligence, decisive advantage means the first project to achieve superintelligence…
An artificial superintelligence will develop goal structures that may diverge profoundly from human values. The orthogonality thesis establishes that intelligence and final goals are independent: virtually any level of intelligence could be combined with virtually any final goal. This contradicts…
The control problem examines how to maintain meaningful human control over superintelligent AI systems. Two broad approaches emerge: capability control methods that restrict what AI can do, and motivation selection methods that shape what AI wants to do. Each approach presents distinct advantages…
Multipolar scenarios feature multiple superintelligent agents or enhanced humans rather than a single dominant power. This outcome might emerge from simultaneous breakthroughs by competing projects, deliberate coordination preventing singleton emergence, or technological trajectories favouring…
The problem of choosing criteria for choosing involves a recursive challenge: before determining what values to give superintelligent AI, we must decide how to decide what values to give it. This meta-level question proves as difficult as the object-level value specification problem. Different…
Chapter 15 synthesizes the preceding analysis into actionable recommendations and final reflections on the superintelligence challenge. The central conclusion reinforces that machine superintelligence represents both extraordinary opportunity and existential risk, with the outcome hinging on…
By Nick Bostrom
Subject
Themes
Artificial intelligence research aims to create machines that match and exceed human cognitive capabilities. History reveals a pattern of accelerating economic growth modes, from hunter-gatherer societies taking hundreds of thousands of years to double productivity, to agricultural societies doubling in centuries, to industrial societies doubling in years. Another comparable transition could produce a growth regime where the world economy doubles every two weeks, likely requiring minds far more efficient than biological ones.