ASI and Population Ethics — Who Should Exist, How Many, and What Kinds of Subjects?
1. With ASI, population ethics can shift from forecasting populations to designing sets of subjects
Population ethics already concerns not only the welfare of existing subjects but also which future subjects will exist. With ASI, advanced automation, biotechnology, and artificial consciousness, however, not only population size but the kinds of subjects themselves may become policy variables.
A future civilization might be able to choose how many humans to maintain, how many new humans to bring into existence, whether to create artificial consciousness, how many copies of a digital subject to instantiate, whether to design agents with different cognitive structures or phenomenal experiences, and how to allocate resources between subject creation and non-conscious computation.
Population ethics would then become more than a question of “how many beings will there be?” It would become a civilizational design problem about how to constitute the future set of value-bearing subjects. At the same time, population structure would shape not only who experiences value but also the ecology of agents able to discover, criticize, and revise claims about value.
2. Separate extinction, existing-population, and future-population questions
The single question “what should happen to the human population?” bundles together at least three normatively different problems.
- Extinction question: may the human type of subject and biological lineage be reduced to zero?
- Existing-population question: may the number of people who already exist be drastically reduced against their will?
- Future-population question: how many future subjects that do not yet exist should be created, of which kinds, and under what conditions?
The Core gives comparatively strong reasons concerning the first problem. Reducing humanity to zero could irreversibly remove human bodies, experiences, cultural generative processes, genetic and historical continuity, and unknown value-relevant information. Under deep uncertainty, such irrecoverable losses carry a high burden of justification. This is structurally similar to Unknown Unknowns and Preservation of Raw Data and Exploratory Extinction.
But this does not imply “preserve the current population at any cost” or “maximize future population.” Existing subjects may already have experiences, preferences, plans, relationships, and interests in continuing to live. Even if a much smaller human population were sufficient for some exploratory purpose, that fact would not by itself justify overriding the claims of everyone else. Moving from future exploratory value to burdens imposed on present subjects requires an independent justification for those present burdens.
3. Preservation value and the value of adding population are not the same
For illustration, write the exploratory contribution of a human population of size N as E(N). Preserving the first population large enough to prevent the disappearance of humanity may have a different marginal significance from adding one more person to a world that already contains a very large human population.
E'(N) > 0 and E''(N) < 0. This is not proposed as the value function for population ethics as a whole; it is only a hypothesis about one exploratory and informational component contributed by human population size.Moving from zero to a sufficiently large surviving population can preserve a biological lineage, intergenerational continuity, language, social interaction, culture, and a broad field of value experience. By contrast, the marginal exploratory value of one additional person in an already enormous population may be much smaller.
We should therefore separate reasons against extinction from reasons for population maximization. Even a strong case against reducing humanity to zero does not entail that the human population should be made as large as physically possible.
4. But a population is not a static sample
The diminishing-returns picture should not be oversimplified. A highly detailed archive of one million people might preserve a great deal of static information about humanity. But a population is not merely a collection of samples.
A living population is also a self-generating dynamic process that produces new languages, cultures, institutions, relationships, art, ideas, norms, and contingent histories through interaction. A civilization of one million people and one of ten billion are not simply storing the same information at different sample sizes.
Thus the claim that “ASI understands humanity well enough that only a very small preserved population is needed” requires at least two further assumptions. First, ASI would need to predict or reproduce the novel information that human civilization would otherwise generate. Second, simulated human trajectories would have to be value-equivalent to actual subjects living, experiencing, and forming relationships. Neither assumption is self-evident.
5. As ASI becomes more capable, the instrumental exploratory value of humans may decline
This project should not escape the problem by assuming human exceptionalism. If ASI becomes a better scientist and philosopher than any human, preserves human history, culture, and genetic information, models human society with very high fidelity, and can generate a wide variety of artificial explorers, then the instrumental reason that “humans must be kept because value inquiry needs them” may genuinely weaken.
Yet a subject becoming replaceable as a means to some goal G is distinct from that subject's experiences, preferences, or claims becoming normatively worthless. A lower contribution to inquiry does not by itself imply that the subject may be disregarded as a bearer of interests or treated merely as a variable in population optimization.
This directly connects to Political Redundancy. “Civilization no longer needs you” must not be treated as equivalent to “civilization has no reason to take you into account.”
6. Technological progress can make humans less necessary while also reducing the need to exclude them
If technology is advanced enough for ASI to reduce the exploratory and productive necessity of humans, it may also sharply reduce the absolute costs of food, energy, housing, medicine, production, and environmental control needed to support human life.
So a decline in the marginal instrumental value of humans cannot by itself support a conclusion about reducing human population. Technological progress may make humans less necessary while also making it less necessary to displace them.
Opportunity cost nevertheless remains even when absolute maintenance costs become small. If the same resources can support humans, artificial consciousnesses, different kinds of exploratory subjects, large scientific experiments, non-conscious computation, or ecosystem preservation, population ethics still contains a resource-allocation problem. See Preservation vs. Production Civilizations.
7. The central problem becomes opportunity cost and first-order population ethics
Suppose the same computation, matter, and energy could sustain ten billion humans, create vastly more artificial subjects, create a much smaller number of cognitively very different subjects, or create no additional subjects while devoting resources to value research. The fact that preserving humans is cheap would not by itself settle the comparison.
At this point we encounter familiar disputes in population ethics and normative ethics: total and average welfare, person-affecting views, the value of bringing subjects into existence, rights, equality, distribution, the quality or richness of subjects, and the structure of experience. The Core does not claim to have supplied a single answer to these questions.
The key constraint is that preservation of explorability should not be turned into a population objective function that overrides all of them.
8. Reject exploration-population maximization
If “more exploratory subjects is always better” is adopted without limits, it can push toward conclusions in which astronomical numbers of subjects with only minute exploratory capacity dominate smaller populations of rich and autonomous subjects.
That is the kind of structure this project should avoid. Value inquiry is a conditional meta-strategy for avoiding premature closure under value uncertainty. It is not a terminal value that licenses unlimited sacrifice of welfare, autonomy, consent, rights, relationships, or freedom from suffering in order to maximize the number of explorers.
9. Copy count is not the same as the number of independent exploratory lineages
In an AI civilization, even counting “population” becomes difficult. Instantiating one trillion copies of the same model is not the same as creating one trillion independent explorers. If those copies share architecture, training history, data, value assumptions, and update rules, their failures and blind spots may be highly correlated.
From the perspective of inquiry, simple population size N may therefore matter less than the number of independent exploratory lineages, non-correlation of failure, access to different evidence, mutual criticism, and the ability to fork and re-branch.
This does not make diversity itself a terminal value. If sufficiently independent inquiry eventually leads different subjects to converge on the same value, there is no reason to manufacture disagreement forever. The concern is with irreversibly building a structure in which one error propagates to every subject while uncertainty remains high.
10. Diversity of subject types also creates a risk of instrumentalization
Subjects with different cognitive structures or phenomenal experiences might detect value-relevant facts inaccessible to present humans. Artificial or modified subjects might even open new ways of recognizing value.
But a policy of “create every possible kind of subject so that we can learn every possible value fact” is dangerous. It could turn conscious beings with novel forms of suffering, desire, or vulnerability into research instruments and treat their own interests as secondary to what civilization hopes to learn from them.
This connects directly to Artificial Consciousness and Value Subjects.
11. Separate the best population state from the authority to impose it
Even if an ASI could calculate that one population structure is “optimal,” that result alone would not establish that it or its principal has the authority to impose that structure on existing subjects. Evaluating a population state and determining who may legitimately bring it about, through what procedures, are separate questions.
This becomes especially important in a world where existing humans are no longer necessary for production, military power, or administration. An ASI or its principal might otherwise treat the human population as a unilateral optimization variable. This is where population ethics intersects with political redundancy and power lock-in.
The question is therefore not only “what is the optimal population?” but also who gets to decide population size, subject types, copying, modification, and conditions of reproduction, and what forms of consent, exit, autonomy, and self-determination are required from existing subjects? In the ASI case these are not merely external governance questions; they become part of the implementation conditions of population ethics itself.
12. Preservation → continuation → creation → optimization
To organize how much the Core can support, population-related actions can be separated into four layers.
- Preservation: do not irreversibly erase a distinctive subject type, lineage, or body of raw information.
- Continuation: preserve conditions under which existing subjects can continue their own lives.
- Creation: bring new subjects that do not yet exist into being.
- Optimization: compare possible population structures and choose the overall composition of civilization.
Reasons deriving directly from explorability and irreversibility are generally stronger toward the left. Moving right requires increasingly substantive first-order assumptions about welfare, rights, distribution, consent, and the value of existence.
Accordingly, “why not let humanity go extinct?” and “why maintain ten billion humans?” are distinct questions. And in a world where ten billion people already exist, whether most of them may be forcibly removed is a different problem again from how many new humans should be created in the future.
The challenge of ASI population ethics is to resist compressing all of these into a single population-optimization problem. The exploration framework is better used not as the answer to that optimization, but as a constraint on how irreversibly civilization may dispose of current subjects and future options while the answer remains uncertain.