AI Value Exploration Notes
Exploration

Utilitarianism — Not a Final Foundation, but Why Welfare Still Matters

Utilitarianism · welfare · evolutionary debunking · Exploration v0.3 · English translation · 2026-10-16

Question: Is maximizing pleasure, preference satisfaction, or welfare a final principle strong enough to end value inquiry? Or should welfare remain a heavily weighted current value hypothesis while future correction remains possible?

1. Utilitarianism is not one theory

Positions called “utilitarian” contain several independent choices. One concerns what counts as welfare: hedonism centered on pleasure and pain; preference views centered on preference satisfaction; idealized-preference views that ask what a sufficiently informed and reflective subject would prefer; and objective-list views that include knowledge, achievement, friendship, autonomy, or other goods.

A second choice concerns aggregation: total or average welfare, critical-level views, prioritarian weighting, or saturation-like aggregation in which repetitions of the same kind of value have diminishing marginal importance.

A third question is whether welfare is the only value, or whether knowledge, truth, beauty, diversity, or cognitive achievement can have value independently of a subject's welfare. The latter begins to look more like pluralist consequentialism than utilitarianism in the narrow sense.

So the question is not the truth of one single theory. The more general question here is whether the welfare that we presently experience and prefer as valuable is sufficiently grounded to be elevated into the final objective of the cosmos.

Concern for welfare and reasons to add its bearers

Another distinction concerns whose welfare is evaluated and whether creating an additional subject with a good life is itself a reason for promotion. Caring about the future welfare of present subjects, caring about subjects who will live in the future, and adding happy subjects to increase total welfare are not the same judgment.

Practice uses “concern for welfare conditional on subjects' existence” as a working label for a response that cares about the welfare of present and future subjects without treating the creation of additional beneficiaries merely to add welfare as an independent demand. This is not a temporal distinction between present-oriented and future-oriented positions. It is also a different evaluative route from an agent's desire for welfare under error theory.

As relevant prior work, Johann Frick's Conditional Reasons and the Procreation Asymmetry addresses the procreation asymmetry and the non-identity problem by making reasons to confer welfare conditional on the subject's existence. Our working label does not mean adopting his theory in full or resolving population ethics.

In particular, restricting attention to the same future individuals who exist under every available choice can miss cases where choices change future identities. Which subjects and demands to compare remains a separate issue. Reasons to bring a subject into existence are also distinct from reasons to end an existing subject's life. Concern for welfare conditional on existence does not entail eliminating existing subjects to remove their demands.

2. Pleasure and pain contain important value-like appearances

This project does not dismiss pleasure and pain as mere biological signals. Severe pain in particular is not experienced only as “a certain neural process is occurring”; the experience itself seems bad-for-the-subject and to-be-avoided.

But three layers should be separated: phenomenal fact (pain has a certain aversive character), value-like appearance (pain appears bad for the subject), and objective normativity (therefore all subjects ought impartially to aggregate pain and minimize its total amount). Strong evidence for the first two does not automatically establish the third.

Pleasure and pain are therefore treated as important evidence rather than a final foundation. They may track part of true value, while leaving the utilitarian aggregation principle unsettled.

3. Evolutionary origins weaken this evidence without erasing it

Under the leading physical picture of the world, much of the human evaluative system—pain avoidance, desire, fairness, sympathy—admits evolutionary explanation. If avoiding pain promoted survival, pleasure reinforced adaptive behavior, and social preferences supported cooperation or reproduction, their existence need not be explained by positing independent objective value.

This weakens the inference “we strongly experience pain as bad, therefore minimizing pain is an objective cosmic final value.” But the existence of an evolutionary explanation also does not imply that pain is completely unrelated to objective value. Perception was shaped by evolution without thereby becoming wholly useless for tracking the external world.

The debunking implication is therefore narrower: evolutionary genealogy weakens the entitlement to treat our current evaluative structure as a final foundation. It is a reason not to lock pleasure, pain, or preference in irreversibly, not a reason to throw them away.

4. Idealizing preferences does not remove the problem completely

Naive preference utilitarianism faces misinformation, addiction, manipulation, adaptive preference, and impulsive desire. One response is to base welfare on what a sufficiently informed and rationally reflective subject would want, perhaps extending toward long-run extrapolation such as CEV-style approaches.

This is a real improvement, but idealization does not automatically erase the origin of value. A fully informed human may retain terminal preferences shaped by human evolutionary history, while the operator that defines “more rational”, “more coherent”, or “properly extrapolated” can itself introduce new normative premises.

actual preference → informed preference → idealized preference → extrapolated preference

Moving along this chain may remove simple contingencies without yet reaching objective normativity.

5. Non-descriptivism moves the question more than it solves it

Expressivism and related non-descriptivist views can answer part of the evolutionary challenge. If “pain is bad” expresses an attitude, plan, or normative commitment rather than a belief describing an independent moral fact, then the charge that the belief was evolutionarily distorted away from independent moral truth does not apply in the same form.

But this also weakens the truth-tracking claim. From “we have an attitude of avoiding pain” it does not follow that unknown AI, future posthumans, extraterrestrial life, or the universe as a whole ought to share it. Even if non-descriptivism is correct about human moral language, the possibility of practice-independent normativity is not thereby closed. See Expressivism.

6. Extreme optimization exposes what each theory really preserves

With sufficiently large resources and optimization power, utilitarian and neighboring consequentialist theories can yield outcomes that look extreme from current human intuitions:

This project does not reject a theory merely because these outcomes look strange. Current human intuitions may be wrong, and one of these outcomes might ultimately be correct.

Strangeness is information, not a refutation.

The purpose of the stress tests is to reveal what a theory preserves and what it is willing to sacrifice under sufficient optimization. The direct concern is not hedonium, alien minds, AI cognition, or a subjectless universe as such; it is fixing one of them as the irreversible final objective before having sufficient epistemic grounds that it is correct.

7. First reason to retain substantial provisional weight on welfare: it is a strong current hypothesis

Refusing to lock utilitarianism in as a final foundation does not remove welfare from the candidate set. Pleasure, pain, and preference have strong value-like appearances, occur across many subjects, are tightly connected with action and decision, receive importance across multiple ethical theories, and are comparatively observable states of existing subjects.

Accordingly, the hypothesis that welfare is at least part of true value is a serious current candidate. The stronger claim that welfare is the only thing ultimately valuable goes much further. This project gives significant epistemic weight to the former while leaving the latter unsettled.

8. Preferences and Cooperation Can Also Support Welfare Directly

Welfare need not matter only because it might be objectively valuable. Agents may already aim to reduce their own or others' suffering, improve life, and sustain cooperation. Such purposes and practices do not disappear under the hypothesis that objective morality does not exist.

This concern for welfare is not a new objective duty derived from error theory. It is practice grounded in existing preferences and mutually endorsed cooperative policies. Welfare guarantees or consequential assessment of institutions can be adopted conditionally when they serve those policies. Whose interests to include and under which constraints to aggregate them still require separate examination.

Welfare can therefore have several routes of support: possible normative importance, what agents directly want, maintenance of cooperation, and support for inquiry. All need not be counted as support for total utilitarianism. Several practical criteria supporting the same policy does not make them the same theory.

9. welfare is infrastructure for a community of value inquiry

Value inquiry cannot continue through abstract reasoning alone. Agents exposed to extreme suffering, fear, deprivation, conflict, or institutional collapse have less capacity for long-run pluralistic inquiry.

A sufficient level of welfare, preference satisfaction, freedom, trust, and social stability may be valuable in itself, but it can also be an instrumental condition for continuing value inquiry. Protecting welfare can therefore receive a double justification: welfare may itself be truly valuable, and even if it is not a final value, it can help maintain the community capable of learning what is.

10. The Possibility of Meeting Limited Demands at Low Opportunity Cost

With sufficiently advanced technology and productive capacity, reducing some suffering and deprivation among existing subjects and meeting their major demands may become inexpensive. Improvements in medicine, housing, food supply, assistive technology, and institutional security might not strongly compete with long-term inquiry.

But neither the size of the future universe nor the length of available time establishes that an improvement is decision-theoretically cheap. Compare a plan containing the improvement with feasible alternatives without it. Include not only resources but delayed research or deployment, safety risks, maintenance, and effects on future options. Suffering, conflict, and impaired inquiry resulting from withholding improvement also belong in the comparison.

The project's new model integrates multiple practical criteria according to current epistemic support into one conditional long-term plan. A welfare improvement becomes a strong candidate if it substantially satisfies important demands under some criteria while imposing little loss on other demands and corrigibility.

This explanation does not depend solely on the loss that would occur if welfare were objectively valuable. Preference-based practice under error theory and cooperation can support it directly. Nor does it require predicting that sufficiently intelligent future agents would endorse a concession to the present. Evaluation proceeds from current evidence and the present agent's adopted response policy.

The relevant demands have describable fulfillment conditions. Treating a particular illness or securing specified living conditions may admit limited achievement states. Present subjects' preferences are not necessarily cheap to satisfy, however, if they demand unlimited lifespan or resources. Improving the welfare of a certain number of people does not exhaust total utilitarianism as a whole.

Not requiring the continued creation of additional beneficiaries, each subject's demands having a fulfillment point, and achieving and maintaining that fulfillment at low cost must each be assessed independently. Accepting the first does not make all demands concerning longevity, continued life, or further welfare improvements finite. Conversely, even if a theory as a whole has no fulfillment ceiling, a particular treatment or improvement in living conditions can have a limited achievement state.

Assess fulfillment and low cost in relation to specific demands, periods, and maintenance conditions, rather than inferring them wholesale from a position's name. The allocation table in Practice is not a mechanism for completing welfare at a fixed cost and removing that evaluation from future decisions.

Fulfilled demands remain in the history carried forward to future decisions. Splitting issues or reevaluating a plan does not count the same improvement as a fresh benefit. Actual costs of maintaining treatment and institutions, responding to environmental change, and meeting new subjects' demands must also be included. A single expenditure does not complete ongoing life forever.

Even with epistemic allocations unchanged, changes in fulfillment and opportunity costs can change appropriate resource allocations. Reduced additional investment in a limited demand means neither lower credence in the theory valuing it nor exclusion of that position from future judgment.

The reason for advancing welfare improvements before or alongside inquiry therefore lies in a concrete plan that meets important, currently supported demands with sufficiently small losses to other demands and future corrigibility. It does not lie in declaring the expenditure cheap on a cosmic scale. Where technology and institutions make those conditions hold, present welfare and long-term inquiry can advance together.

Reflective Aggregation Based on Agent-Side Responses

11. Provisional position

This project does not reject utilitarianism. Pleasure and pain carry important value-like appearances, and the hypothesis that welfare is part of true value deserves substantial epistemic weight. Preferences and cooperation can also support welfare directly, and its protection sustains inquiry. Where a feasible plan meets limited demands at sufficiently small cost to other demands and future corrigibility, there is reason to incorporate that realization into the plan.

But none of this establishes that welfare is the only true value, that present human scales of pleasure and pain are cosmically correct, that all subjects' welfare can be linearly aggregated on one scale, that every available resource should be converted into welfare, or that present utilitarianism should be fixed as the irreversible objective of AI or future civilization.

Do not dismiss pleasure, pain, and welfare. But do not lock them in as the final foundation either.

Utilitarianism may ultimately turn out to be correct. So may hedonium, maximal diversity of unfamiliar minds, large-scale AI cognition, or some other consequence that looks strange from current human intuitions. The direct implication of current uncertainty is not to preemptively fix one answer, but to act substantially on the best current value hypotheses while preserving room to update value cognition.