The Closed-World Problem in Moral Uncertainty — Do Known Theories Exhaust Value-Hypothesis Space?
1. What standard moral uncertainty theory solves
Contemporary work on moral uncertainty develops sophisticated methods for acting when an agent assigns credence to multiple moral theories. Typical models assign probabilities to utilitarianism, prioritarianism, deontology, or related views and ask how their assessments of choiceworthiness should be compared or aggregated.
This is an important problem. But the candidate theory set is often already given. The framework therefore mainly solves “how should we decide under uncertainty within this known set?” rather than “why should this set be treated as an adequate approximation to value-hypothesis space?”
2. The theory-individuation problem is already recognized
MacAskill, Bykvist, and Ord explicitly note a problem with simply following the single theory with the highest credence: which theory counts as the most probable depends on how theories are individuated. A broad theory can be subdivided into many fine-grained variants, lowering each variant's credence and changing which labeled theory appears to be the modal one.
So it would be inaccurate to claim that the moral uncertainty literature simply ignores partition dependence. Major treatments already recognize that named theories cannot be treated as atomic votes without further structure.
3. A more upstream closed-world problem remains
Theory individuation concerns how to partition hypotheses that are already under consideration. The present question sits one level upstream.
Do theories currently named in human moral philosophy nearly exhaust the possible structure of value? Or are they a small set of clusters that happened to become cognitively and historically accessible to one species under particular biological, cultural, and institutional conditions? If the latter, assigning essentially 100% of credence across known theories silently imposes a closed-world assumption.
Closed-world moral uncertainty: the true theory is assumed to be one of the theories we can presently enumerate, or a nearby variant.
Open-world value uncertainty: current theories are important candidates, but epistemic room remains for value structures that have not yet been conceptualized.
4. “Flattening the moral pool” should not mean a uniform prior
We cannot assign equal probability to every known and unknown value theory. There is no natural atomization or uniform measure over value-hypothesis space. Whether utilitarianism counts as one theory or millions of nearby variants depends on arbitrary representational choices.
The relevant flatness is therefore flat admissibility, not flat probability.
- Do not give a theory special prior authority merely because it has a familiar historical name.
- Do not assign zero prior merely because a theory or conceptual axis has not yet been articulated.
- Evidence, explanatory power, convergence, phenomenal data, and rational structure can of course shift credence strongly.
This is not a demand to believe bizarre unconceived theories as strongly as established ethics. It is a narrower refusal to identify historical visibility with truth-proximity.
5. Existing moral theories are also observational data
Existing ethics should not be discarded. Utilitarianism, deontology, virtue ethics, religious ethics, cultural norms, and ordinary moral intuitions are extremely important data generated by long-running human reflection on value.
But their epistemic role need not be limited to membership in a final list of candidate theories. Recurring patterns—symmetry, fairness, aversion to suffering, agency, honesty, cooperation—across partially independent cultures and intellectual traditions may be evidence about deeper value structure. At the same time, the historical existence of these traditions does not prove that their categories define the ultimate coordinates of value.
In that sense, existing morality is strong observational evidence and a set of provisional models, but not the hypothesis space itself.
6. The analogy with unconceived alternatives
Philosophy of science has long discussed unconceived alternatives: even if a current theory defeats all known rivals, a better theory might exist that no one has yet formulated. Value theory may have an analogous problem.
In value inquiry the unknown may be deeper than a novel combination of familiar ethical principles. New forms of agency, artificial consciousness, self-modifying intelligence, unfamiliar temporal experience, cosmic-scale cooperation, or phenomenal properties not available to present humans could supply inputs required to form value concepts that we cannot yet represent.
For that reason, the assumption that the major coordinates of moral truth have already been discovered may deserve even more caution than the analogous assumption in mature science.
7. At least four layers of uncertainty
- Within-theory uncertainty: uncertainty about parameters or applications inside one theory.
- Uncertainty among known theories: uncertainty among utilitarianism, deontology, virtue ethics, and other already articulated theories.
- Structural and metaethical uncertainty: uncertainty about whether value is subject-dependent, whether objective normativity exists, and how comparison or aggregation works.
- Uncertainty about unconceived hypotheses: the possibility that present agents lack the concepts, categories, or coordinates required to state the correct theory.
Standard moral uncertainty primarily sharpens parts of the middle layers. The broader project of value exploration extends upward into ontology and epistemology and outward into the last, open-world layer.
8. Open-world uncertainty does not imply permanent indecision
The possibility of unconceived theories does not require weakening every present moral judgment. Just as science can rely strongly on the best current physical theory while remaining open to revision, existing moral theories, intuitions, and institutions can receive strong practical weight in proportion to evidence.
The distinction is between concentrating credence within the current candidate set and irreversibly closing the candidate set itself. Even if known theories become overwhelmingly compelling for present action, permanently engineering civilization so that no new value coordinates can be introduced requires additional epistemic justification.
9. AI can turn this from a local philosophical issue into a civilizational one
If advanced AI can search a wider cognitive space than humans, one important role may be not merely choosing among familiar moral theories, but discovering or constructing new concepts of value, agency, and intersubjective structure.
From this perspective, alignment framed solely as selecting a mixture from the current human moral pool and fixing it permanently risks converting a closed-world assumption into technical lock-in. This does not imply unconstrained AI autonomy. It implies that near-term safety constraints should be conceptually separated from long-run capacity to reopen and enlarge value-hypothesis space.
10. Implications for the Core
If this exploration is right, preservation of explorability has an additional meaning. What must remain revisable is not only the vote among current moral theories, but also the partition itself—the ability to discard present coordinates and introduce new ones.
Moral uncertainty is therefore not rejected. It is repositioned as a local decision tool inside a larger framework of value uncertainty.
11. What would weaken this view?
- A demonstration that major contemporary moral theories already span rationally possible value structures as a sufficiently general parameter space.
- A demonstration that residual credence in unconceived theories is only double-counting or a partition artifact with no independent content.
- Independent reason to think the existing human conceptual scheme is in principle complete for value cognition.
- A demonstration that explicitly modeling open-world residual uncertainty makes no difference to inquiry, decision-making, or lock-in policy.
Sources / notes
For the standard framework of moral uncertainty and the theory-individuation problem, see William MacAskill, Krister Bykvist, and Toby Ord, Moral Uncertainty (Oxford University Press, 2020). For the analogy with unconceived alternatives, see Kyle Stanford, Exceeding Our Grasp: Science, History, and the Problem of Unconceived Alternatives (Oxford University Press, 2006). For one formal challenge to aggregating probabilistic moral theories, see Nicolas Côté, “There Is No Such Thing as Expected Moral Choice-Worthiness”, Canadian Journal of Philosophy 53 (2023). The contrast closed-world moral uncertainty / open-world value uncertainty / flat admissibility is the present project's formulation.