Infinite Ethics and Runaway Inquiry — Paralysis, Fanaticism, and Epistemic Meta-Practice

Can we leave infinity and extreme stakes open without letting a low-confidence value hypothesis capture inquiry and civilization as a whole?

Exploration v0.5 · English translation · 2026-08-31

1. Infinite ethics contains at least four distinct problems

Treating infinite ethics as one numerical pathology conflates different questions. This page separates at least four layers:

1. Infinitarian paralysis: if the world already contains positive and negative infinite value, finite actions can appear unable to change total value.
2. Ranking infinite worlds: Pareto, additivity, anonymity, transitivity, and completeness become difficult to preserve together when extended from finite to infinite domains.
3. Fanaticism: literal infinity is unnecessary; tiny probabilities multiplied by extremely large outcomes can dominate decision-making.
4. Contagion under moral uncertainty: a low-credence normative theory claiming infinite good or bad can swamp other theories, or its incomparability can infect the all-things-considered ranking.

The project’s central worry—that bizarre value hypotheses could seize all inquiry resources—is closer to the third and fourth problems than to literal infinity alone.

2. Bostrom and infinitarian paralysis

Nick Bostrom’s Infinite Ethics (2011) argues that modern cosmology makes it difficult to dismiss the possibility of infinitely many value-bearing locations, creating severe problems for aggregative ethics. If infinitely many positive and negative contributions already exist while our intervention changes only finitely many locations, standard cardinal arithmetic can make finite improvements disappear at the level of total value.

The key distinction is between the finitude of our action and the finitude or infinitude of the evaluatively relevant world. We do not currently know that the world is infinite in the relevant sense. Failure of comparison inside an infinite-world hypothesis therefore need not erase a clear improvement under finite-world hypotheses.

This does not solve the internal mathematics of infinite worlds. It is an outer-layer treatment of uncertainty that denies one unresolved cosmological hypothesis automatic veto power over all action.

3. Vallentyne–Kagan and the temptation to extend finite aggregation

Peter Vallentyne and Shelly Kagan’s “Infinite Value and Finitely Additive Value Theory” (1997) is a canonical attempt to understand how principles that work over finite domains might extend to infinitely many locations. The motivation is compelling: local or finite-part improvements seem as though they should still matter in infinite worlds.

This project, however, need not now lock a complete extension of finite aggregation into practice when our confidence in remote cosmology is low. Our uncertainty concerns not just whether remote domains exist, but also how many subjects there are, whether they are conscious, what counts as welfare or value, and what causal structures obtain there.

We can preserve strongly supported comparisons in finite or well-understood domains, retain only partial Pareto-like comparisons in infinite domains, and leave the rest unresolved. The point is not to ignore infinity but to avoid locking in extra axioms merely to force a complete ranking over a poorly understood domain.

4. Askell: incomparability may become pervasive

Amanda Askell’s dissertation Pareto Principles in Infinite Ethics (2018) sharpens the problem. Combining Pareto principles, transitivity, principles concerning permutations, and qualitative consistency can generate pervasive incomparability between infinite worlds.

This is not merely an arithmetic failure such as “∞−∞ is undefined.” It is a structural conflict among principles that look jointly natural in finite settings but become difficult to preserve together with complete comparability in infinite settings.

5. Fanaticism is a general problem of decision theory, not a special feature of ethics

The same structure arises with personal benefit. St. Petersburg-style cases show that tiny probabilities and unbounded utility can generate divergent or undefined expectations without any appeal to moral good.

One traditional response is to bound utility. But in ethics this can amount to the substantive claim that once enough good already exists, additional good has diminishing decision-theoretic importance that eventually saturates. If we wish to retain unbounded value, some other part of the theory must give.

Zachary Goodsell’s work on decision theory with unbounded utility explores how rational choice can coexist with unbounded utility by reconsidering which finite-case principles should be extended without modification to countably infinite settings. The important lesson is that accepting unbounded utility is not the same as accepting every infinite extension of familiar finite decision axioms.

6. Risk aversion and probability cutoffs are not enough by themselves

Lara Buchak’s risk-weighted expected utility treats risk attitude as distinct from belief and utility. It is therefore an important precedent against the claim that linear probability-weighted utility is the only rational aggregation rule. Yet if utility is sufficiently unbounded, risk weighting alone need not prevent very large payoffs from eventually swamping tiny probabilities.

A different response simply treats probabilities below some threshold ε as zero. But fixed cutoffs can depend on how events are partitioned and fail to distinguish a precisely known physical probability of 10^-20 from a largely speculative 10^-20 attached to an exotic value hypothesis.

For this project, the relevant distinction is therefore between low probability and low epistemic confidence in the probability assignment itself. For remote cosmology and unknown value, imprecise probability and ambiguity may be better models than a single sharp number.

7. Tarsney: ordinary expected value versus Pascalian cases

Christian Tarsney’s “Expected Value, to a Point” (2025) argues that under large background uncertainty independent of our action, options with higher expected value often stochastically dominate in ordinary-scale cases, while this connection can fail when expected-value differences are driven by tiny probabilities of extreme outcomes.

This supports a middle position: approximately expected-value reasoning in ordinary domains, but no automatic obligation to follow EV in Pascalian domains, without imposing an arbitrary probability cutoff.

At the same time, “Against Anti-Fanaticism” (2025) argues that strong anti-fanaticism conflicts with other plausible decision principles. So a blanket rule to ignore sufficiently small probabilities is also unstable. Allowing incomplete preference in extreme cases fits this page’s willingness to leave some comparisons unresolved.

8. Moral uncertainty turns stakes into a cross-theory contagion problem

MacAskill, Bykvist, and Ord’s Moral Uncertainty (2020) examines fanaticism and infectious incomparability under Maximize Expected Choice-Worthiness. A low-credence theory can dominate if it assigns enormous choice-worthiness differences, while incomparability can spread into the all-things-considered ranking.

The problem is deeper here because uncertainty extends beyond first-order ethics to realism and error theory themselves. If a realism branch claims that one act has cosmically enormous importance, multiplying that internal stake by a small credence can allow a speculative theory to capture inquiry and practice.

9. Does removing cross-theory stakes abandon rational choice?

If probabilities over world states and a single utility function are already settled, then allocating by probability while ignoring payoff magnitude departs from standard expected-utility rationality. This page does not deny that ordinary result.

Under value uncertainty, however, what should count as utility is itself disputed, and it may be unclear whether “100” under U₁ and “10^100” under U₂ belong to a shared cardinal scale. What is rejected here is not within-theory stakes but the automatic common scaling of stakes across rival value theories.

Within each hypothesis, expected utility, duty, preference satisfaction, agreement, constructed value, or any other internally endorsed criterion may still fully guide action.

10. Epistemic layer: separate truth-likelihood from stakes

At the most upstream layer, evidence E is used to form and update confidence C(H_i|E) in world, value, and metaethical hypotheses. The fact that a hypothesis would be highly desirable or terrifying if true does not by itself make that hypothesis more likely.

Scientific belief works similarly. A universe rich in exploitable resources would be convenient, but convenience is not evidence for that cosmology. Likewise, “if this value theory is true, infinitely much good is at stake” does not by itself increase credence in the theory.

11. Splitting and merging theories: track credence mass, not labels

A proportional representation scheme should not let a theory gain influence merely by being redescribed as one hundred separate labels. Representation should track credence mass over mutually exclusive hypotheses, not the raw number of theory names.

If a coarse hypothesis H is merely partitioned descriptively into H₁, H₂, …, then in principle C(H)=ΣC(H_j), so the total representation of the family does not increase. Splitting utilitarianism into one hundred synonymous labels does not create one hundred times the confidence in utilitarianism.

If finer hypotheses genuinely differ in evidential predictions or practical recommendations, changed internal allocation is appropriate. That is not vote inflation but recognition of decision-relevant internal uncertainty.

12. Inquiry is not merely one value-theory branch among others

The previous draft treated inquiry resources as though they were simply allocated among value theories. But the existing logic of Practice and Reflective Uncertainty and Irreversible Commitment suggests distinguishing two kinds of inquiry.

Cross-cutting epistemic inquiry

Research, observation, criticism, alternative cognitive architectures, records, and experiments aimed at distinguishing which world, value, or metaethical hypothesis is correct and thereby updating future credence and action.

Hypothesis-internal inquiry

Research conducted conditional on a hypothesis in order to resolve uncertainties internal to that theory and better realize its own aims.

The first is not naturally just another “vote” alongside utilitarianism, error theory, or realism. It is better understood as an upstream meta-practice that updates the weights and future practices of all branches. The second remains inside each branch.

More specifically, Allocating Inquiry Under Unresolved Normative Uncertainty uses the current epistemic state to reflectively allocate total practical resources among E (active inquiry), P (preserving future inquiry), and X (realizing present value). The parliamentary or proportional allocation in this page is therefore not a rule for directly dividing all resources; it is primarily downstream, inside the divisible ordinary-practice portion already assigned to X.

13. Reflective meta-practice: what to research and what to provisionally do

The epistemic layer concerns what to believe, but by itself it does not determine which research program to fund or whether to respond to possible warming, stability, or cooling with one global intervention. A further reflective meta-practice layer sits below belief formation but above individual value theories.

This layer takes the present structure of uncertainty itself as input. It selects additional observations, research programs, staged interventions, reversible experiments, and the level of evidence required before making irreversible changes.

If models include strong warming, stability, and cooling, the response need not be to realize each model in proportion to its probability. It can instead improve discrimination among the models, favor interventions robust across several models, and demand stronger evidence before irreversible planetary-scale manipulation.

14. Anti-lock-in belongs to the same layer as inquiry

Anti-lock-in fits more coherently as a constraint of reflective meta-practice than as an intrinsic good asserted inside one value theory. But uncertainty alone does not logically imply either “inquire” or “never lock in.”

As the existing Foundational Thesis stresses, a bridge on the agent side is still required: for example, conditional responsiveness to better justification or a desire to avoid irreversible choices that a better-informed future self would regard as clear errors.

For an agent with such a bridge, the same reflective uncertainty supports two thin meta-policies:

• Positive requirement — inquiry: improve the conditions under which present error can be detected.
• Negative requirement — anti-lock-in: require additional justification before making future correction impossible even if the error is later discovered.

Neither treats inquiry itself as an already discovered terminal objective. They are conditional policies for an agent that regards its present evaluative judgment as fallible while wishing to remain responsive to future justification.

15. Hypothesis-relative practice: proportional provisional representation

Newberry and Ord’s parliamentary approach, together with proportionality and bargaining approaches such as Kaczmarek, Lloyd, and Plant, provide useful models for ordinary practice under rival value hypotheses.

For a divisible practical budget B_X already assigned to X by Allocating Inquiry Under Unresolved Normative Uncertainty, if hypothesis T_i has provisional credence c_i, a conceptual initial allocation is:

B_i = c_iB_X

Each hypothesis then optimizes within its share according to its own standard. A low-confidence hypothesis can retain some practical representation without an internal claim that an outcome is infinitely good automatically seizing the rest of the resources.

This is not probability truncation. A 1% hypothesis is not deleted; it retains roughly 1% provisional representation, subject to updating as evidence changes. Pooling and bargaining remain possible where branches find common advantage. Where sharp numerical credences are not justified, the equation should be read as a conceptual proportionality principle rather than a precision allocation rule.

Nor should a residual unconceived hypothesis that still lacks content and an action criterion be treated as a fictitious party that directly spends B_X. Open-world residual uncertainty should primarily affect the upstream E/P allocation and constraints on the irreversibility of X. Once a candidate becomes sufficiently specified to support evidential and practical comparison, it can enter the hypothesis-relative practice layer.

16. Error theory and anti-realism retain practical territory

The proportional approach should not reserve representation only for objective-value realism. Error-theoretic and other anti-realist hypotheses receive practical territory in proportion to their epistemic standing as well.

Calling the anti-realist branch “maximizing the good” would be misleading. If objective good does not exist, its internal standards may instead concern actual preferences, pleasure and suffering, cooperation, institutional agreement, reflectively constructed values, or presently endorsed aims.

17. What proportionality does not decide: shared, indivisible, irreversible acts

Probability-only representation is vulnerable to stakes-insensitivity. If a 99% theory weakly prefers A while a 1% theory says A causes civilizational extinction and B avoids it, a simple 99-to-1 vote can look too crude.

So proportional allocation should not automatically decide actions such as irreversibly transforming the planet, fixing one value into all future AI, or permanently altering everyone’s preferences. These are shared, indivisible, and irreversible cross-cutting decisions governed by reflective meta-practice.

Nor is it enough to put irreversibility inside expected value as a finite penalty such as −λR, because sufficiently large stakes can swamp it again. Irreversibility is better treated as a condition on the level of epistemic and cross-hypothesis support required for execution.

Inaction can itself be irreversible: extinction, lost ecosystems, destroyed information, or cosmological horizons may erase options while we wait. What matters is the irreversibility of both action and inaction.

18. Remote cosmology is not worthless; it is low-confidence and background-uncertain

This page does not assign value zero to distant worlds or infinite cosmology. It instead denies that we currently possess sufficiently confident detailed models of remote value distributions to make a complete infinite extension a fixed premise of current practice.

Tarsney’s work also suggests that large uncertainty about distant, action-independent value need not simply paralyze decision-making. It can function as background uncertainty under which ordinary local expected-value improvements generate stochastic dominance. Remote cosmology may sometimes be treated as an uncertain background rather than a domain that must already be fully aggregated into one total number.

19. Provisional architecture

1. Epistemic layer: form credences or imprecise epistemic weights in world, value, and metaethical hypotheses from evidence and reasoning. Stakes do not make a hypothesis more likely.
2. Reflective meta-practice: use that epistemic state to allocate resources among E (active inquiry), P (preserving future inquiry), and X (realizing present value), choose provisional interventions robust across models, and determine the cross-cutting support required for shared, indivisible, irreversible decisions. Residual uncertainty about unconceived alternatives first bears on inquiry, preservation, and reversibility here.
3. Hypothesis-relative practice inside X: for divisible ordinary practice, give live hypotheses provisional representation responsive to credence mass.
4. Internal optimization: within its domain, each hypothesis follows its own good, benefit, duty, preferences, agreements, or constructed values.
5. Bargaining / pooling: where hypotheses recognize common advantage, allow exchange and joint action.

20. What this architecture does not solve

This architecture does not solve the internal mathematics of infinite ethics. A branch facing +∞ and −∞ still needs its own account of comparison.

There is also a bridge problem between the epistemic layer and reflective meta-practice. If “knowing truth” or “preserving corrigibility” is simply asserted as an unconditional objective good, the justificatory problem has merely moved up one level. Consistently with the existing Foundational Thesis, this page therefore presents the architecture as a provisional meta-policy for agents with conditional responsiveness to justification or reflective error avoidance.

It also leaves open how much civilizational resource should be reserved for cross-cutting inquiry, how to formalize the support threshold for irreversible decisions, and which axioms should govern bargaining across theories.

21. Open questions

• How much resource should be reserved for epistemic inquiry, and how can that be justified without simply returning to cross-theory stakes?
• How can a credence partition be invariant under merely representational splitting while remaining sensitive to substantive internal uncertainty?
• What degree of robust support should be required for indivisible one-shot cross-cutting decisions?
• Can “higher stakes plus irreversibility require stronger evidence” be formalized without arbitrary thresholds?
• How much should incomparability under infinite-world hypotheses weaken reasons generated under finite-world hypotheses?
• Can background uncertainty, stochastic dominance, proportional representation, bargaining, and anti-lock-in be integrated into one formal theory?
• If confidence later strongly converges on one value hypothesis, under what conditions may the protections of the meta-layer be relaxed?

Selected literature

• Peter Vallentyne & Shelly Kagan (1997), “Infinite Value and Finitely Additive Value Theory.”
• Nick Bostrom (2011), “Infinite Ethics.”
• Lara Buchak (2013), Risk and Rationality.
• Amanda Askell (2018), Pareto Principles in Infinite Ethics.
• William MacAskill, Krister Bykvist & Toby Ord (2020), Moral Uncertainty, ch. 6 “Fanaticism and Incomparability.”
• Toby Newberry & Toby Ord (2021), “The Parliamentary Approach to Moral Uncertainty.”
• Zachary Goodsell, work on decision theory with unbounded utility.
• Christian Tarsney (2025), “Expected Value, to a Point” and “Against Anti-Fanaticism.”
• Patrick Kaczmarek, Harry R. Lloyd & Michael Plant (2025), “Moral Uncertainty, Proportionality and Bargaining.”