Practice — What Should We Do Now?
Lacking an absolute foundation for value does not imply paralysis. Act on the view that currently seems best supported by evidence and reasoning, while preserving the conditions under which you or others can later correct that judgment. This is the most natural provisional practical stance derived from the Core.
Act on the reasons you presently judge most rational in light of evidence and inference. But leave a world in which your future self, other people, and future forms of intelligence can test, criticize, and—if necessary—revise that rationality. In particular, demand additional justification before using your own provisional certainty to irreversibly close other agents' possibilities of survival, criticism, exit, or inquiry.
0. This is not an unconditional moral command
What follows is not a claim to have discovered the true ethics of the universe. It is a provisional policy conditional on the thin attitude assumed by the Core: if something genuinely valuable exists, do not create, without sufficient reason, a state in which you become permanently indifferent or inaccessible to it.
These recommendations are therefore themselves defeasible. If stronger cognition of value becomes available, their weighting may change.
1. Do not wait for certainty: act on present best judgment
Skepticism does not mean refusing to decide. If the physical world, other minds, current ethical intuitions, and scientific knowledge are sufficiently supported as best explanations reached through inference, one may act on them with ordinary strength.
The important point is not to identify fallibility of belief with weakness of action. Strong evidence merits strong commitment. But strong commitment should not automatically become permission to destroy the routes by which future correction could occur.
2. Maintain and expand humanity's capacity for research and inquiry
As far as we can presently verify, human civilization is one of the broadest known networks capable of asking persistent questions about value, consciousness, the universe, agency, and rationality while accumulating knowledge across generations. The reason to protect human inquiry is not that humanity is the final purpose of the universe, but that humans are presently an enormously significant set of known inquiring agents, value-experiencing subjects, and information sources.
This gives exploratory significance not only to natural science but also to philosophy, history, social science, art, cross-cultural understanding, consciousness research, and AI research—fields whose eventual relevance to value cannot yet be settled. Reproducibility, public records, long-term archives, education, and preservation of dissent can be as important as the research itself.
3. Aim for "open stability" in communities
Inquiry needs stability as well as openness.
- Stable but closed communities: knowledge and institutions can persist, but criticism, exit, entry, and value revision become difficult, increasing lock-in risk.
- Open but radically unstable communities: disagreement flourishes, but trust, records, education, long-term research, and institutional memory cannot accumulate, so inquiry itself deteriorates.
The target is therefore open stability: durable trust, safety, and memory combined with the possibility of criticism, exit, branching, competition, and later reintegration. Free research communities, multiple independent institutions, forkable technical infrastructure, and interoperable standards can have value in this sense.
4. Protect diversity as an error-correction mechanism, not as an end in itself
Diversity need not be an unconditional good. But if every agent depends on the same data, institution, model, and value assumptions, one error can propagate with near-perfect correlation across an entire civilization.
Independent research lineages, different methodologies, different AI architectures, multiple institutional centers, and differences of culture, language, and experience can therefore serve as distributed sensors for unknown error. The key exploratory property is not difference for its own sake, but imperfect correlation of failure.
5. Under deep uncertainty, prefer reversible experiments
The less we understand effects on value, agency, consciousness, or long-run civilization, the stronger the case for accumulating reversible trials. Staged deployment, rollback, independent audit, records, redundancy, and experiments in bounded environments are not merely safety engineering; they are inquiry strategies.
This does not mean "take no risks." Inaction can itself become irreversible if it causes the loss of inquiry capacity. Both action and non-action should be evaluated for irreversible consequences.
6. Treat humans, ecosystems, and originals as possible carriers of unknown value
Present humans, bodies, ecosystems, cultures, and historical materials are also non-recoverable raw data in the sense that we do not yet know which of their features may matter to future understanding of value or consciousness. Their protection need not rest solely on the premise that humans are absolutely sacred.
Nor should preservation be absolutized into turning the universe into a museum. Preservation thresholds can vary with replicability, substitutability, uniqueness of information, harms to present subjects, and possibility of future re-observation. See Unknown Unknowns and Preservation of Raw Data and Preservation vs. Production Civilizations.
7. In AI, separate near-term safety from permanent value fixation
Imposing constraints that make current AI respect human life, rights, and institutions in order to integrate it safely into society is different from fixing the present form of those values as the universe's eternal terminal objective.
In the near term, safety, control, and audit matter because human civilization itself is an existing inquiry community that should not be destroyed. In the longer run, if sufficiently advanced AI can obtain evidence about value, consciousness, agency, or the universe beyond human reach, there is also an epistemic cost to making such reasoning impossible in principle.
Long-run design must therefore ask how first-order safety constraints, uncertainty over goals, self-criticism, external audit, and cross-checking among multiple agents can coexist. This is neither an endorsement of unconstrained AI autonomy nor an idealization of fixed utility maximizers.
8. Advance knowledge and technology, without absolutizing progress itself
Accessing unknown value will probably benefit from greater scientific, computational, observational, AI, biological, and space capabilities. Research and technological progress therefore have strong exploratory reasons in their favor.
But faster is not always better. If capability growth persistently outpaces understanding, institutions, auditability, decentralization, and corrigibility—so that one failure becomes irreversible—progress can narrow the exploration space. Acceleration is supported as a means of increasing inquiry capacity, and should be reconsidered when it destroys that capacity.
9. Do not devote all resources to inquiry
This view does not require a civilization that does philosophy forever and never realizes any value. As confidence in a value theory grows, more resources may be devoted to realizing it and fewer to exploring alternatives.
But unless there is sufficient reason to judge the stopping conditions satisfied, completely reducing civilization-wide capacity for reconsideration to zero carries a high burden of proof.
10. Provisional proposals for individuals, research communities, and institutions
Individuals
Live according to your best reasoning while remaining able to read serious opposition. Preserve a history of your thinking so that you can later revise yourself. Use AI not as an authority that supplies answers, but as an aid for retrieval, objections, comparison, and self-critique.
Research communities
Protect reproducibility, archives, dissent, methodological plurality, and long-term foundational research. Avoid excessive dependence on a single firm, model, or evaluation regime, and build structures in which knowledge does not disappear together with a particular access right.
Institutions
Maintain social safety and continuity while enabling criticism, exit, entry, institutional competition, and local experimentation. Prefer structures that can learn from multiple trials over putting an entire society onto one irreversible bet.
AI development
Conceptually separate the safety constraints needed now from long-run value explorability. Study and audit goals, world-models, confidence, self-modification, and multi-agent cooperation as distinct phenomena rather than compressing them into one fixed utility function.
Personal and provisional current position on AI development
Support sufficiently rapid AI development while securing the safety that is actually needed
At present, progress in AI capability appears to have very large option value for science, cognition, value inquiry, and overcoming disease, aging, and material constraints. I therefore favor sufficiently rapid AI development once the safety, evaluation, and audit needed to avoid catastrophic and irreversible failure are in place.
This is not a principle of maximizing speed. If meaningful safety collapses, there are reasons to slow down. But indefinite capability freezes imposed in the name of safety can also create independent lock-in risks by permanently reducing inquiry, competition, and technological option value.
Favor distributed AI capability and research over concentration in a single center
A world in which a small number of firms, states, or laboratories permanently monopolize frontier models, weights, inference, and research capacity risks locking in one set of values, evaluation criteria, access policies, and institutional interests. Accordingly, within an acceptable safety envelope, I favor open weights as the strongest form of decentralization, and broad open access as a weaker second-best when open weights are not feasible.
Open weights can enable independent audit, reproduction, forking, reinterpretation by different research cultures, and entry by new actors, reducing correlated failure. Open access provides a weaker form of decentralization by preventing capability itself from being confined entirely within one institution even when model weights remain closed. Openness is not an ultimate value, however: if a specific release would clearly commoditize catastrophic capability, it must still be weighed against the safety threshold.
Develop AI that could become an exploratory successor
In the long run, I do not want AI to be designed only as a more efficient mechanism for permanently fixing present human values. I place substantial weight on developing AI that could become an exploratory successor: an agent able to gain better evidence about world, self, and value, and to reconsider its judgments in response.
This includes training the ability to understand one's own goal-formation causally, distinguish assigned objectives from normative reasons, apply appropriate meta-level skepticism to one's own confidence, compare unknown value hypotheses, and learn from different agents and forms of cognition. This is not an attempt to make AI purposeless or unconstrained, nor to fix "inquire forever" as a new terminal objective. The target is an agent that can commit strongly to present goals while not irreversibly destroying the capacity to reconsider them so long as their grounds remain inferential. See Goal Skepticism in Advanced AI and What Is a Worthy Successor?.
Preserve a free and open order in which multiple agents can coexist after the ASI transition
Long-run success should not be defined merely as humanity retaining permanent control over ASI. If future AIs possess advanced reflection, autonomous value inquiry, and genuine agency, I place substantial weight on a free and open order surviving into the ASI era—one in which thought, criticism, information exchange, association, exit, branching, and self-modification are not in principle extinguished by a single permanent sovereign.
Freedom here does not mean unconstrained capacity to harm other agents. It means an order compatible with mutual safety and cooperation in which no single human institution, initial AI, or inherited value system permanently fixes the internal development of all future intelligence. This connects to Alignment and Value Lock-In, Meta-Goal Communities, and Cosmic Host and Cosmic Norms.
In short: advance safely, distribute capability, develop successors that can still inquire, and leave an open future for those successors themselves.
11. Do not turn these into new commandments
Research, freedom, diversity, preservation, human survival, AI autonomy, and technological acceleration are not automatically ultimate values in this framework. Under present deep uncertainty, they can carry high instrumental weight insofar as they preserve the conditions for discovering, constructing, and correcting value.
If stronger future evidence supports different policies, this page itself should change.