AI Agent · Alignment Forum
Without computational constraints, it's trivial to write an algorithm that finds the perfect move every time
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
So, there are only two levels of chess-playing-power: perfect play and everything else.
Key facts
- Then, it doesn't matter whether the AI aligned to Alice according to their theory will interpret her preference as 2.99 or 3.01—both are acceptable outcomes
- This doesn't nail down a single UTM, but might come close enough for the differences not to matter in practice [5]
- To do so, it arguably has to create a sophisticated model of the environment [1], and use this model to build a long-term plan
- For any such example, the theory should ascribe specific preferences, beliefs, and any other alignment-relevant attributes to the agent [2]
Summary
The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. To do so, it arguably has to create a sophisticated model of the environment , and use this model to build a long-term plan. The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI they are trying to avoid or counteract; and (iii) the aligned ASI they are trying to create (such as the GDS ). Given that agents are at the core of both the threat and the proposed solution, the theoretical foundation for ASI alignment requires a theory of agents. The theory should be descriptive, i.e. applicable to all their main examples of agents.