AI Agent · Alignment Forum
Preface to the Computational Agent Theory and ASI Alignment Sequence
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI.
Key facts
- This was argued by Eliezer Yudkowsky since 2001 [2], and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al
- Although Alan Turing warned about machines taking control already around 1951, himself citing a fiction novel by Samuel Butler from 1872. ↩︎
- More recently the reporter founded the Computational Rational Agents Laboratory (CORAL), a dedicated research organization [5]
- In 2018, the reporter created the learning-theoretic AI alignment research agenda (LTA)
Summary
In light of this risk comes the question of how can they make ASI safe . Personally, the reporter started to be concerned with this problem around 2013. In 2023, the reporter wrote an updated overview of the agenda, including a sketch of their proposed alignment protocol: Computational Superimitation. This year has seen a breakthrough in a key missing piece of LTA: compositional learning theory (see Stringological Sequence Prediction, more on this in following posts). In this sequence, the reporter is planning to cover:.