OpenAI · AI Safety Institute · Google · Alignment Forum
Sequent: scale and automation for higher confidence in alignment
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Alignment is not on trackArtificial superintelligence (ASI) may be developed in the next few years.
Key facts
- The Alignment Team ran the £30m Alignment Project, and Timaeus has pioneered applying singular learning theory (SLT) to alignment
- Founding team:Geoffrey Irving, Chief Scientist at UK AISI; ex-DeepMind, OpenAI, and Google Brain
- Most expected impact comes from high success at automation, which means lots of tokens: Their goal is to raise $100-150M initially, but prepare to raise at least one order of magnitude more if they
- Taking a more principled approach offers better filters for deciding which directions of automated research are promising (a proof is worth a thousand experiments, and even a pseudo-proof is worth
Summary
At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well. They're aiming at 40-80 FTE two years from now. The Alignment Team ran the £30m Alignment Project, and Timaeus has pioneered applying singular learning theory (SLT) to alignment. Founding team:Geoffrey Irving, Chief Scientist at UK AISI; ex-DeepMind, OpenAI, and Google Brain. Daniel Murfet, Head of Research at Timaeus; left tenure to pioneer SLT for alignment. In this world, they probably have to settle well short of this ideal. Most AI lab approaches are reactive, resulting in methods that, while functional, do not yield principled insight into if or when they will fail.
Their hope is to build towards that confidence by exploring many different research bets in parallel, using a single organization setting to increase both sharing and amortization of work. Why a new big organizationThere are several existing non-profit organizations pursuing theoretical aspects of AI alignment. However, none of these have yet succeeded in making the transition to affecting how deep learning works at frontier scale, and the challenges to doing so on short timelines (say the next 2-3 years) are immense. This is most true for theoretical ideas and for new empirical approaches based in theory. Their experience working together between AISI Alignment and Timaeus is a key motivating example, and is one of the driving factors behind their joining forces.