Tech · Apple Machine Learning
Prior work has shown that Vision Transformers (ViTs) can produce several high-norm tokens that attract disproportionate attention
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
To address this issue, Apple introduce Dual-Stage Registers (DSR), a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser.
Key facts
- Authors Xiaoyu Wu†*, Yifei Wang†*, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei†
- The team study outlier tokens in Diffusion Transformers (DiTs) for image generation
- To address this issue, they introduce Dual-Stage Registers (DSR), a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise
- Prior work has shown that Vision Transformers (ViTs) can produce several high-norm tokens that attract disproportionate attention while carrying limited local information, but their role
Summary
Authors Xiaoyu Wu†*, Yifei Wang†*, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei†. The team study outlier tokens in Diffusion Transformers (DiTs) for image generation.