Tech · Hugging Face
DiScoFormer: One transformer for density and score, across distributions
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Pinning down that distribution means estimating two quantities: the distribution's density and, more useful as dimensionality grows, its score.
Key facts
- In 100 dimensions, it isn't close—against the best hand-tuned KDE, it cuts score error by about 6.5x and density error by more than 37x, and it keeps improving as you add samples, while KDE runs out
- Diffusion-based generative models (the technology behind AI image generators like Stable Diffusion and DALL-E) start from random noise and repeatedly follow the score, turning that noise
- Across the board, DiScoFormer beats KDE at both density and score estimation, and the gap widens exactly where KDE struggles
- DiScoFormer doesn't discard the classical method for a black box but instead includes KDE as a special case and improves on it
Summary
Many problems in machine learning and the sciences come down to the same task: you have a collection of data points and want to recover the distribution they came from—which values are common, and which are rare. Diffusion-based generative models (the technology behind AI image generators like Stable Diffusion and DALL-E) start from random noise and repeatedly follow the score, turning that noise into a realistic image. Extracting the density and score from a finite sample is challenging, and today's tools force a trade-off between generalizability and accuracy. The team introduce a new solution called the DiScoFormer (Density and Score Transformer) —one model that, given a set of data points, estimates both the density and the score of the distribution in a single forward pass without retraining. DiScoFormer maps an entire sample to the density and score of the distribution behind it using stacked layers of transformer blocks.