Apple · Apple Machine Learning
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Locking Pretrained Weights via Deep Low-Rank Residual Distillation.
Key facts
- Locking Pretrained Weights via Deep Low-Rank Residual Distillation
- Authors Keitaro Sakamoto†**, Pierre Ablin, Federico Danieli, Marco Cuturi
- The team propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network (DLR-Net) of comparable parameter count
- The quality of open-weight language models has dramatically improved in recent years
Summary
Authors Keitaro Sakamoto†**, Pierre Ablin, Federico Danieli, Marco Cuturi. The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardware and software platforms. In some cases, however, concerns on modifying these weights towards unauthorized uses may outweigh the pros of giving users such a freedom. The team propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network (DLR-Net) of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation.