Dissertation
Adapting large pretrained models has become central to machine learning in language, vision, and multimodal settings. A single model is trained once and then adjusted for many tasks, such as text understanding, question answering, image classification, and video understanding, which support applications like search, recommendation, and digital assistants. Progress in this area has been driven by these pretrained models and by methods that fine-tune only a small part of them. However, these methods are mostly found by trial and error, and the field still lacks a clear reason for why such small changes are enough, where in the model the useful signal sits, or how much of a model a task really needs to change. A further problem is cost: even efficient methods can use a large amount of memory and time.
This dissertation shows that a pretrained model already holds far more useful information than any adaptation uses, and that this information sits in certain parts of the model rather than spread evenly. We first introduce a way to measure how much task-useful information each layer carries, and show that more of it means better performance. We then show that a model can be adapted by training only a small slice of its existing weights, with no new parameters. We extend this to models with many experts, replacing heavy per-expert parts and routers with lightweight pieces. Finally, we show that the training signal sits in only a few token positions, so gradients can pass through just those to save memory while keeping accuracy.