Efficient Learning & Inference

This work provides a theoretical framework for why fine-tuning small, randomly selected subnetworks (slices) within pretrained models can suffice for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property driven by (1) spectral balance—similar eigenspectra across different weight-matrix slices—and (2) high task energy—backbone representations retain rich task-relevant features. Building on this Universal Winning Slice Hypothesis, we propose SliceFine, a PEFT method that updates only selected slices of original weights, introducing zero new parameters (unlike adapter-based approaches). SliceFine matches state-of-the-art PEFT performance across language and vision tasks while improving training speed, memory efficiency, and model compactness.

We propose RoCoFT, a parameter-efficient fine-tuning approach for transformer language models that updates only a small number of rows and columns in weight matrices. Across medium-scale models (e.g., BERT/RoBERTa) and larger LMs (e.g., Bloom-7B, Llama2-7B/13B), RoCoFT achieves comparable or better accuracy than strong PEFT baselines with improved memory and compute efficiency. We analyze why the method works using neural tangent kernel tools, showing that kernels built from restricted row/column parameters can be numerically close to full-parameter kernels and yield similar classification performance. Ablations study row/column selection strategies and rank choices for effective implementation.

Deep Metric Learning (DML) often adapts pretrained models to local domains, but full fine-tuning can overwrite useful pretrained knowledge. This paper studies parameter-efficient fine-tuning for DML and introduces a Visual Prompt Tuning (VPT) framework for pretrained Vision Transformers. Within a proxy-based DML paradigm, we augment proxies by incorporating semantic information derived from the input image and ViT, optimizing visual prompts for each class. The resulting semantic proxy approximations improve metric learning performance while tuning only a small fraction of parameters, achieving comparable or better results than recent full fine-tuning approaches on standard DML benchmarks.

Video transformers commonly follow an image-pretrain then video-finetune pipeline, but full finetuning can be costly and unnecessary given strong pretrained image models. We propose AIM, a method to adapt pretrained image models for efficient video understanding by freezing the backbone and inserting lightweight adapters. AIM introduces spatial, temporal, and joint adaptations to gradually equip image models with spatiotemporal reasoning capability. Across multiple video action recognition benchmarks, AIM achieves competitive or better performance than prior work with substantially fewer tunable parameters and can flexibly leverage stronger image foundation models.

In slice-by-slice 3D medical image segmentation, the difficulty of segmenting each slice can vary widely, yet many methods apply the same 2D network to all slices, ignoring slice-level heterogeneity. We propose Med-DANet, a dynamic architecture network for multi-modal 3D MRI brain tumor segmentation that learns a slice-specific decision to select an appropriate model from a predefined model bank. This adaptive model selection provides a strong accuracy–efficiency trade-off. Experiments on BraTS 2019/2020 show comparable or better performance than prior methods with significantly reduced complexity, improving efficiency over strong 3D baselines without sacrificing accuracy.

Most neural networks are static and operate at a fixed inference complexity, while resource budgets can vary across devices and even over time on the same device. Training separate models for each budget is expensive. MutualNet trains a single network that supports diverse resource constraints by jointly training a cohort of configurations with varying widths and input resolutions. A mutual learning scheme transfers knowledge among configurations, enabling strong performance across budgets and improving representations overall. MutualNet generalizes across architectures (2D/3D) and tasks (classification, detection, segmentation, action recognition), reducing training cost versus training multiple independent models and often boosting single-configuration performance as well.