2026
Kowsher, Md; Polat, Ali; Ardehaly, Ehsan; Salehi, Mehrdad; Ghiasi, Zia; Murali, Prasanth; Chen, Chen
SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks Conference
Forty-Third International Conference on Machine Learning (ICML) 2026, 2026.
Abstract | Tags: ICML | Links:
@conference{Kowsher2026,
title = {SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks},
author = {Md Kowsher and Ali Polat and Ehsan Ardehaly and Mehrdad Salehi and Zia Ghiasi and Prasanth Murali and Chen Chen},
url = {https://facebookresearch.github.io/SliceFine/
https://icml.cc/virtual/2026/poster/65370
https://github.com/facebookresearch/SliceFine},
year = {2026},
date = {2026-07-09},
urldate = {2026-07-09},
publisher = {Forty-Third International Conference on Machine Learning (ICML) 2026},
abstract = {This paper presents a theoretical framework that explains why fine-tuning small, randomly selected subnetworks (slices) within pre-trained models is sufficient for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property, arising from two phenomena: (1) spectral balance— the eigenspectra of different weight matrix slices are remarkably similar—and (2) high task energy—their backbone representations (pretrained weights) retain rich, task-relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter-efficient fine-tuning (PEFT) in large-scale models. Inspired by this, we propose SliceFine, a PEFT method that uses this inherent redundancy by updating only selected slices of the origi- nal weights—introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of SOTA PEFT methods across various language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and prac- tice, offering a theoretically grounded alternative to existing PEFT techniques.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
This paper presents a theoretical framework that explains why fine-tuning small, randomly selected subnetworks (slices) within pre-trained models is sufficient for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property, arising from two phenomena: (1) spectral balance— the eigenspectra of different weight matrix slices are remarkably similar—and (2) high task energy—their backbone representations (pretrained weights) retain rich, task-relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter-efficient fine-tuning (PEFT) in large-scale models. Inspired by this, we propose SliceFine, a PEFT method that uses this inherent redundancy by updating only selected slices of the origi- nal weights—introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of SOTA PEFT methods across various language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and prac- tice, offering a theoretically grounded alternative to existing PEFT techniques.
Fioresi, Joseph; Vayani, Ashmal; Kulkarni, Parth Parag; Wang, Song; Shah, Mubarak
Learning to Share: Selective Memory for Efficient Parallel Agentic Systems Conference
Forty-Third International Conference on Machine Learning (ICML) 2026, 2026.
Abstract | Tags: ICML | Links:
@conference{Fioresi2026,
title = {Learning to Share: Selective Memory for Efficient Parallel Agentic Systems},
author = {Joseph Fioresi and Ashmal Vayani and Parth Parag Kulkarni and Song Wang and Mubarak Shah},
url = {https://arxiv.org/abs/2602.05965},
year = {2026},
date = {2026-06-14},
publisher = {Forty-Third International Conference on Machine Learning (ICML) 2026},
abstract = {Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results. To improve robustness and solution quality, recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories. However, parallel execution comes at a significant computational cost: when different teams independently reason about similar sub-problems or execute analogous steps, they repeatedly perform substantial overlapping computation. To address these limitations, in this paper, we propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks that enables selective cross-team information reuse while controlling context growth. LTS introduces a global memory bank accessible to all teams and a lightweight controller that decides whether intermediate agent steps should be added to memory or not. The controller is trained using stepwise reinforcement learning with usage-aware credit assignment, allowing it to identify information that is globally useful across parallel executions. Experiments on the AssistantBench and GAIA benchmarks show that LTS significantly reduces overall runtime while matching or improving task performance compared to memory-free parallel baselines, demonstrating that learned memory admission is an effective strategy for improving the efficiency of parallel agentic systems.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results. To improve robustness and solution quality, recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories. However, parallel execution comes at a significant computational cost: when different teams independently reason about similar sub-problems or execute analogous steps, they repeatedly perform substantial overlapping computation. To address these limitations, in this paper, we propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks that enables selective cross-team information reuse while controlling context growth. LTS introduces a global memory bank accessible to all teams and a lightweight controller that decides whether intermediate agent steps should be added to memory or not. The controller is trained using stepwise reinforcement learning with usage-aware credit assignment, allowing it to identify information that is globally useful across parallel executions. Experiments on the AssistantBench and GAIA benchmarks show that LTS significantly reduces overall runtime while matching or improving task performance compared to memory-free parallel baselines, demonstrating that learned memory admission is an effective strategy for improving the efficiency of parallel agentic systems.
Zhai, Kevin; Mollah, Sabbir; Wang, Zhenyi; Shah, Mubarak
CoRe: Context-Robust Remasking for Diffusion Language Models Conference
Forty-Third International Conference on Machine Learning (ICML) 2026, 2026.
Abstract | Tags: ICML | Links:
@conference{Zhai2026,
title = {CoRe: Context-Robust Remasking for Diffusion Language Models},
author = {Kevin Zhai and Sabbir Mollah and Zhenyi Wang and Mubarak Shah},
url = {https://arxiv.org/abs/2602.04096},
year = {2026},
date = {2026-03-13},
publisher = {Forty-Third International Conference on Machine Learning (ICML) 2026},
abstract = {Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
2025
Cui, Xuanming; Chionh, Wei Peng; Kuek, Adriel; Lim, Ser-Nam
Improving Soft Unification with Knowledge Graph Embedding Methods Conference
Forty-Second International Conference on Machine Learning, 2025.
Abstract | Tags: ICML | Links:
@conference{Cui2025b,
title = {Improving Soft Unification with Knowledge Graph Embedding Methods},
author = {Xuanming Cui and Wei Peng Chionh and Adriel Kuek and Ser-Nam Lim},
url = {https://openreview.net/forum?id=OOqvY9yvVG
},
year = {2025},
date = {2025-07-13},
urldate = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
abstract = {Neural Theorem Provers (NTPs) present a promising framework for neuro-symbolic reasoning, combining end-to-end differentiability with the interpretability of symbolic logic programming. However, optimizing NTPs remains a significant challenge due to their complex objective landscape and gradient sparcity. On the other hand, Knowledge Graph Embedding (KGE) methods offer smooth optimization with well-defined learning objectives but often lack interpretability. In this work, we propose several strategies to integrate the strengths of NTPs and KGEs. By incorporating KGE objectives into the NTP framework, we demonstrate substantial improvements in both accuracy and computational efficiency.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Neural Theorem Provers (NTPs) present a promising framework for neuro-symbolic reasoning, combining end-to-end differentiability with the interpretability of symbolic logic programming. However, optimizing NTPs remains a significant challenge due to their complex objective landscape and gradient sparcity. On the other hand, Knowledge Graph Embedding (KGE) methods offer smooth optimization with well-defined learning objectives but often lack interpretability. In this work, we propose several strategies to integrate the strengths of NTPs and KGEs. By incorporating KGE objectives into the NTP framework, we demonstrate substantial improvements in both accuracy and computational efficiency.
Li, Zhuoling; Xu, Xiaogang; Xu, Zhenhua; Lim, Ser-Nam; Zhao, Hengshuang
LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence Conference
Forty-Second International Conference on Machine Learning, 2025.
Abstract | Tags: ICML | Links:
@conference{Li2025,
title = {LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence},
author = {Zhuoling Li and Xiaogang Xu and Zhenhua Xu and Ser-Nam Lim and Hengshuang Zhao},
url = {https://lizhuoling.github.io/LARM_webpage/},
year = {2025},
date = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
abstract = {Recent embodied agents are primarily built based on reinforcement learning (RL) or large language models (LLMs). Among them, RL agents are efficient for deployment but only perform very few tasks. By contrast, giant LLM agents (often more than 1000B parameters) present strong generalization while demanding enormous computing resources. In this work, we combine their advantages while avoiding the drawbacks by conducting the proposed referee RL on our developed large auto-regressive model (LARM). Specifically, LARM is built upon a lightweight LLM (fewer than 5B parameters) and directly outputs the next action to execute rather than text. We mathematically reveal that classic RL feedbacks vanish in long-horizon embodied exploration and introduce a giant LLM based referee to handle this reward vanishment during training LARM. In this way, LARM learns to complete diverse open-world tasks without human intervention. Especially, LARM successfully harvests enchanted diamond equipment in Minecraft, which demands significantly longer decision-making chains than the highest achievements of prior best methods.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Recent embodied agents are primarily built based on reinforcement learning (RL) or large language models (LLMs). Among them, RL agents are efficient for deployment but only perform very few tasks. By contrast, giant LLM agents (often more than 1000B parameters) present strong generalization while demanding enormous computing resources. In this work, we combine their advantages while avoiding the drawbacks by conducting the proposed referee RL on our developed large auto-regressive model (LARM). Specifically, LARM is built upon a lightweight LLM (fewer than 5B parameters) and directly outputs the next action to execute rather than text. We mathematically reveal that classic RL feedbacks vanish in long-horizon embodied exploration and introduce a giant LLM based referee to handle this reward vanishment during training LARM. In this way, LARM learns to complete diverse open-world tasks without human intervention. Especially, LARM successfully harvests enchanted diamond equipment in Minecraft, which demands significantly longer decision-making chains than the highest achievements of prior best methods.
Giampouras, Paris; Cai, HanQin; Vidal, René
Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recovery Conference
Forty-Second International Conference on Machine Learning, 2025.
Abstract | Tags: ICML | Links:
@conference{nokey,
title = {Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recovery},
author = {Paris Giampouras and HanQin Cai and René Vidal},
url = {https://arxiv.org/abs/2410.16826},
year = {2025},
date = {2025-07-13},
urldate = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
abstract = {In this paper, we focus on a matrix factorization-based approach for robust low-rank and asymmetric matrix recovery from corrupted measurements. We address the challenging scenario where the rank of the sought matrix is unknown and employ an overparameterized approach using the variational form of the nuclear norm as a regularizer. We propose a subgradient algorithm that inherits the merits of preconditioned algorithms, whose rate of convergence does not depend on the condition number of the sought matrix, and addresses their current limitation, i.e., the lack of convergence guarantees in the case of asymmetric matrices with unknown rank. In this setting, we provide, for the first time in the literature, linear convergence guarantees for the derived overparameterized preconditioned subgradient algorithm in the presence of gross corruptions. Additionally, by applying our approach to matrix sensing, we highlight its merits when the measurement operator satisfies the mixed-norm restricted isometry properties. Lastly, we present numerical experiments that validate our theoretical results and demonstrate the effectiveness of our approach.
},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
In this paper, we focus on a matrix factorization-based approach for robust low-rank and asymmetric matrix recovery from corrupted measurements. We address the challenging scenario where the rank of the sought matrix is unknown and employ an overparameterized approach using the variational form of the nuclear norm as a regularizer. We propose a subgradient algorithm that inherits the merits of preconditioned algorithms, whose rate of convergence does not depend on the condition number of the sought matrix, and addresses their current limitation, i.e., the lack of convergence guarantees in the case of asymmetric matrices with unknown rank. In this setting, we provide, for the first time in the literature, linear convergence guarantees for the derived overparameterized preconditioned subgradient algorithm in the presence of gross corruptions. Additionally, by applying our approach to matrix sensing, we highlight its merits when the measurement operator satisfies the mixed-norm restricted isometry properties. Lastly, we present numerical experiments that validate our theoretical results and demonstrate the effectiveness of our approach.
Chehade, M. F. El Hajj; Ghosal, S. S.; Chakraborty, S.; Reddy, A.; Manocha, Dinesh; H. Zhu, H.; Bedi, Amrit Singh
Inference-Time Alignment of LLMs via User-Specified Multi-Criteria Transfer Decoding Conference
Forty-Second International Conference on Machine Learning, 2025.
Tags: ICML | Links:
@conference{nokey,
title = {Inference-Time Alignment of LLMs via User-Specified Multi-Criteria Transfer Decoding},
author = {M. F. El Hajj Chehade and S. S. Ghosal and S. Chakraborty and A. Reddy and Dinesh Manocha and H. Zhu, H. and Amrit Singh Bedi },
url = {https://arxiv.org/html/2410.09300v3},
year = {2025},
date = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Chen, Ziang; Chen, Xiaohan; Liu, Jialin; Wang, Xinshang; Yin, Wotao
Expressive Power of Graph Neural Networks for (Mixed-Integer) Quadratic Programs Conference
2025.
Abstract | Tags: ICML | Links:
@conference{Chen2025c,
title = {Expressive Power of Graph Neural Networks for (Mixed-Integer) Quadratic Programs},
author = {Ziang Chen and Xiaohan Chen and Jialin Liu and Xinshang Wang and Wotao Yin},
url = {https://openreview.net/forum?id=iqd8aHKwGA},
year = {2025},
date = {2025-07-13},
abstract = {Quadratic programming (QP) is the most widely applied category of problems in nonlinear programming. Many applications require real-time/fast solutions, though not necessarily with high precision. Existing methods either involve matrix decomposition or use the preconditioned conjugate gradient method. For relatively large instances, these methods cannot achieve the real-time requirement unless there is an effective preconditioner. Recently, graph neural networks (GNNs) opened new possibilities for QP. Some promising empirical studies of applying GNNs for QP tasks show that GNNs can capture key characteristics of an optimization instance and provide adaptive guidance accordingly to crucial configurations during the solving process, or directly provide an approximate solution. Despite notable empirical observations, theoretical foundations are still lacking.
In this work, we investigate the expressive or representative power of GNNs, a crucial aspect of neural network theory, specifically in the context of QP tasks, with both continuous and mixed-integer settings. We prove the existence of message-passing GNNs that can reliably represent key properties of quadratic programs, including feasibility, optimal objective value, and optimal solution. Our theory is validated by numerical results.},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Quadratic programming (QP) is the most widely applied category of problems in nonlinear programming. Many applications require real-time/fast solutions, though not necessarily with high precision. Existing methods either involve matrix decomposition or use the preconditioned conjugate gradient method. For relatively large instances, these methods cannot achieve the real-time requirement unless there is an effective preconditioner. Recently, graph neural networks (GNNs) opened new possibilities for QP. Some promising empirical studies of applying GNNs for QP tasks show that GNNs can capture key characteristics of an optimization instance and provide adaptive guidance accordingly to crucial configurations during the solving process, or directly provide an approximate solution. Despite notable empirical observations, theoretical foundations are still lacking.
In this work, we investigate the expressive or representative power of GNNs, a crucial aspect of neural network theory, specifically in the context of QP tasks, with both continuous and mixed-integer settings. We prove the existence of message-passing GNNs that can reliably represent key properties of quadratic programs, including feasibility, optimal objective value, and optimal solution. Our theory is validated by numerical results.
In this work, we investigate the expressive or representative power of GNNs, a crucial aspect of neural network theory, specifically in the context of QP tasks, with both continuous and mixed-integer settings. We prove the existence of message-passing GNNs that can reliably represent key properties of quadratic programs, including feasibility, optimal objective value, and optimal solution. Our theory is validated by numerical results.
Roch, Zachary Andrew; Atia, George; Wang, Yue
Efficient and Scalable Reinforcement Learning for Average Reward under Model Uncertainty Conference
Forty-Second International Conference on Machine Learning, 2025.
Tags: ICML
@conference{Roch2025,
title = {Efficient and Scalable Reinforcement Learning for Average Reward under Model Uncertainty},
author = {Zachary Andrew Roch and George Atia and Yue Wang},
year = {2025},
date = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Zhang, Chi; Jia, Ziying; Atia, George; He, Sihong; Wang, Yue
Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning Conference
2025.
Tags: ICML
@conference{Zhang2025c,
title = {Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning},
author = {Chi Zhang and Ziying Jia and George Atia and Sihong He and Yue Wang},
year = {2025},
date = {2025-07-13},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Chan-Santiago, Jeffrey A.; Tirupattur, Praveen; Nayak, Gaurav Kumar; Liu, Gaowen; Shah, Mubarak
MGD^3: Mode-Guided Dataset Distillation using Diffusion Models Conference
Forty-Second International Conference on Machine Learning, 2025.
Abstract | Tags: ICML | Links:
@conference{Chan-Santiago2025,
title = {MGD^3: Mode-Guided Dataset Distillation using Diffusion Models},
author = {Jeffrey A. Chan-Santiago and Praveen Tirupattur and Gaurav Kumar Nayak and Gaowen Liu and Mubarak Shah},
url = {https://icml.cc/virtual/2025/poster/45507},
year = {2025},
date = {2025-07-13},
publisher = {Forty-Second International Conference on Machine Learning},
abstract = {Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment.Recent advances have leveraged generative models to distill datasets by capturing the underlying data distribution. Unfortunately, existing methods require model fine-tuning with distillation losses to encourage diversity and representativeness. However, these methods do not guarantee sample diversity, limiting their performance.We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diversity in three stages: Mode Discovery to identify distinct data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to mitigate artifacts in synthetic samples that affect performance.We evaluate our approach on ImageNette, ImageIDC, ImageNet-100, and ImageNet-1K, achieving accuracy improvements of 4.4%, 2.9%, 1.6%, and 1.6%, respectively, over state-of-the-art methods. Our method eliminates the need for fine-tuning diffusion models with distillation losses, significantly reducing computational costs.
},
keywords = {ICML},
pubstate = {published},
tppubtype = {conference}
}
Dataset distillation has emerged as an effective strategy, significantly reducing training costs and facilitating more efficient model deployment.Recent advances have leveraged generative models to distill datasets by capturing the underlying data distribution. Unfortunately, existing methods require model fine-tuning with distillation losses to encourage diversity and representativeness. However, these methods do not guarantee sample diversity, limiting their performance.We propose a mode-guided diffusion model leveraging a pre-trained diffusion model without the need to fine-tune with distillation losses. Our approach addresses dataset diversity in three stages: Mode Discovery to identify distinct data modes, Mode Guidance to enhance intra-class diversity, and Stop Guidance to mitigate artifacts in synthetic samples that affect performance.We evaluate our approach on ImageNette, ImageIDC, ImageNet-100, and ImageNet-1K, achieving accuracy improvements of 4.4%, 2.9%, 1.6%, and 1.6%, respectively, over state-of-the-art methods. Our method eliminates the need for fine-tuning diffusion models with distillation losses, significantly reducing computational costs.