Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning

Do Yeon Kim, Dong Jun Han, Jun Seo, Jaekyun Moon

Research output: Contribution to journalConference articlepeer-review

Abstract

Handling the substantial communication burden in federated learning (FL) still remains a significant challenge. Although recent studies have attempted to compress the local gradients to address this issue, they typically perform compression only within the original parameter space, which may potentially limit the fundamental compression rate of the gradient. In this paper, instead of restricting our scope to a fixed traditional space, we consider an alternative space that provides an improved compressibility of the gradient. To this end, we utilize the structures of input activation and output gradient in designing our mapping function to a new space, which enables lossless gradient sparsification, i.e., mapping the gradient to our new space induces a greater number of near-zero elements without any loss of information. In light of this attribute, employing sparsification-based compressors in our new space allows for more aggressive compression with minimal information loss than the baselines. More surprisingly, our model even reaches higher accuracies than the full gradient uploading strategy in some cases, an extra benefit for utilizing the new space. We also theoretically confirm that our approach does not alter the existing, best known convergence rate of FL thanks to the orthogonal transformation properties of our mapping.

Original languageEnglish
Pages (from-to)23867-23900
Number of pages34
JournalProceedings of Machine Learning Research
Volume235
Publication statusPublished - 2024
Event41st International Conference on Machine Learning, ICML 2024 - Vienna, Austria
Duration: 2024 Jul 212024 Jul 27

Bibliographical note

Publisher Copyright:
Copyright 2024 by the author(s)

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability

Fingerprint

Dive into the research topics of 'Achieving Lossless Gradient Sparsification via Mapping to Alternative Space in Federated Learning'. Together they form a unique fingerprint.

Cite this