Roadmap of Federated Learning: from Motivation to Practice

In the second section, we will discuss the current challenges in FL and their corresponding solutions as follows:

1. Data heterogeneity: Clients hold data drawn from different distributions. For example, some specific classes of samples concentrate on few clients, while other clients only hold very limited data of them. Data heterogeneity can often slow down or even prevent model convergence. The typical mitigations of this issue include using a regularization term [5], knowledge distillation [12], model contrastive loss [3], and control-variate correction [4].
2. System heterogeneity: Clients in FL usually differ in computing resources, power, and network quality. It leads to the drop-off and even bias against weaker devices in the system. We may alleviate it by setting up a proper system design [6] or empowering the weaker devices to train a thinner sub-layer of the model [11].
3. Efficiency: FL is a distributed system, which requires considering the communication efficiency. There are some cutting-edge algorithms to cut off the communication overhead by using the prototype training [7] and one-shot FL [15] so that clients and servers only interact with each other in one round of communication.
4. There is still a probability that the malicious user may obtain the original data from the model uploaded by clients [8]. The typical methods used to enhance security include homomorphic encryption [9] and secure aggregation [10].

Timeline

1. Introduction (60 mins)
   1. Motivation of FL
   2. Introduction of FedAvg [1]
   3. Taxonomy of FL (PFL vs global model, Cross-devices vs Cross-silos, and VFL and HFL)
   4. Popular FL software frameworks
2. Challenges (60 mins)
   1. Data heterogeneity [2,3,4,5,12]
   2. System heterogeneity [6, 7, 11]
   3. Efficiency [7, 15]
   4. Privacy [8,9,10]
3. Practical study cases (30 mins)
   1. IoT [13]
   2. Finance [14]
   3. Healthcare (cross-silos with multi-modal case)

References

[1] McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Artificial intelligence and statistics. PMLR, 2017: 1273-1282.
[2] Zhao Y, Li M, Lai L, et al. Federated learning with non-iid data[J]. arXiv preprint arXiv:1806.00582, 2018.
[3] Li Q, He B, Song D. Model-contrastive federated learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 10713-10722.
[4] Karimireddy S P, Kale S, Mohri M, et al. Scaffold: Stochastic controlled averaging for federated learning[C]//International conference on machine learning. PMLR, 2020: 5132-5143.
[5] Li T, Sahu A K, Zaheer M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine learning and systems, 2020, 2: 429-450.
[6] Bonawitz K, Eichner H, Grieskamp W, et al. Towards federated learning at scale: System design[J]. Proceedings of machine learning and systems, 2019, 1: 374-388.
[7] Tan Y, Long G, Liu L, et al. Fedproto: Federated prototype learning across heterogeneous clients[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(8): 8432-8440.
[8] Zhu L, Liu Z, Han S. Deep leakage from gradients[J]. Advances in neural information processing systems, 2019, 32.
[9] Zhang C, Li S, Xia J, et al. {BatchCrypt}: Efficient homomorphic encryption for {Cross-Silo} federated learning[C]//2020 USENIX annual technical conference (USENIX ATC 20). 2020: 493-506.
[10] Bonawitz K, Ivanov V, Kreuter B, et al. Practical secure aggregation for privacy-preserving machine learning[C]//proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017: 1175-1191.
[11] Diao E, Ding J, Tarokh V. Heterofl: Computation and communication efficient federated learning for heterogeneous clients[J]. arXiv preprint arXiv:2010.01264, 2020.
[12] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning[C]//International conference on machine learning. PMLR, 2021: 12878-12889.
[13] Wu J, Dong F, Leung H, et al. Topology-aware federated learning in edge computing: A comprehensive survey[J]. ACM Computing Surveys, 2024, 56(10): 1-41.
[14] Zhang H, Hong J, Dong F, et al. A privacy-preserving hybrid federated learning framework for financial crime detection[J]. arXiv preprint arXiv:2302.03654, 2023.
[15] Jhunjhunwala D, Wang S, Joshi G. Fedfisher: Leveraging fisher information for one-shot federated learning[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2024: 1612-1620.