Federated learning has emerged as a promising approach to training machine learning models while preserving user data privacy. Recent research by Northwestern researcher Fanfei Meng and a team from the Institute of Computing Technology, Chinese Academy of Sciences, has made significant strides in the field of vertical and hybrid federated learning. Their innovative approach, outlined in the paper titled “FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation,” addresses several critical challenges in federated learning, including computational complexity, data privacy, and inference accuracy.
Federated learning, often referred to as collaborative learning, offers a decentralized method for training machine learning models. It eliminates the need for exchanging raw data between client devices and global servers, thus enhancing data privacy. While deep learning-based horizontal federated learning has made significant progress, vertical and hybrid federated learning approaches have been hampered by high computational complexity, typically scaling with the number of clients involved.
The core innovation of the team’s research lies in the concept of incremental learning, which allows deep networks to preserve the feature distribution of user data. The team developed a vertical and hybrid federated learning algorithm that focuses on deploying encrypted aggregated feature vectors and partial networks from clients to perform centralized training on servers. The idea of our algorithm is characterized by higher inference accuracy, stronger privacy- preserving properties, and lower client-server communication bandwidth demands as compared with existing work.
The team’s algorithm achieved a significant improvement in inference accuracy of 0.3% to 4.2%. This enhancement is achieved while maintaining limited privacy revealing for local datasets, making it a valuable advancement in real-world applications.And by design, the algorithm ensures that decrypted features are not exposed on the server, enhancing data privacy and security.At the same time,The algorithm minimizes the communication overhead between clients and servers, reducing the burden on network resources. This is crucial for federated learning in scenarios with limited bandwidth or high-latency connections.
One of the standout features of the team’s approach is its fixed time complexity of 2, which remains unaffected by the number of clients involved. This significant reduction in computational complexity makes the algorithm highly scalable and efficient, even in scenarios with a large number of participating clients. The reduction in time complexity, which amounts to an 88.9% improvement over current baseline methods, is a significant achievement.
To enable communication between different feature spaces from various clients, the team introduced the concept of concatenated feature embedding vectors. This approach allows the server to vertically combine all partial networks, incrementally acquiring knowledge of the entire data distribution during global rounds. Importantly, the algorithm eliminates the need for gradient exchange, passing only weights of partial networks and aggregation-level feature embeddings between clients and servers. This streamlined communication process further contributes to its efficiency.
The research results have been published in the “Proceedings on Engineering Sciences” under the title “FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation.” Fanfei Meng, a researcher at Northwestern University, served as the first author of the paper. The collaboration between Northwestern University and the Institute of Computing Technology, Chinese Academy of Sciences, demonstrates the global nature of advancements in federated learning.
This groundbreaking work was made possible through funding from the National Science Foundation, highlighting the importance of public and private investment in advancing privacy-preserving machine learning techniques.
In conclusion, Fanfei Meng and the team’s research in vertical and hybrid federated learning represents a significant leap forward in the field. Their innovative algorithm not only reduces computational complexity and enhances inference accuracy but also prioritizes data privacy, making it a valuable contribution to the evolving landscape of privacy-preserving machine learning. As federated learning continues to gain prominence in various applications, the insights and techniques developed by this research will play a pivotal role in shaping the future of collaborative and privacy-conscious machine learning.
