Federated Machine Learning with Resource-Constrained Devices

Federated learning (FL) is a recently introduced machine learning (ML) training scheme, in which devices train locally with their data but exchange knowledge through the parameters of the trained neural network (NN) model. The exchanged parameters are aggregated centrally, forming a global NN model. Compared to centralized training of NNs, FL improves privacy as it does not require sending raw user data to a centralized entity for training. However, while privacy is improved, FL imposes the resource-intensive training task on typically constrained edge devices, such as internet of things (IoT) devices or smartphones. These devices often vary in their ability to perform training on the NN due to differences in the communication channel, computational capabilities, or memory availability. Training FL systems remains a major challenge due to the heterogeneous nature of devices, as current production deployments often exclude devices that do not meet the required capabilities from the FL training process.
This dissertation presents several techniques that allow constrained devices to participate in FL training, enhancing the effectiveness of FL systems and increasing fairness among participants. ... mehrThis dissertation examines in detail common devices and how their capabilities impact their ability to perform training. Further, the implications of these constraints for participation in FL are discussed, as well as the extent to which existing techniques tackle these problems. Importantly, the issue of interdependencies between specific types of data distributions (non-independent and identically distributed (non-iid)) present on devices and resource heterogeneity among devices is explored.
To combat these challenges, this dissertation introduces a set of techniques. First, a technique is introduced that enables constrained devices to participate in synchronous FL despite round-varying resource constraints. This is achieved by allowing devices to partially freeze and quantize the NN during a round of training, enabling them to adjust to communication, computation, and peak memory constraints independently by modifying the specific layer freezing setup. Furthermore, this technique is the first to evaluate the existing interdependencies between resource heterogeneity and data distribution heterogeneity (non-iid), demonstrating that the proposed algorithm promotes high fairness among devices despite differences in resources, which state-of-the-art solutions fail to provide.
The technique is extended to incorporate centrally available existing pre-training datasets. As a consequence of having a pre-trained NN, not all of its layers require federated training. An algorithm is presented that selects a suitable NN structure, trainable by all heterogeneous devices, while maximizing achievable accuracy.
Second, to tackle fixed peak memory constraints, a technique is proposed that allows the NN model to grow step by step while freezing early layers. This enables constrained devices in FL to train a large global FL model that no device is capable of training end-to-end on its own. Communication and computation should thereby be used efficiently; however, they are not treated as per-round constraints. While adhering to memory constraints, FL training is significantly accelerated and achieves higher accuracies compared to existing works and end-to-end training of feasible smaller models.
Third, a hardware-design-based technique is introduced that tackles heterogeneity at design time, specifically energy, by proposing designs for resource-specific ASIC-based accelerators. These accelerators utilize compressed arithmetic formats and approximate computing in convolutional and linear operations, significantly reducing the energy overhead of computations and memory accesses for constrained devices, with minimal impact on accuracy. The evaluation shows that approximate computing principles are suitable for FL, enabling constrained devices to participate and ensuring their training impacts the global FL model.
Ultimately, the contributions of this dissertation enhance the practical deployment of FL in real-world scenarios, improving FL’s resilience in heterogeneous settings. The proposed techniques enable resource-constrained devices to meaningfully participate in FL, ensuring their knowledge is effectively integrated into the global FL model—an aspect where previous works have fallen short. This promotes a fairer FL framework, thereby improving the quality of services for a broader range of devices and users.