Deep Diaries · · 3 min read
Federated Learning for Decentralized Training
Federated Learning enables mobile phones to collaboratively learn a shared prediction model while keeping all the training data on the device.
Originally published on Deep Diaries on Substack. Reproduced here as written.
"Federated Learning enables mobile phones to collaboratively learn a shared prediction model while keeping all the training data on the device, decoupling the ability to do machine learning from the need to store the data in the cloud."
Introduction
Have you ever asked yourself how the AI models on your phone get updated ?!
How do applications like Google assistant or GBoard get updated?
Nowadays machine and deep learning are used in everything in our lives starting from smartphone applications to self-driven cars, and as fast as our lives change these models should be updated to keep pace with the changes in users’ behavior and interactions.
Even if the deployed model was well-trained on real-life data and accomplished the best accuracy ever it should not stop training on the new data to stay updated and effective.
But how could this update happen ?!
Traditional method
Updating a deployed model was done by uploading the user’s data to a centralized server to be used in the training process and then sending an update to the deployed model, regardless of the internet usage for uploading this data, this way violating user privacy by putting his data in risk of being exposed.
The following figure shows the figure of updating the deployed model
A - The client device sends its data to the cloud
B - The cloud group all collected data and sends it for training
C - The model gets updated using the new data
D - The updated model was sent back to the client's devices
As this way could put the users in danger and threaten their privacy this might be uncomfortable for many users which could lead to a decrease in the number of users for an application.
So, how to protect our data?
New updating method
To overcome these problems researchers used the evolution of devices and the increase of resources like CPUs and GPUs to develop a new approach for training on decentralized data called “Federated Learning”.
Federated learning makes use of device resources to train the model on the device with the user’s data without any need to upload this data to the server, this way provided more security for users like the security of the data is now limited to the device’s security and cannot be affected by the cloud security, also reducing the internet usage by only uploading the updated version of model weights to the server to update the general model and then download the updated version of the model.
The following figure shows the full process of federated learning
A - The device uses its data to train the model on the device
B - The local model weights have been submitted to the cloud
C - The cloud groups all collected models and apply “Federated Averaging” and sends the new general model to devices
As you can see the process started with downloading the general model from the cloud to use it on the device, then training this model with the user’s data, and then sending the model weights to the cloud. After that, the cloud uses the submitted local model’s weights and applies federated averaging to generate the new general model and then sends it back to the devices.
What is the “Federated Averaging”?
Federated Averaging means taking the average of the weights concerning the size of the data used to get these weights, the following equation explains how federated averaging works for the weight W (t + 1)
where K is the number of clients, n is the total size of data used for training for all clients and nk is the size of data used for training for user.
This new approach offers a good way to update deployed models concerning data privacy and security but one of the disadvantages of this approach is
Using the user’s device resources to train the model
Making the model vulnerable to a data poisoning attack
It does not guarantee the privacy of the local model weights which can expose some information about the user
But currently, Google is developing a new method to encrypt these weights and ensure its security called “secure aggregation protocol” to overcome the resources usage problem of the model being trained when the device is idle or plugged in and only sends and receives updates on a free wireless connection.
Conclusion
Eventually, training on decentralized data can represent the future of deployed deep learning models even if it has some issues right now, and it could redefine the traditional way of training that we know.
It also can open the door for training bigger models on multiple servers using huge amounts of data to get better results.
References
Federated Learning: Collaborative Machine Learning without Centralized Training Data