Tian Li et al. Federated Learning Challenges, Methods, and Future Directions

• Federated Learning: Challenges, Methods, and Future Directions

Intro

• Edge Computing is getting big
  • Fog Computing
• Recent trend: Train centrally, Inference locally
• FL challenges this (see also Qiang Yang et al.)

Challenges

• Expensive communication
  • Reduce comms round
  • Reduce comms size
• System Heterogeneity
  • Everyone uses different phones
• Statistical Heterogeneity
• Privacy Concerns

Communication Efficiency in Federated Networks

Communication is a primary bottleneck in federated learning.

Local Updating

• Mini-batch optimization methods are used for distributed learning.
• Newer methods focus on flexible local updating, so each device does more work on its own data.
• In federated settings, FedAvg is a commonly used method for local updating.
• Federated Averaging (FedAvg) works well in practice but does not always converge.

Compression Schemes

• Model compression schemes like sparsification, subsampling, and quantization reduce message size.
• In federated settings these get harder, because which devices show up changes every round.

Decentralized Training

• Federated learning often uses a star network topology.
• Decentralized topologies may reduce central server communication costs.
• Hierarchical communication uses edge servers to aggregate data first.

Systems Heterogeneity in Federated Settings

Devices in federated settings vary a lot in hardware, network, and battery power.

Asynchronous Communication

• Asynchronous schemes mitigate stragglers but face challenges in federated settings with unpredictable delays.

Active Sampling

• Only a subset of devices participate in each training round.
• Active selection of devices can influence outcomes.
• Methods can sample devices based on system resources or data quality.

Fault Tolerance

• Essential in federated settings due to potential device dropouts.
• Coded computation introduces redundancy to tolerate failures.

Statistical Heterogeneity in Federated Models

When data is not distributed the same way across devices, federated models get harder. It changes both how we model the data and how we analyze whether training converges.

Modeling Heterogeneous Data

Literature Backgrounds

A lot of research models statistical heterogeneity with meta-learning and multi-task learning, and recently these ideas moved into federated learning.

• MOCHA. Learns a separate model for each device for personalization, but keeps a shared representation through multi-task learning.
• Bayesian Networks. Models the star topology and runs variational inference.
• Meta-learning. Sets a within-task learning rate from multi-task information.
• Transfer Learning. Personalizes starting from a trained global model.

Challenges

Making these methods robust, scalable, and automated is still hard.

Fairness in Federated Data

Fairness matters separately from accuracy. Some devices may get favored because they have more data or show up more often.

Modified Modeling

Some approaches aim to reduce model performance variance across devices.

Convergence Guarantees for Non-IID Data

Challenge

Non-IID (independent and identically distributed) data causes difficulties in analyzing convergence behavior in federated settings.

• FedAvg. Has been seen to diverge in practice.
• FedProx. A proposed change to FedAvg meant to make it converge.

Other Approaches

Some approaches deal with statistical heterogeneity by sharing local device data or server-side proxy data. That can hurt privacy.

Privacy in Federated Learning

Privacy is the whole point of federated settings, since raw data stays on each device. But model updates shared during training can still leak sensitive information.

Privacy in General Machine Learning

Strategies

• Differential Privacy. Popular for its strong guarantees, simplicity, and small systems overhead. A small change in the input can't change the output distribution much.
• Homomorphic Encryption. Lets you compute on encrypted data.
• Secure Multiparty Computation (SMC). Several parties compute a function together without leaking their inputs.

Trade-offs

Privacy trades off against accuracy. Adding noise helps privacy but can cost accuracy.

Privacy in Federated Learning

Challenges

Federated learning needs methods that are cheap, communicate efficiently, and survive device dropouts, all without losing accuracy.

• Privacy Definitions. Global privacy (from outside third parties) and local privacy (from the central server).

Approaches

• Secure Multi-Party Computation (SMC). Protects individual model updates.
• Differential Privacy. Applied to federated learning, it gives global differential privacy. Some methods add adaptive gradient clipping for better performance.
• Local Privacy. A relaxed version that performs better than strict local privacy.
• Combining Methods. Differential privacy plus model compression helps both privacy and communication.

Future Directions and Themes

Communication in Federated Learning

Extreme Communication Schemes

• How much communication federated optimization really needs.
• How much precision optimization methods can lose, and whether that helps generalization.
• How one-shot and few-shot heuristics do in federated settings.

Communication Reduction and Pareto Frontier

• Techniques for reducing communication in federated training, e.g., local updating and model compression.
• The trade-off between accuracy and communication.
• Which techniques get the best accuracy under a fixed communication budget.

Novel Models of Asynchrony

• Synchronous versus asynchronous communication in distributed optimization.
• Device-centric communication schemes in federated networks.

Heterogeneity in Federated Learning

Heterogeneity Diagnostics

• Methods to quantify statistical heterogeneity.
• Diagnostics that measure heterogeneity before training.
• Whether heterogeneity can actually help federated optimization converge.

Privacy Concerns

Granular Privacy Constraints

• Local and global privacy definitions in federated networks.
• Methods that respect device-specific or sample-specific privacy constraints.

Extending the Scope of Federated Learning

Beyond Supervised Learning

• Data in federated networks that is unlabeled or weakly labeled.
• Tasks other than model fitting, like exploratory data analysis, aggregate statistics, or reinforcement learning.

Operational Challenges

Productionizing Federated Learning

• Handling practical concerns like concept drift, diurnal variations, and cold start problems.

Standards and Benchmarks

Benchmarks

• Ground federated learning research in real-world settings and datasets.
• Improve existing benchmarking tools so results are reproducible and solutions spread.