Refined Analysis of Federated Averaging's Bias and Federated Richardson-Romberg Extrapolation
Résumé
In this paper, we present a novel analysis of FedAvg with constant step size, relying on the Markov property of the underlying process. We demonstrate that the global iterates of the algorithm converge to a stationary distribution and analyze its resulting bias and variance relative to the problem's solution. We provide a first-order bias expansion in both homogeneous and heterogeneous settings. Interestingly, this bias decomposes into two distinct components: one that depends solely on stochastic gradient noise and another on client heterogeneity. Finally, we introduce a new algorithm based on the Richardson-Romberg extrapolation technique to mitigate this bias.
Domaines
Optimisation et contrôle [math.OC]Origine | Fichiers produits par l'(les) auteur(s) |
---|