Error analysis of the Gram low-rank approximation (and why it is not as unstable as one may think)
Résumé
Abstract. Given \(A\in \mathbb {R}^{m\times n}\) and its singular value decomposition (SVD) \(U\Sigma V^T\), the eigenvalue decomposition (EVD) of the Gram matrix \(G=A^TA\) is \(V\Sigma^2 V^T\). When \(m\gg n\), it is computationally attractive to compute the truncated SVD of \(A\) from the truncated EVD of \(G\). This idea has in particular been used to efficiently compress low-rank tensors. In finite precision arithmetic, however, there is a good reason to fear instability from this approach, since the Gram matrix \(G\) has condition number \(\kappa (A)^2\). We carry out an error analysis of this approach that uses eigenvector perturbation theory. We first explain that a naive application of standard results from this theory leads to an error bound proportional to \(\kappa (\bar {A})^2u\), where \(u\) is the machine precision, \(\bar {A}\) is the matrix truncated to the target rank \(k\), and \(\kappa (\bar {A})=\sigma\_1/\sigma\_k\) is its generalized condition number. Importantly, this bound is pessimistic and we prove that we can significantly improve it with a more careful analysis. Specifically, we obtain two improvements: First, we show that the error bound is at most proportional to \(\kappa (\bar {A})u\), instead of \(\kappa (\bar {A})^2u\). Second, we show that regardless of how large \(\kappa (\bar {A})\) is, the error cannot exceed a constant multiple of \(\sqrt {u}\). Hence our final bound is of order \(\min (\kappa (\bar {A})u, \sqrt {u})\). Moreover, we also propose the use of iterative refinement to further improve the accuracy in some cases. We illustrate the unusual and attractive behavior of this algorithm with numerical experiments that showcase its effectiveness, despite its partial instability. We believe that our results explain the success that this approach has encountered in large scale tensor computations.
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|---|
| licence |
