# Singular Value Decomposition
Singular Value Decomposition (SVD) is a matrix factorization method that is applicable to all matrices, including rectangular ones (unlike Eigendecomposition, for example).
![[SVD visualization.png]]
SVD of $A=U \Sigma V^{T}$
$U$ is a $|\mathscr{U}| \times n$ matrix of left singular vectors
$\mathrm{V}$ is a $|\mathcal{F}| \times n$ matrix of right singular vectors
$\Sigma$ is a $n \times n$ diagonal matrix of singular values
$
\begin{aligned}
A=U \Sigma V^{\mathbf{T}}=\left[\begin{array}{ccc}
1 & \mid & \mid \\
\boldsymbol{u}_1 & \boldsymbol{u}_2 & \boldsymbol{u}_3 \\
\mid & \mid & \mid
\end{array}\right]\left[\begin{array}{cc}
\sigma_1 & \\
& \sigma_2 \\
&
\end{array}\right]\left[\begin{array}{c}
-\boldsymbol{v}_1^{\mathbf{T}}- \\
-\boldsymbol{v}_2^{\mathbf{T}}-
\end{array}\right] & =\sigma_1\left[\begin{array}{c}
\mid \\
\boldsymbol{u}_1 \\
\mid
\end{array}\right]\left[-\boldsymbol{v}_1^{\mathbf{T}}-\right]+\sigma_2\left[\begin{array}{c}
\mid \\
\boldsymbol{u}_2 \\
\mid
\end{array}\right]\left[-\boldsymbol{v}_2^{\mathbf{T}}-\right] \\
& =\sigma_1 \boldsymbol{u}_1 \boldsymbol{v}_1^{\mathbf{T}}+\sigma_2 \boldsymbol{u}_2 \boldsymbol{v}_2^{\mathbf{T}}
\end{aligned}
$
## Low Rank
$
A_k=U D_k V^T, \quad D_k=\operatorname{diag}\left(\sigma_1, \ldots, \sigma_k, 0, \ldots, 0\right) .
$
Truncating the matrix A $k<r$ terms gives the best rank- $k$ approximation to $A$ in both the 2 -norm and the Frobenius norm.
## Eigen values
Any $n \times m$ matrix $\mathbf{A}$ can be written as
$
\mathbf{A}=\mathbf{U \Sigma V}^T,
$
where
$
\begin{array}{ll}
\mathbf{U}=\text { eigenvectors of } \mathbf{A A}^T & n \times n \\
\Sigma=\sqrt{\operatorname{diag}\left(\operatorname{eig}\left(\mathbf{A} \mathbf{A}^T\right)\right)} & n \times m \\
\mathbf{V}=\text { eigenvectors of } \mathbf{A}^T \mathbf{A} & m \times m
\end{array}
$
---
## References
1. The Art of Linear Algebra, Kenji Hirannbe 2021
2. https://www.cs.princeton.edu/~smattw/Teaching/Fa19Lectures/lec11/lec11.pdf