Lecture 9 (Mar 8)

Def ($N$ singular values of $A$): length of the $N$ principal semiaxes of $AS$, i.e. $\sigma_1, \sigma_2, \cdots, \sigma_N$ such that $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_N >0$.

Def ($N$ left singular vectors of $A$): $\{ u_1, u_2, \cdots, u_N \}$ orthonormal, principal semiaxes directions.

Def ($N$ right singular vectors of $A$): $\{ v_1, v_2, \cdots, v_N \}$, preimage of $\sigma_i u_i$. It turns out these vectors are orthonormal.

Reduced SVD:
$$
A=\hat{U} \hat{\Sigma} V^*
$$
where $\hat{U}\in\mathbb{R}^{M\times N}$, $\hat{\Sigma}\in\mathbb{R}^{N\times N}$ and $V\in\mathbb{R}^{N\times N}$.

Full SVD

$$
A=U \Sigma V^*
$$
where $U\in\mathbb{R}^{M\times M}$, $\Sigma\in\mathbb{R}^{M\times N}$ and $V\in\mathbb{R}^{N\times N}$.

Properties:
1. $v_i$: eigenvectors of $A^*A$.
2. $u_i$: eigenvectors of $AA^*$.
3. Nonzero singular values are the square roots of the nonzero eigenvalues of $A^*A$ or $AA^*$.

4. Rank$(A)=r=$ the number of nonzero singular values.
5. Range$(A)=$span$(u_1,u_2,\cdots,u_r)$.
6. Null$(A)=$span$(v_{r+1},v_{r+2},\cdots,v_N)$.
7. $A$ is the sum of $r$ rank-one matrices,
$$
A=\sum_{i=1}^r \sigma_i u_i v_i^* \, .
$$

Low rank approximation of $A$

For any $\nu$ such that $0\le \nu \le r$, define
$$
A_{\nu}=\sum_{i=1}^{\nu} \sigma_i u_i v_i^*
$$
is an rank-$\nu$ approximation of $A$.

Image Compression using SVD

Vector norm

A function $\| \cdot \|: \mathbb{R}^m \rightarrow \mathbb{R}$ such that
(1) $\|x\| \ge 0$, $\| x\|=0$ only if $x=0$;
(2) $\|x+y\| \ge \|x\| + \|y\|$;
(3) $\|\alpha x\| = |\alpha| \cdot \|x\|$.

Examples: $\|x\|_1$, $\|x\|_2$, $\|x\|_p$, $\|x\|_{\infty}$.

Matrix norm induced by vector norm

Def: $\|A\|$: the smallest $C\in \mathbb{R}$ such that
$$
\|A x\| \le C \|x\|
$$
for all $x\in \mathbb{R}^m$. Or,
$$
\|A\| = \mbox{sup}_{x\in \mathbb{R}^m, x\ne 0} \frac{ \|Ax\|}{\|x\|} = \mbox{sup}_{\|x\|=1, x \in \mathbb{R}^m} \|Ax\| \, .
$$

Reading materials

Lecture notes: p.56-60.