Due Friday, March 11, at 11am.
The SVD of an $m \times n$ matrix $A$ of rank $r$, from class, is $$A = U\Sigma V^T,$$ where $U$ is an $m \times r$ matrix with orthonormal columns (the "best" orthonormal basis for $C(A)$), $\Sigma$ is an $r \times r$ diagonal matrix of singular values $\sigma_k$, and $V$ is an $n \times r$ matrix with orthonormal columns (the "best" orthonormal basis for the row space $C(A^T)$).
Suppose that $r = n$ (i.e. $A$ has full column rank).
(a) $(V^T)^{-1}$ simplifies to ______________ because $V$ in this case is ______________?
(b) Show that the matrix $B = (A^T A)^{-1}A^T$ has an especially simple form in terms of the SVD factors $U,\Sigma,V$.
(c) $B$ is even simpler if $r = n = m$: what is it?
(This matrix $B$ plays an important role in least-squares problems, as we shall see.
Suppose that $v_1, \ldots, v_r$ are any basis for $C(A^T)$ (not necessarily orthogonal as in the SVD), where $A$ is an $m \times n$ matrix of rank $r$.
The vectors $Av_1, \ldots, Av_r$ are not necessarily orthogonal either, but you can show that they must be linearly independent and hence a basis for $C(A)$. Fill in the blanks of the following proof by contradiction:
If $Av_1, \ldots, Av_r$ were linearly dependent, that would mean that $c_1 Av_1 + \cdots + c_rAv_r = 0$ for some coefficients $\vec{c} \ne 0$. But that would require $c_1 v_1 + \cdots + c_r v_r$ to be in the _______________ space of $A$, which means that $c_1 v_1 + \cdots + c_r v_r = \_\_\_$ because the _______________ space is _______________ to $v_1, \ldots, v_r \in C(A^T)$. But it's impossible to have $c_1 v_1 + \cdots + c_r v_r = \_\_\_$ for $\vec{c} \ne 0$ because the vectors $v_1, \ldots, v_r$ are _______________.
Let $A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{pmatrix}$ and $b = \begin{pmatrix} 2 \\ 3 \\ 4 \end{pmatrix}$.
(a) Compute (by hand) the orthogonal projection of $b$ onto the _____-dimensional column space $C(A)$ using the projection matrix $P = A(A^TA)^{-1}A^T$ derived in class. (As usual, you should be able to multiply $P$ by a vector without explicitly computing any matrix inverses.)
(b) Your answer from (a) could have been obtained by no computation at all, just using geometric reasoning: $C(A)$ in this case is simply the subspace of $\mathbb{R}^3$ consisting of _______________, and hence the orthgonal projection of any vector onto this subspace is simply _______________.
(c) Compute $x = (A^T A)^{-1} A^T b$ in Julia (see code below). What is the relationship between $x$ and your answer from (a)?
(d) Compute A \ b
in Julia. $A$ is non-square/non-invertible, so this can't be $A^{-1} x$ anymore. What is it?
(e) In terms of the SVD $A = U\Sigma V^T$ computed in Julia with U,Σ,V = svd(A)
(see code below), how does $UU^Tb$ (computed in Julia) compare to your answers from the previous parts? Why is this true in general (for any $A$)?
A = [1 1; 0 1; 0 0]
b = [2, 3, 4]
x = (A'A)\(A'b)
using LinearAlgebra # for the svd(A) function
U,Σ,V = svd(A)
U*U'*b
(From Strang 4.2, problem 10.)
Project $a_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$ onto the line spanned by $a_2 = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$. Then project the result back onto the line spanned by $a_1$. Multiply these projection matrices $P_1 P_2$: is this a projection?
(Extension of Strang 4.2 problem 19.)
(a) Find the projection matrix $P$ onto the plane $x-y-2z=0$: choose two vectors in that plane (the null space of what matrix?) and make them columns of $A$ so that the plane is $C(A)$. Then compute (by hand) the projection of the point $\begin{pmatrix} 0 \\ 6 \\ 12 \end{pmatrix}$ onto this plane.
(b) Alternatively, $P = I - \mbox{(rank-1 matrix)}$ for the rank-1 projection matrix _________ because the plane is the orthogonal complement of _________.
(From Strang, section 4.2, problem 30.)
(a) Find the projection matrix $P_C$ onto the column space $C(A)$ (after looking closely at the matrix!) for $A = \begin{pmatrix} 3 & 6 & 6 \\ 4 & 8 & 8 \end{pmatrix}$.
(b) Find the 3 by 3 projection matrix $P_R$ onto the row space of $A$. (There is a simple way to do it if you realize that the row space is _____
-dimensional.) Multiply $B = P_C A P_R$. Your answer $B$ may be a little surprising at first — can you explain it?