• The projection matrix is \(P=A(A^TA)^{-1}A^T\)

  • If you project \(b\) and \(b\) is in the column space, then \(Pb=b\):

    \[Pb=A(A^TA)^{-1}A^Tb\] \[\text{note: }b = Ax\] \[Pb=A(A^TA)^{-1}A^TAx\] \[Pb=A\frac{A^TA}{A^TA}x\] \[Pb=Ax\] \[Pb=b\]

  • If you project \(b\) and \(b\) is perpendicular to the column space, then \(Pb=0\) (note, \(b\) is in the nullspace of \(A^T\) because it is perpendicular to each column of \(A\)):

    \[Pb=A(A^TA)^{-1}A^Tb\] \[\text{note: }A^Tb = 0\] \[Pb=A(A^TA)^{-1}0\] \[Pb=0\]

  • When we project \(b\) into the columnspace of \(A\), the projection \(p=Pb\) plus the error \(e\) results in \(b\) (\(p+e=b\)). Note that we are also projecting \(b\) into the left nullspace to find \(e\). Thus \(e=(I-P)b\). It is not clear to me why the \(I-P\) appears other than:

    \[p+e = Pb + (I-P)b = Pb + Ib - Pb = Ib = b\]

Least squares

  • Find the best straight line \(y=C+Dt\) through the points (1,1), (2,2), and (3,2).

  • Each point defines a linear equation:

    \[C + D = 1\] \[C + 2D = 2\] \[C + 3D = 2\]

  • This translates to matrix form:

    \[\begin{bmatrix} \begin{array}{rr} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ \end{array} \end{bmatrix} * \begin{bmatrix} \begin{array}{r} C \\ D \\ \end{array} \end{bmatrix} = \begin{bmatrix} \begin{array}{r} 1 \\ 2 \\ 2 \\ \end{array} \end{bmatrix}\]

  • What are we minimizing when we try to find the best possible projection of \(b\) into the columnspace. Minimize \(\lvert Ax-b\rvert^2=\lvert e\rvert^2\). The error is how far each point lies from the line.

  • For the least squares method of calculating the error, outliers can unduly influence the result.

  • Pretty good discussion of the projections vs the original points at 17:15. Essentially we project all the points in our dataset onto a line. We choose the line such that the line minimizes the total error squared.

  • The goal of least squares: Find \(\hat{x} = \begin{bmatrix} \hat{C} \\ \hat{D} \end{bmatrix}\) and \(P\)

    \[A^TA\hat{x} = A^Tb\] \[A^TA = \begin{bmatrix} \begin{array}{rrr} 1 & 1 & 1\\ 1 & 2 & 3\\ \end{array} \end{bmatrix} * \begin{bmatrix} \begin{array}{rr} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ \end{array} \end{bmatrix} = \begin{bmatrix} \begin{array}{rr} 3 & 6 \\ 6 & 14 \\ \end{array} \end{bmatrix}\] \[A^TAb = \begin{bmatrix} \begin{array}{rrr} 1 & 1 & 1\\ 1 & 2 & 3\\ \end{array} \end{bmatrix} * \begin{bmatrix} \begin{array}{rrr} 1 & 1 & 1\\ 1 & 2 & 2\\ 1 & 3 & 2\\ \end{array} \end{bmatrix} = \begin{bmatrix} \begin{array}{rrr} 3 & 6 & 5\\ 6 & 14 & 11\\ \end{array} \end{bmatrix}\]

  • Normal equations:

    \[3C + 6D = 5\] \[6C + 14D = 11\]

  • Solution: \(D=\frac{1}{2}\) and \(C=\frac{2}{3}\).

  • 30:00 - good discussion of the error vector and how it relates to the matrix world. He shows how \(b=p+e\) in this example.

    \[b = p+e\] \[\begin{bmatrix} \begin{array}{r} 1 \\ 2 \\ 2 \\ \end{array} \end{bmatrix} = \begin{bmatrix} \begin{array}{r} \frac{7}{6} \\ \frac{10}{6} \\ \frac{13}{6} \\ \end{array} \end{bmatrix} + \begin{bmatrix} \begin{array}{r} \frac{-1}{6} \\ \frac{2}{6} \\ \frac{-1}{6} \\ \end{array} \end{bmatrix}\]

  • Note \(e\) is perpendicular to all vectors in the columnspace.

Back to Linear Algebra

  • If \(A\) has independent columns, then \(A^TA\) is invertible.

  • How to prove?

    \[\text{Suppose } A^TAx=0 \text{ then prove } x=0\] \[x^TA^TAx=0\] \[(Ax)^T(Ax) = 0\] \[Ax=0\] \[A \text{ has independent columns and } Ax=0 \text{ then } x=0\]

  • Columns are definitely independent if they are perpendicular unit vectors (orthonormal vectors).

TODO

  • What is the projection matrix for \(b\) into the left nullspace of \(A\)?