Problem Set 5: Projections and A=QR

Problem Set 5: Projections and $A=QR$#

Section 1: Foundations (Span, Orthogonality, and Orthogonal Matrices)#

Problem 1. Consider the vectors $\mathbf{v}_1 = \begin{bmatrix} 1 \\ 2 \\ 0 \end{bmatrix}$ and $\mathbf{v}_2 = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}$ in $\mathbb{R}^3$.

(a) Describe geometrically what $\text{span}\{\mathbf{v}_1, \mathbf{v}_2\}$ represents.

(b) Determine whether $\mathbf{w} = \begin{bmatrix} 2 \\ 5 \\ 1 \end{bmatrix}$ is in $\text{span}\{\mathbf{v}_1, \mathbf{v}_2\}$.

(c) Find a vector in $\mathbb{R}^3$ that is NOT in $\text{span}\{\mathbf{v}_1, \mathbf{v}_2\}$ and explain why.

Problem 2. Let $\mathbf{u} = \begin{bmatrix} 1 \\ -1 \\ 2 \end{bmatrix}$ and $\mathbf{v} = \begin{bmatrix} 2 \\ 2 \\ a \end{bmatrix}$.

(a) Find the value of $a$ such that $\mathbf{u}$ and $\mathbf{v}$ are orthogonal.

(b) Verify your answer by computing $\mathbf{u} \cdot \mathbf{v}$.

(c) If $\mathbf{u}$ and $\mathbf{v}$ are orthogonal, what can you say about the angle between them?

Problem 3. Determine which of the following sets of vectors are orthogonal, and which are orthonormal:

(a) $\left\{\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}\right\}$

(b) $\left\{\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 0 \\ 2 \end{bmatrix}\right\}$

(c) $\left\{\begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \\ 0 \end{bmatrix}, \begin{bmatrix} \frac{1}{\sqrt{2}} \\ -\frac{1}{\sqrt{2}} \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}\right\}$

Problem 4. Let $Q = \begin{bmatrix} \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \end{bmatrix}$.

(a) Verify that $Q$ is an orthogonal matrix by showing that $Q^TQ = I$.

(b) Compute $Q^{-1}$.

(c) Show that $Q$ preserves lengths: for $\mathbf{x} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$, verify that $\|\mathbf{x}\| = \|Q\mathbf{x}\|$.

(d) Make up a vector $\mathbf{x}$. Plot $\mathbf{x}$ and $Q\mathbf{x}$. How are they related?

Problem 5. Consider the matrix $Q = \begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix}$.

(a) Prove that $Q$ is an orthogonal matrix for any value of $\theta$.

(b) What geometric transformation does this matrix represent?

(c) Using $\theta = \frac{\pi}{4}$, compute $Q\begin{bmatrix} 1 \\ 0 \\ 2 \end{bmatrix}$.

Section 2: Projections (Warm-up)#

Problem 6. Let $\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $\mathbf{u} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$.

(a) Find the projection of $\mathbf{v}$ onto $\mathbf{u}$.

(b) Find the component of $\mathbf{v}$ orthogonal to $\mathbf{u}$.

(c) Verify that these two components are orthogonal and sum to $\mathbf{v}$.

Problem 7. Find the projection of $\mathbf{b} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}$ onto $\mathbf{a} = \begin{bmatrix} 2 \\ -1 \\ 1 \end{bmatrix}$.

Problem 8. Let $L$ be the line in $\mathbb{R}^3$ spanned by $\mathbf{v} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}$.

(a) Find the projection matrix $P$ that projects vectors onto $L$.

(b) Verify that $P^2 = P$ (i.e., $P$ is idempotent).

(c) Use your matrix to project $\mathbf{w} = \begin{bmatrix} 6 \\ 0 \\ 3 \end{bmatrix}$ onto $L$.

Section 3: Projections (Intermediate)#

Problem 9. Let $W$ be the subspace of $\mathbb{R}^3$ spanned by $\mathbf{u}_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$ and $\mathbf{u}_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$ (the $xy$-plane).

(a) Find the projection matrix $P$ onto $W$.

(b) Find the projection of $\mathbf{v} = \begin{bmatrix} 2 \\ 3 \\ 5 \end{bmatrix}$ onto $W$.

(c) What is the geometric meaning of $I - P$?

Problem 10. Consider the plane in $\mathbb{R}^3$ given by the equation $x + y + z = 0$.

(a) Find an orthonormal basis for this plane.

(b) Construct the projection matrix onto this plane.

(c) Project $\mathbf{v} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}$ onto the plane.

Section 4: Gram-Schmidt Orthogonalization (Basic)#

Problem 11. Apply the Gram-Schmidt process to orthogonalize the following vectors in $\mathbb{R}^3$:

\[\begin{split} \mathbf{a}_1 = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}, \quad \mathbf{a}_2 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} \end{split}\]

Find an orthonormal basis $\{\mathbf{q}_1, \mathbf{q}_2\}$ for the subspace spanned by these vectors.

Problem 12. Use Gram-Schmidt to orthogonalize the vectors:

\[\begin{split} \mathbf{v}_1 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}, \quad \mathbf{v}_2 = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}, \quad \mathbf{v}_3 = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} \end{split}\]

Normalize your results to obtain an orthonormal basis for $\mathbb{R}^3$.

Section 5: Gram-Schmidt Orthogonalization (Intermediate)#

Problem 13. Consider the vectors in $\mathbb{R}^4$:

\[\begin{split} \mathbf{a}_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}, \quad \mathbf{a}_2 = \begin{bmatrix} 1 \\ 1 \\ 0 \\ 0 \end{bmatrix}, \quad \mathbf{a}_3 = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 0 \end{bmatrix} \end{split}\]

(a) Apply Gram-Schmidt to find an orthogonal basis $\{\mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3\}$.

(b) Normalize to obtain an orthonormal basis $\{\mathbf{q}_1, \mathbf{q}_2, \mathbf{q}_3\}$.

(c) Express $\mathbf{a}_2$ as a linear combination of $\mathbf{q}_1$ and $\mathbf{q}_2$.

Problem 14. Given the linearly independent vectors:

\[\begin{split} \mathbf{w}_1 = \begin{bmatrix} 2 \\ 0 \\ 1 \end{bmatrix}, \quad \mathbf{w}_2 = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \end{split}\]

(a) Use Gram-Schmidt to find an orthonormal basis for $\text{span}\{\mathbf{w}_1, \mathbf{w}_2\}$.

(b) Find the $QR$ factorization of the matrix $A = \begin{bmatrix} 2 & 1 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}$ whose columns are $\mathbf{w}_1$ and $\mathbf{w}_2$.

Section 6: Advanced Problems#

Problem 15. Let $V$ be the subspace of $\mathbb{R}^4$ spanned by:

\[\begin{split} \mathbf{v}_1 = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix}, \quad \mathbf{v}_2 = \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}, \quad \mathbf{v}_3 = \begin{bmatrix} 1 \\ 3 \\ 6 \\ 10 \end{bmatrix} \end{split}\]

(a) Apply Gram-Schmidt to find an orthonormal basis for $V$.

(b) Construct the projection matrix $P$ onto $V$.

(c) Find the closest point in $V$ to $\mathbf{b} = \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \end{bmatrix}$.

Problem 16. Suppose $\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_k$ are orthonormal vectors in $\mathbb{R}^n$.

(a) Prove that the projection matrix onto $W = \text{span}\{\mathbf{u}_1, \ldots, \mathbf{u}_k\}$ is given by:

\[ P = \mathbf{u}_1\mathbf{u}_1^T + \mathbf{u}_2\mathbf{u}_2^T + \cdots + \mathbf{u}_k\mathbf{u}_k^T \]

(b) Show that $P$ is symmetric and satisfies $P^2 = P$.

(c) Prove that for any $\mathbf{v} \in \mathbb{R}^n$, the vector $\mathbf{v} - P\mathbf{v}$ is orthogonal to $W$.

Problem 17. Let $A$ be an $m \times n$ matrix with linearly independent columns $\mathbf{a}_1, \ldots, \mathbf{a}_n$.

(a) Explain why the projection of $\mathbf{b} \in \mathbb{R}^m$ onto $\text{Col}(A)$ is given by:

\[ \text{proj}_{\text{Col}(A)}\mathbf{b} = A(A^TA)^{-1}A^T\mathbf{b} \]

(b) If $A = QR$ is the $QR$ factorization (where $Q$ has orthonormal columns), show that the projection simplifies to: $$ \text{proj}_{\text{Col}(A)}\mathbf{b} = QQ^T\mathbf{b} $$

(c) Apply this to find the projection of $\mathbf{b} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}$ onto the column space of:

\[\begin{split} A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \\ 0 & 0 \end{bmatrix} \end{split}\]