Othogonality

Table of Contents

• Dot Product
  • Cauchy-Schwarz Inequality
• Orthogonal Vectors
• Orthonormal vectors
  • Fourier Expansion
  • Parseval’s Theorem
• Orthogonal Projections
• Projection over a subspace
• Gram-Schmidt Process
• Orthogonal Matrix
• Orthogonal Complement

Dot Product

$\underline{\text{Definition}}-$ The inner product or the dot product in a vector space $V$ over $\mathbb{R}$ is a map, that for any 2 vectors $u,v\in V$ there is a real number $<u,v>$ such that,

1. $<u,\alpha v+\beta w>=\alpha <u,v>+\beta <u,w>$ (Linearity Property)
2. $<u,v>=<v,u>$ (Symmetric Property)
3. For any $u\in V$, $<u,v>=0$ if and only if $u$ is the zero vector. ()

The vector space for which the inner product $(\cdot )$ is defined is called an inner product vector space.

Suppose the two vectors $u,v\in {\mathbb{R}}^2$ are like this. We can write $u_1,u_2,v_1,v_2$ using trigonometric identities as -

$$\begin{array}{rl}{u}_1=||u||\cos \theta ,& u_2=||u||\sin \theta \\ v_1=||v||\cos (\theta +\varphi ),& v_2=||v||\sin (\theta +\varphi )\end{array}$$

$u\cdot v$ is defined as $u_1{v}_1+u_2{v}_2$. So using the above values for this gives us,

$$\begin{array}{rl}u\cdot v& =u_1{v}_1+u_2{v}_2\\ & =(||u||\cos \theta \cdot ||v||\cos (\theta +\varphi ))+(||u||\sin \theta \cdot ||v||\sin (\theta +\varphi ))\\ & =||u||||v||\cos (\varphi +\theta -\theta )\\ & =\boxed{||u||||v||\cos (\varphi )}\end{array}$$

Cauchy-Schwarz Inequality

$$\begin{array}{rl}|u\cdot v|& =||u||||v||\cos \theta \\ |u\cdot v|& \le ||u||||v||\end{array}$$

Orthogonal Vectors

If $u,v\in V$ are vectors such that $u\cdot v=0$, then we say that $u$ and $v$ are orthogonal to each other.

• The zero vector is orthogonal to all vectors in any vector space.
• If $u,v$ are non-zero vectors, then $u\cdot v=0$ can only happen when the angle between $u,v$ is 90$°$.

---

Let $W^{\perp }$ be a set of all vectors orthogonal to every vector in $W$, where $W$ is a subspace of the inner product vector space $V$.

Let $u,v\in W^{\perp }$ and $w\in V$ -

1. The zero vector is always orthogonal to any vector in any vector space. Thus $W^{\perp }$ contains the zero vector and is non-empty.
2. Because $u\cdot w=0$ and $v\cdot w=0$, we can say

$$\begin{array}{rlrl}& & u\cdot w+v\cdot w& =0\\ & \Rightarrow & (u+v)\cdot w& =0\end{array}$$

This implies that $u+v\in W^{\perp }$ too. Thus $W^{\perp }$ is closed under vector addition. 3. Because $u\cdot w=0$, we can say $\alpha (u\cdot w)=0$. Consequently this would mean that $\alpha u\cdot w=0$, and hence $\alpha u\in W^{\perp }$. Thus $W^{\perp }$ is closed under scalar multiplication.

Hence we can say that $W^{\perp }$ is a vector space too as it is closed under vector addition and scalar multiplication and contains the zero vector.

The vector space $W^{\perp }$ is called the orthogonal complement of the space $W$.

---

Orthonormal vectors

A set of vectors is said to be orthonormal if -

• Each vector in the set is orthogonal to every other vector.
• Each vector has a magnitude of 1.

---

Suppose $s=\{v_1,v_2,\dots ,v_n\}$ is a set of orthonormal vectors, what can we say about the dependence/independence of this set?

The set of vectors is independent only when ${\alpha }_1{v}_1+{\alpha }_2{v}_2+\cdots +{\alpha }_n{v}_n=0$ only when ${\alpha }_i=0\forall i\in [1,n]$. Let,

$$\begin{array}{rlrl}& & {\alpha }_1{v}_1+{\alpha }_2{v}_2+\cdots +{\alpha }_n{v}_n& =0\\ & \Rightarrow & v_1\cdot ({\alpha }_1{v}_1+{\alpha }_2{v}_2+\cdots +{\alpha }_n{v}_n)& =v_1\cdot 0\\ & \Rightarrow & {\alpha }_1(v_1\cdot v_1)+{\alpha }_2(v_1\cdot v_2)+\cdots +{\alpha }_n(v_1\cdot v_n)& =0\\ & \Rightarrow & {\alpha }_1||v_1|{|}^2& =0\\ & \Rightarrow & {\alpha }_1\ast 1& =0\\ & \Rightarrow & {\alpha }_1& =0\end{array}$$

Similarly we can show that ${\alpha }_i=0\forall i\in [1,n]$. Thus we can say that every set of orthonormal vectors is linearly independent and can be chosen as the basis for their vector space.

---

Fourier Expansion

Let $b$ be the basis expansion of any vector. We can get the scalars for each basis vector by doing,

$$\begin{array}{rlrl}& & b& ={\alpha }_1{v}_1+{\alpha }_2{v}_2+\cdots +{\alpha }_n{v}_n\\ & \Rightarrow & v_1\cdot b& =v_1\cdot ({\alpha }_1{v}_1+{\alpha }_2{v}_2+\cdots +{\alpha }_n{v}_n)\\ & \Rightarrow & v_1\cdot b& ={\alpha }_1(v_1\cdot v_1)+{\alpha }_2(v_1\cdot v_2)+\cdots +{\alpha }_n(v_1\cdot v_n)\\ & \Rightarrow & v_1\cdot b& ={\alpha }_1\\ & \Rightarrow & {\alpha }_1& =v_1\cdot b\end{array}$$

Similarly we can find ${\alpha }_i=v_i\cdot b\forall i\in [1,n]$.

Using this knowledge we can rewrite $b$ as,

$$\begin{array}{rl}b& =(b\cdot v_1)v_1+(b\cdot v_2)v_2+\cdots +(b\cdot v_n)v_n\\ & =||b||||v_1||\cos {\theta }_1{v}_1+||b||||v_2||\cos {\theta }_2{v}_2+\cdots +||b||||v_n||\cos {\theta }_n{v}_n\\ & =||b||(\cos {\theta }_1{v}_1+\cos {\theta }_2{v}_2+\cdots +\cos {\theta }_n{v}_n)\\ & =\boxed{||b||\sum _{i=1}^n\cos {\theta }_i{v}_i}\end{array}$$

This is called the Fourier Expansion for $b$.

Parseval’s Theorem

We know that $b={\sum }_{i=1}^n{\alpha }_i{v}_i={\sum }_{i=1}^n(b\cdot v_i)v_i$.

$$\begin{array}{rlrl}& & b& =\sum _{i=1}^n(b\cdot v_i)v_i\\ & \Rightarrow & ||b|{|}^2& =⟨\sum _{i=1}^n(b\cdot v_i)v_i,\sum _{j=1}^n(b\cdot v_j)v_j⟩\\ & \Rightarrow & & =\sum _{i,j}(b\cdot v_i)(b\cdot v_j)<v_i,vj>\\ & \Rightarrow & & =\sum _{i,j}|b\cdot v_i{|}^2\end{array}$$

Orthogonal Projections

$\underline{\text{Definition}}-$ A projection of a vector is the best approximation of a vector in a vector space.

Let $u,v$ be two non-zero vectors in an inner product space $V$. What is the information of $v$ available in the direction of $u$?

We can drop a perpendicular line $v_{\perp }$ onto $ku$ from $v$, where $k$ is some real scalar. By vector addition -

$$\begin{array}{rlrl}& & v& =ku+v_{\perp }\\ & \Rightarrow & v_{\perp }& =v-ku\end{array}$$

As $v_{\perp }$ is orthogonal to $ku$ by design, we can say that it’s orthogonal to $u$ as well. So,

$$\begin{array}{rlrl}& & v_{\perp }\cdot u& =0\\ & \Rightarrow & (v-ku)\cdot u& =0\\ & \Rightarrow & v\cdot u-ku\cdot u& =0\\ & \Rightarrow & k& =\frac{v\cdot u}{u\cdot u}\\ & \Rightarrow & ku& =\frac{u^{T}v}{||u|{|}^2}u\end{array}$$

Hence we can call $ku$ the orthogonal project of $v$ along the direction of $u$. The above equation can be rewritten as,

$$\begin{array}{rlrl}& & P_v& =u\frac{u^{T}v}{||u|{|}^2}\\ & \Rightarrow & P_v& =\frac{u{u}^T}{||u|{|}^2}v\end{array}$$

Here $u{u}^T$ would be an $n\times n$ matrix if $u$ is an $n-$dimensional vector. It is also called as the outer product of the $u$. This is also known as the projection matrix. Properties of the outer product -

• $P$ is a rank-1 matrix as all $n$ columns of the outer product would just be multiples of $u$.
• $P$ is a symmetric matrix, meaning $P^T=P$.
• $P$ is an idempotent matrix, meaning $P^2=P$.

$$\begin{array}{rlrl}& & P^2& =\frac{u{u}^{T}u{u}^T}{u^{T}u{u}^{T}u}\\ & & & =\frac{u\cancel{u^{T}u}{u}^T}{\cancel{u^{T}u}{u}^{T}u}\\ & & & =\frac{u{u}^T}{u^{T}u}\\ & & P^2& =P\end{array}$$

Projection over a subspace

Now instead of projecting $v$ onto a line $u$, what if we project it onto a subspace spanned by the vectors $u_1$ and $u_2$. Let the projection of $v$ onto this subspace be $u_p$. Then we can write,

$$v=e+u_p,\text{where }{u}_p=a{u}_1+b{u}_2$$

Because $e$ is orthogonal to $u_p$, we can show that it is orthogonal to both $u_1$ and $u_2$ (Exercise to try out). So we can write,

$$\begin{array}{rl}{v}^T{u}_1& =u_p^T{u}_1\\ & =a{u}_1^T{u}_1+b{u}_2^T{u}_1\\ v^T{u}_2& =u_p^T{u}_2\\ & =a{u}_1^T{u}_2+b{u}_2^T{u}_2\end{array}$$

In matrix form this can be written as,

$$\begin{array}{rlrl}& & \left[\begin{array}{c}{v}^T{u}_1\\ v^T{u}_2\end{array}\right]& =\left[\begin{array}{cc}{u}_1^T{u}_1& u_1^T{u}_2\\ u_2^T{u}_1& u_2^T{u}_2\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]\\ & \Rightarrow & \left[\begin{array}{c}a\\ b\end{array}\right]& ={\left[\begin{array}{cc}{u}_1^T{u}_1& u_1^T{u}_2\\ u_2^T{u}_1& u_2^T{u}_2\end{array}\right]}^{-1}\left[\begin{array}{c}{v}^T{u}_1\\ v^T{u}_2\end{array}\right]\end{array}$$

If the basis for the subspace are orthonormal, this can be simplified to

$$\left[\begin{array}{c}a\\ b\end{array}\right]=\left[\begin{array}{c}{v}^T{u}_1\\ v^T{u}_2\end{array}\right]$$

Thus giving us the clean formula that,

$$u_p=(v^T{u}_1)u_1+(v^T{u}_2)u_2$$

For some $n$ dimensional subspace with $n$ orthonormal basis vectors $u_1,u_2,\dots ,u_n$ this can be generalized to

$$u_p=\sum _{i=1}^n(v^T{u}_i)u_i$$

But now, how do we obtain these orthonormal basis for any given subspace? This is where the Gram-Schmidt Process comes in.

Gram-Schmidt Process

Let $W$ be a $d$-dimensional subspace of a $k$-dimensional vector space $V$ where $d\le k$. Let the basis of $W$ be $B=\{u_1,u_2,\dots ,u_n\}$ and we wish to create orthogonal basis $O=\{v_1,v_2,\dots ,v_n\}$ using these.

We can write,

$$\begin{array}{rl}{v}_1& =u_1\\ v_2& =u_2-\frac{u_2^T{v}_1}{v_1^T{v}_1}{v}_1\\ v_3& =u_3-\frac{u_3^T{v}_2}{v_2^T{v}_2}{v}_2-\frac{u_3^T{v}_1}{v_1^T{v}_1}{v}_1\\ & ⋮\\ v_d& =u_d-\left(\sum _{i=1}^{d-1}\frac{u_3^T{v}_i}{v_i^T{v}_i}{v}_i\right)\end{array}$$

From this $u_i$ can be written as -

$$\begin{array}{rl}{u}_i& =v_i+\left(\sum _{i=1}^{d-1}\frac{u_3^T{v}_i}{v_i^T{v}_i}{v}_i\right)\end{array}$$

In matrix form this could be written as -

$$\begin{gathered} A=\left[\begin{array}{cccc}{u}_1& u_2& \dots & u_d\end{array}\right]=\left[\begin{array}{cccc}{v}_1& v_2& \dots & v_d\end{array}\right] \\ \left[\begin{array}{cccc}1& \frac{u_2^T{v}_1}{v_1^T{v}_1}& \dots & \frac{u_d^T{v}_1}{v_1^T{v}_1}\\ 0& 1& \dots & \frac{u_{d-1}^T{v}_2}{v_2^T{v}_2}\\ & & ⋮& \\ 0& 0& \dots & 1\end{array}\right] \end{gathered}$$

If we instead wish to obtain orthonormal basis $\{q_1,q_2,\dots ,q_d\}$, we can just substitute $v_i=||v_i||q_i$. Thus the matrix representation becomes,

$$\begin{gathered} A=QR=\left[\begin{array}{cccc}{q}_1& q_2& \dots & q_d\end{array}\right] \\ \left[\begin{array}{cccc}||v_1||& \frac{u_2^T{v}_1}{v_1^T{v}_1}& \dots & \frac{u_d^T{v}_1}{v_1^T{v}_1}\\ 0& ||v_2||& \dots & \frac{u_{d-1}^T{v}_2}{v_2^T{v}_2}\\ & & ⋮& \\ 0& 0& \dots & ||v_d||\end{array}\right] \end{gathered}$$

To summarize, we start of with a matrix $A$ such that its column space spanned all of $W$. Such a matrix $A$ can be decomposed into product of two matrices $Q$ and $R$, where $Q$ is an orthogonal matrix.

Note - The vector $v_2$ lies in the orthogonal complement of $v_1$. The vector $v_3$ lies in the orthogonal complement of the space spanned by $v_1$ and $v_2$.

Orthogonal Matrix

A matrix is an orthogonal matrix if for all column vectors $u_i$ -

$$u_i\cdot u_j=\{\begin{array}{l}0,\text{if }i\ne j\\ 1,\text{if }i=j\end{array}$$

This means that -

• The columns vectors of the matrix are all of unit length.
• All columns vectors of the matrix are orthogonal to each other.

For orthogonal matrices $Q^{T}Q=I$, which means that $Q^{-1}=Q^T$.

Orthogonal Complement

Let $W$ be a subspace of ${\mathbb{R}}^n$. The orthogonal complement of $W$, denoted $W^{\perp }$ is the set of all vectors in ${\mathbb{R}}^n$ that are orthogonal to every vector in $W$.

$$W^{\perp }=\{v\in {\mathbb{R}}^n:v\cdot w=0,\forall w\in W\}$$

• If $\dim (W)=d$, $\dim (W^{\perp })=n-d$.
• Let $B_W$ be the orthonormal basis of $W$ and $B_{W_{\perp }}$ be the orthonormal basis of $W^{\perp }$. The orthonormal basis of $V$ will be $B_W\cup B_{W_{\perp }}$.

Using this property of orthogonal complements, any vector in $V$ can be written as a unique composition of any two vectors in $W$ and $W^{\perp }$.

$$x=x_W+x_{W_{\perp }}$$

$\text{Proof to show that this decomposition is unique -}$

• Assume that there are two decompositions of $x$, $x_W+x_{W_{\perp }}$ and $x_W^{\prime }+x_{W_{\perp }}^{\prime }$.
• Because both lead to $x$, we can say that $x_W+x_{W_{\perp }}=x_W^{\prime }+x_{W_{\perp }}^{\prime }$. Upon rearranging this we can say that $x_W-x_W^{\prime }=x_{W_{\perp }}^{\prime }-x_{W_{\perp }}$, where the LHS vectors belong to $W$ and the RHS vectors belong to $W_{\perp }$.
• $W\cap W_{\perp }=\{0\}$ only, thus if $x_W-x_W^{\prime }=x_{W_{\perp }}^{\prime }-x_{W_{\perp }}$ then $x_W-x_W^{\prime }$ and $x_{W_{\perp }}^{\prime }-x_{W_{\perp }}$ both need to be 0. Meaning that $x_W^{\prime }=x_W$ and $x_{W_{\perp }}^{\prime }=x_{W_{\perp }}$.
• Hence proved that the decomposition of a vector into a sum of two vectors from orthogonal subspaces is unique.