Dot Product

$\underline{Definition} -$ The inner product or the dot product in a vector space $V$ over $R$ is a map, that for any 2 vectors $u, v \in V$ there is a real number $< u, v >$ such that,

$< u, αv + βw >= α < u, v > + β < u, w >$ (Linearity Property)
$< u, v >=< v, u >$ (Symmetric Property)
For any $u \in V$ , $< u, v >= 0$ if and only if $u$ is the zero vector. ()

The vector space for which the inner product $(\cdot)$ is defined is called an inner product vector space.

Suppose the two vectors $u, v \in R^{2}$ are like this. We can write $u_{1}, u_{2}, v_{1}, v_{2}$ using trigonometric identities as -

u_{1} = ∣∣ u ∣∣ cos θ, v_{1} = ∣∣ v ∣∣ cos (θ + ϕ), u_{2} = ∣∣ u ∣∣ sin θ v_{2} = ∣∣ v ∣∣ sin (θ + ϕ)

$u \cdot v$ is defined as $u_{1} v_{1} + u_{2} v_{2}$ . So using the above values for this gives us,

u \cdot v = u_{1} v_{1} + u_{2} v_{2} = (∣∣ u ∣∣ cos θ \cdot ∣∣ v ∣∣ cos (θ + ϕ)) + (∣∣ u ∣∣ sin θ \cdot ∣∣ v ∣∣ sin (θ + ϕ)) = ∣∣ u ∣∣ ∣∣ v ∣∣ cos (ϕ + θ - θ) = ∣∣ u ∣∣ ∣∣ v ∣∣ cos (ϕ)

Cauchy-Schwarz Inequality

∣ u \cdot v ∣ ∣ u \cdot v ∣ = ∣∣ u ∣∣ ∣∣ v ∣∣ cos θ \leq ∣∣ u ∣∣ ∣∣ v ∣∣

Orthogonal Vectors

If $u, v \in V$ are vectors such that $u \cdot v = 0$ , then we say that $u$ and $v$ are orthogonal to each other.

The zero vector is orthogonal to all vectors in any vector space.
If $u, v$ are non-zero vectors, then $u \cdot v = 0$ can only happen when the angle between $u, v$ is 90 $°$ .

Let $W^{⊥}$ be a set of all vectors orthogonal to every vector in $W$ , where $W$ is a subspace of the inner product vector space $V$ .

Let $u, v \in W^{⊥}$ and $w \in V$ -

The zero vector is always orthogonal to any vector in any vector space. Thus $W^{⊥}$ contains the zero vector and is non-empty.
Because $u \cdot w = 0$ and $v \cdot w = 0$ , we can say

\Rightarrow u \cdot w + v \cdot w (u + v) \cdot w = 0 = 0

This implies that $u + v \in W^{⊥}$ too. Thus $W^{⊥}$ is closed under vector addition. 3. Because $u \cdot w = 0$ , we can say $α (u \cdot w) = 0$ . Consequently this would mean that $αu \cdot w = 0$ , and hence $αu \in W^{⊥}$ . Thus $W^{⊥}$ is closed under scalar multiplication.

Hence we can say that $W^{⊥}$ is a vector space too as it is closed under vector addition and scalar multiplication and contains the zero vector.

The vector space $W^{⊥}$ is called the orthogonal complement of the space $W$ .

Orthonormal vectors

A set of vectors is said to be orthonormal if -

Each vector in the set is orthogonal to every other vector.
Each vector has a magnitude of 1.

Suppose $s = {v_{1}, v_{2}, \dots, v_{n}}$ is a set of orthonormal vectors, what can we say about the dependence/independence of this set?

The set of vectors is independent only when $α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} = 0$ only when $α_{i} = 0 \forall i \in [1, n]$ . Let,

\Rightarrow \Rightarrow \Rightarrow \Rightarrow \Rightarrow α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} v_{1} \cdot (α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n}) α_{1} (v_{1} \cdot v_{1}) + α_{2} (v_{1} \cdot v_{2}) + \dots + α_{n} (v_{1} \cdot v_{n}) α_{1} ∣∣ v_{1} ∣ ∣^{2} α_{1} * 1 α_{1} = 0 = v_{1} \cdot 0 = 0 = 0 = 0 = 0

Similarly we can show that $α_{i} = 0 \forall i \in [1, n]$ . Thus we can say that every set of orthonormal vectors is linearly independent and can be chosen as the basis for their vector space.

Fourier Expansion

Let $b$ be the basis expansion of any vector. We can get the scalars for each basis vector by doing,

\Rightarrow \Rightarrow \Rightarrow \Rightarrow b v_{1} \cdot b v_{1} \cdot b v_{1} \cdot b α_{1} = α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} = v_{1} \cdot (α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n}) = α_{1} (v_{1} \cdot v_{1}) + α_{2} (v_{1} \cdot v_{2}) + \dots + α_{n} (v_{1} \cdot v_{n}) = α_{1} = v_{1} \cdot b

Similarly we can find $α_{i} = v_{i} \cdot b \forall i \in [1, n]$ .

Using this knowledge we can rewrite $b$ as,

b = (b \cdot v_{1}) v_{1} + (b \cdot v_{2}) v_{2} + \dots + (b \cdot v_{n}) v_{n} = ∣∣ b ∣∣ ∣∣ v_{1} ∣∣ cos θ_{1} v_{1} + ∣∣ b ∣∣ ∣∣ v_{2} ∣∣ cos θ_{2} v_{2} + \dots + ∣∣ b ∣∣ ∣∣ v_{n} ∣∣ cos θ_{n} v_{n} = ∣∣ b ∣∣ (cos θ_{1} v_{1} + cos θ_{2} v_{2} + \dots + cos θ_{n} v_{n}) = ∣∣ b ∣∣ i = 1 \sum n cos θ_{i} v_{i}

This is called the Fourier Expansion for $b$ .

Parseval’s Theorem

We know that $b = \sum_{i = 1}^{n} α_{i} v_{i} = \sum_{i = 1}^{n} (b \cdot v_{i}) v_{i}$ .

\Rightarrow \Rightarrow \Rightarrow b ∣∣ b ∣ ∣^{2} = i = 1 \sum n (b \cdot v_{i}) v_{i} = ⟨ i = 1 \sum n (b \cdot v_{i}) v_{i}, i = 1 \sum n (b \cdot v_{j}) v_{j} ⟩ = i, j \sum (b \cdot v_{i}) (b \cdot v_{j}) < v_{i}, v j > = i, j \sum ∣ b \cdot v_{i} ∣^{2}

Orthogonal Projections

$\underline{Definition} -$ A projection of a vector is the best approximation of a vector in a vector space.

Let $u, v$ be two non-zero vectors in an inner product space $V$ . What is the information of $v$ available in the direction of $u$ ?

We can drop a perpendicular line $v_{⊥}$ onto $k u$ from $v$ , where $k$ is some real scalar. By vector addition -

\Rightarrow v v_{⊥} = k u + v_{⊥} = v - k u

As $v_{⊥}$ is orthogonal to $k u$ by design, we can say that it’s orthogonal to $u$ as well. So,

\Rightarrow \Rightarrow \Rightarrow \Rightarrow v_{⊥} \cdot u (v - k u) \cdot u v \cdot u - k u \cdot u k k u = 0 = 0 = 0 = \frac{v \cdot u}{u \cdot u} = \frac{u ^{T} v}{∣∣ u ∣ ∣ ^{2}} u

Hence we can call $k u$ the orthogonal project of $v$ along the direction of $u$ . The above equation can be rewritten as,

\Rightarrow P_{v} P_{v} = u \frac{u ^{T} v}{∣∣ u ∣ ∣ ^{2}} = \frac{u u ^{T}}{∣∣ u ∣ ∣ ^{2}} v

Here $u u^{T}$ would be an $n \times n$ matrix if $u$ is an $n -$ dimensional vector. It is also called as the outer product of the $u$ . This is also known as the projection matrix. Properties of the outer product -

$P$ is a rank-1 matrix as all $n$ columns of the outer product would just be multiples of $u$ .
$P$ is a symmetric matrix, meaning $P^{T} = P$ .
$P$ is an idempotent matrix, meaning $P^{2} = P$ .

P^{2} P^{2} = \frac{u u ^{T} u u ^{T}}{u ^{T} u u ^{T} u} = \frac{u u ^{T} u u ^{T}}{u ^{T} u u ^{T} u} = \frac{u u ^{T}}{u ^{T} u} = P

Gram-Schmidt Process

Let $W$ be a $d$ -dimensional subspace of a $k$ -dimensional vector space $V$ where $d \leq k$ . Let the basis of $W$ be $B = {u_{1}, u_{2}, \dots, u_{n}}$ and we wish to create orthogonal basis $O = {v_{1}, v_{2}, \dots, v_{n}}$ using these.

We can write,

v_{1} v_{2} v_{3} v_{d} = u_{1} = u_{2} - \frac{u _{2}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} = u_{3} - \frac{u _{3}^{T} v _{2}}{v _{2}^{T} v _{2}} v_{2} - \frac{u _{3}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} ⋮ = u_{d} - (i = 1 \sum d - 1 \frac{u _{3}^{T} v _{i}}{v _{i}^{T} v _{i}} v_{i})

From this $u_{i}$ can be written as -

u_{i} = v_{i} + (i = 1 \sum d - 1 \frac{u _{3}^{T} v _{i}}{v _{i}^{T} v _{i}} v_{i})

In matrix form this could be written as -

A = [u_{1} u_{2} \dots u_{d}] = [v_{1} v_{2} \dots v_{d}] 100 \frac{u _{2}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} 10 \dots \dots ⋮ \dots \frac{u _{d}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} \frac{u _{d - 1}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} 1

If we instead wish to obtain orthonormal basis ${q_{1}, q_{2}, \dots, q_{d}}$ , we can just substitute $v_{i} = ∣∣ v_{i} ∣∣ q_{i}$ . Thus the matrix representation becomes,

A = QR = [q_{1} q_{2} \dots q_{d}] ∣∣ v_{1} ∣∣ 00 \frac{u _{2}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} ∣∣ v_{2} ∣∣ 0 \dots \dots ⋮ \dots \frac{u _{d}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} \frac{u _{d - 1}^{T} v _{1}}{v _{1}^{T} v _{1}} v_{1} ∣∣ v_{d} ∣∣

To summarize, we start of with a matrix $A$ such that its column space spanned all of $W$ . Such a matrix $A$ can be decomposed into product of two matrices $Q$ and $R$ , where $Q$ is an orthogonal matrix.

Orthogonal Matrix

A matrix is an orthogonal matrix if -

u_{i} \cdot u_{j} = ⎩ ⎨ ⎧ 0, if i \neq = j 1, if i = j

This means that -

The columns vectors of the matrix are all of unit length.
All columns vectors of the matrix are orthogonal to each other.

For orthogonal matrices $Q^{T} Q = I$ , which means that $Q^{- 1} = Q^{T}$ .

GATE & BS Notes

Explorer

Othogonality