goormNLP [Linear Independence]

Auspice by Goorm, Manage by DAVIAN @ KAIST

Lecture 3: Linear combination, Vector equation, Four views of matrix multiplication

Linear combination

c1v₁ + … + c1v_p is called a linear combination of v₁, … , v_p with weights or coefficients c₁**, … , **c_p.

• The weights in a linear combination can be any real numbers, including zero.

Span

• Definition: Span {V1, …, Vp} is defined as the set of all linear combinations of V1, … , Vp. Span is also called the sub set of Rⁿ spanned.

• b가 span 안에 있으면 O -> 해가 존재함.
• b가 span 안에 없으면 X -> 해가 존재 하지 않음.

Matrix Multiplications as Column Combinations

• Left matrix: bases, Right matrix: coefficients

Matrix Multiplications as Row Combinations

• Left matrix: coefficients, Right matrix: bases

Matrix Multiplications as Sum of (Rank-1) Outer Products

• (Rank-1) outer product

• Sum of (Rank-1) outer products

=> Sum of (Rank-1) outer products is widely used in machine learning.

• Covariance matrix in multivariate Gaussian.
• Gram matrix in style transfer.

---

Lecture 4: Linear Independence, Span, Subspace

Uniqueness of Solution for Ax = b

• It is unique when a₁, a₂, and₃ are linearly independent. 선형 독립 –> 해가 1개 존재.

• Infinitely many solutins exist when a₁, a₂, and₃ are linearly dependent. 선형 의존 –> 해가 무수히 존재.

• if at least one such V_j is found, then {V1, … , Vp} is linearly dependent.

• if no such V_j is found, the {V1, … , Vp} is linearly independent.

• **A linearly dependent vector does not increase Span ! **

Span and Subspace

• Definition: A subspace H is defined as a subset of Rⁿ closed under linear combination.

In fact, a subspace is always represented as Span {V1, ... , Vp}

Basis of a Subspace

• Definition: A basis of a sub space H is a set of vectors that satisfies both of the following:
  1. Fully spans the given subspace H.
  2. Linearly independent.

Column Space of Matrix

• Definition: The column space of a matrix A is the subspace spanned by the columns of A.
  • We call the column spac eof A as Col A.

What is dim Col A? ==> 2

What is dim Col A? ==> 2

Rank of Matrix

• Definition: The rank of a matrix A, denoted by rank A, is the dimension of the column space of A:
  • rank A = dim Col A.