1. Introduction.

We are going to present part of the basics of number theory, culminating in the demonstration of the fact that the root of any natural number is rational iff this natural number is a perfect square.

The results shown in the present post are a single deductive chain with no added ingredient that starts from a single premise axiom of the natural numbers $\mathbb{N}$ (good order) and stops at the irrationality of $\sqrt{2}$. The idea that is worth remembering throughout the paper is: the good order produces the division algorithm, the division algorithm produces Bézout, which is a result that translates divisibility into linearity, and that expands to the rest.

1.1. Conventions.

Before starting, let's fix some conventions:

• Being $a,b \in \mathbb{Z}$ then we say "$b$ divides $a$" and write $b \mid a$ when:

  \[b \mid a \iff \exists k \in \mathbb{Z} : a = kb\]

  Let's observe that divisibility respects linearity:

  \[c \mid a \wedge c \mid b \implies c \mid (ma + nb) \quad \forall m,n \in \mathbb{Z}\]

  
  

• Being $a,b \in \mathbb{Z}: ab \neq 0$, we call $gcd(a,b) = \max\Set{c \in \mathbb{N} : (c \mid a \wedge c \mid b)}$, this is the maximum integer that divides both numbers at once. Let's observe some interesting facts:

  • $1 \in gcd(a,b) \quad \forall a,b \in \mathbb{Z}$

  • $\max \Set{|a|,|b|} \geq c \quad \forall c \in gcd(a,b)$, observe immediately that any greater number would not divide both of them so it is not in $gcd(a,b)$.

  • $gcd(a,b) = gcd(b,a) \quad \forall a,b \in \mathbb{Z}$

  • $gcd(a,b) > 0 \quad \forall a,b \in \mathbb{Z}$

    
    

2. Good Order in $\mathbb{N}$. Induction.

2.1. Conceptual presentation.

Having clarified these concepts, let's talk about the "good order" and the induction principle. This idea is expressed as a property of a set; a set is said to have good order or be well-ordered if it has a minimum, this is, a least element. In a more generic way; a good order is a total order in which any non-empty subset has a minimum.

The canonical example is the set of the natural numbers. This will be our departure point:

$\mathbb{N}$ is a well-ordered set

Let's now talk about the induction principle which is tightly related to $\mathbb{N}$ and co-implies good order.

2.2. Formal definition. Good order.

1. Introduction.

2. Finite, Countable and Uncountable Sets.

   
   

1. Introduction.

First, let's explain why there is a topology chapter in an analysis book.

As we asserted in other chapter, Analysis is the study of the limits, and the limit is essentially a statement about closeness. Then, the Topology is the branch which isolates and study the notions of closeness, proximity or continuity in abstract without appealing to distances or metrics.

Hence, this topology chapter brings to the reader a precise vocabulary-kit in which the terms limit, continuity or convergence have complete sense. The hardest theorems of elementary analysis are, in fact, topological theorems.

Analysis emerges to give foundation to Calculus works which, before analysis, it worked with a non-fully-deployed but intuitive idea of limit, then called infinitesimals; $dx$. Primordial analysis stablished a first limit definition that, despite being rude, was mathematically accurate:

Observe that this first definition involves the metric, this means that this limit idea is not property of the real field, is property of the space in which a proximity notion exists which is what the topology studies.

Later, topologic concepts rebuilt the limit concept in a simplier way, disregarding metrics and epsilons and using heavy geometric nuance.

Thus, there is three level of abstractions. First let's barely introduce what the open sets are; an open set is a set in which all its points are interior points. Intuitively, this means that the set does not contain its boundary, allowing you to approach any element without stepping outside the set's limits.

Thus:

• A topologic space; is a set $X$ along witha a collection of open subsets following three axioms.

• A metric space; is a particular case of toplogic space in which the open subsets gets generated with the metric. In some sense, an open set is a set in which every element has an "enviroment" of elements (defined by the metric) within the set, like a circule or sphere centered in the element which enterly falls inside the set.

• A euclidean space is a particular case of the metric space in which the metric is the euclid norm.

  
  

Thus, the topology introduced in this section is the metric spaces topology.

2. Finite, Countable and Uncountable Sets.

We begin this section with a definition of the function concept.

2.1. Functions. Applications.

2.1.1. Main concepts.

Consider two sets. $A,B \neq \varnothing$.

Then we define a function or application $f$ from $A$ to $B$ and we denote it as:

This is a relation which connects each $x \in A$ to one $y \in B$. We denote $f(x)$ to the subset of elements of $B$ related with $x$ by $f$:

Observe that we implicitly define a rule that every function $f$ must satisfy; $f(x)$ is a unary set, it only has one element and thus, through an economization of notation we simply state $f(x) = y$ when $f$ is a function:

In this context we state that $A$ is the domain and $B$ is the codomain of $B$ which not necesarily coincide with the set of all the images of $A$ through $f$, denoted by $f(A) := \Set{y \in B \mid \exists x \in A : y = f(x)}$ and called range of $f$.

2.1.2. Injectivity, Surjectivity and Bijectivity.

Let's explore three important concepts about how relations connect input with outputs. Be $f : A \to B$ a application, then:

• Inyective: We say that $f$ is inyective it verifies no colisions with images, formally:

  \[\forall x,y \in A : x \neq y \implies f(x) \neq f(y)\]

  
  

• Surjective: We say that $f$ is surjective when the range of $f$ coincide with the codomain $f(A) = B$, intuitively it has no gaps, every element on the codomain is reached:

  \[\forall y \in B \ \exists x \in A : f(x) = y\]

  
  

• Biyective: We say that $f$ is biyective when is a "one-to-one" correspondence between the elements of $A$ and $B$:

  \[\forall y \in B \ \exists ! x \in A : f(x) = y\]

  Let's observe that $f$ is biyective if and only if is at the same time surjective and inyective.

2.1.3. Inverse.

Consider $f : A \to B$ to been a relation. In this context we talk about a mappping of $A$ into $B$ through $f$, then we consider the inverse mapping from $B$ to $A$, denoted by $f^{-1}$ as:

Let's observe that, if we now turn $f$ to be a function, $f^{-1}$ is a only a function if $f$ is biyective and obviously $f^{-1}$ is biyective as well. Note that:

Thus, we assert that for any function $f$ the inverse function exists if and only if $f$ is biyective.

2.2. Cardinality. Count. Finiteness.

The following section stablishes the mathematically notion of count. Let's present the tools used to count.

First, let's introduce the cardinality. Take $A,B \neq \varnothing$, then we say that both sets has the same cardinality if we can define a biyection between them.

Let's observe that:

• We can define a biyection over any set with himself (reflexivity).
• If a biyection is stablished from $A$ to $B$, then the inverse is a biyection of $B$ to $A$ (symmetric).

• If there are biyection from $A$ to $B$ and from $B$ to $C$, the composition of the both is a biyection from $A$ to $C$ (transitivity).

  
  

Thus, we can define a equivalent relation around cardinality; two sets are equivalent if they share his cardinality.

Now, let's introduce that in mathematics, count consist basically in measure the cardinality of a set, this is, to count is to relate the elements of a set with another set through a biyection. Let's explore this idea formally, let be $A$ some set:

• $A$ is finite if exists some $n \in \mathbb{Z}^+ : A \sim [n]$, remember that $[n] = \Set{1,2,\ldots,n}$. In this terms we say that $A$ has cardinality or cardinal number of $n$, $[n]$ is the canonical representant of the cardinal-equivalence class of $A$.

• $A$ is infinite if its not finite.

• $A$ is countable (or enumerable or denumerable) if $A \sim \mathbb{Z}^+$.

• $A$ is at most countable if its finite.

• $A$ is uncountable if $\nexists S \in \mathcal{P}(\mathbb{Z}^+) : A \sim S$, note that this involves $\mathbb{Z}^+$ it self.

Note that with this notions two finite cardinal-equivalents sets $A, B$ has the same number of elements, but observe that with infinite sets the idea of have the same number of elements becomes quite vague but the biyection idea retains its clarity.

Let's see an example with $\mathbb{Z}$ and $\mathbb{Z}^+$ and consider $f: \mathbb{Z}^+ \to \mathbb{Z}$:

Observe that this function is injective and suprajective so is a biyection and $\mathbb{Z}$ and $\mathbb{Z}^+$ has the same cardinality.

Observe that this happens despite the fact that $\mathbb{Z}^+ \subset \mathbb{Z}$, buy a finite set cannot be equivalent to one of its proper subsets.

2.3. Sequences.

2.3.1. Definition.

Let's introduce the notion of a sequence. Intuitively a sequence is an infinite sorted list. Formally, a sequence is a function $s : \mathbb{Z}^+ \to A : s(n) \in A$, where $A \neq \varnothing$.

This function contains two objects that characterizes the information and the set:

• An index over the elements of $A$, which is given by the preimage $n \in \mathbb{Z} : x = s(n) \to x_n$

• To each index, each position, there is only one element ocupping the slot (this is given by the fact that $s$ is a function) which is called the $n$-th term of the sequence. Observe that two terms of distinct index not need to be distinct.

We often call as $\Set{x_n}$ to the sequence.

Observe some interesting relation between sequences and countable sets. Since every countable set is the range of a $1-1$ function defined on $\mathbb{Z}^+$, we may regard every countable set as the range of a sequence of distinct terms. Speaking more loosely, we may say that the elements of any countable set can be arranged in a sequence.

2.3.2. Theorem; Every infinite subset of a countable subset is countable.

Take some $A \subset E$, such $E$ is countable and $A$ is not finite. Since $E$ is countable then exists some biyection $f : \mathbb{Z}^+ \to E$, then we can consider $g: \mathbb{Z}^+ \to A$ as follows; since $A \subset E$ then we can consider the range of $f$ over $A$. Then, each element of $f(A)$ has an index, and since $\mathbb{Z}^+ \subset \mathbb{N}$ has good order, then we can consider first some minimum index $m$ and then we can order $f(A)$ items through his index. Then, $g(1) = x_m$, and the sucesor of $x_m$ in $A$ receives $g(2)$ and so on, ultimately, we crafted a sequence, $g$ in $A$, thus $g$ is a biyection and $A$ is countable.

2.4. Families. Intersection and Union of a family of sets. Countable and Uncountable example sets.

2.4.1. Definition.

Let's now introduce the family languages, the indexes family sets and define the union and intersection of an arbitrary family set.

Let $A$ and $\Omega$ be sets such each $\alpha \in A$ is asociated with a subset of $\Omega$, denoted by $E_\alpha \subset \Omega$. We denote as $\Set{E_\alpha}$ to set whose elements are the sets $E_\alpha$ and call it collection of sets or family of sets.

2.4.2. Union and Intersection of sets.

The union of the sets $E_\alpha$ is defined as:

The intersection:

2.4.3. Relation between Union/Intersection and Sum/Product.

Many properties of unions and intersections are quite similar to those of sums and products; in fact, the words sum and product were sometimes used in this connection, and the symbols $\sum$ and $\prod$ were written in place of $\cup$ and $\cap$. The asociativity and distributivity laws comes from this very same laws in disjuntion and conjuntion.

2.4.4. Cantor's diagonal methods and auxiliary results.

We have the following results:

The arbitrary union of countable sets is a countable set. First Cantor's diagonal method.

Let' be $\Set{E_n} : n = 1,2,3,\ldots$ be a collection or family of countable sets, then consider:

First, since $E_n$ for each $n =1,2,3,\ldots$ is numerable, then we can consider a biyection $f : \mathbb{Z}^+ \to E_n$ for each $n$, this biyection allow us to arrange or list the elements of $E_n$ in a row

Let's do this for each $n$ obtaining something like a matrix where each row is the sorted $E_n : n = 1,2,3,\ldots$

Note that the 'matrix' denomination of this structure is just an intuitive approximation and we are not saying that it is properly a matrix at all.

Then, observe we can sort this elements by diagonalizating them, forming the sequence:

Observe then that as an arranged list, we can define a biyection over it, thus $S$ is countable.

The at most countable union of at most countable sets is at most countable

The proof is similar to the above.

If $A$ is countable, then $A^n$ is countable for any $n \in \mathbb{N}$

Let's proof this result by induction.

Take $n = 1$, then this result is immediate since $A^1$ is $A$.

Then, let's give it true for some $n \in \mathbb{N}$, and consider $A^{n}$, each element is of the form $(a_1,\ldots, a_{n-1}, a_n) \in A^n$, calling $\alpha = (a_1,\ldots,a_{n-1}) \in A^{n-1}$, then the first is identifiable with the pair $(\alpha,a_n)$, observe that fixed some $\alpha$, the set of the pairs $(\alpha,a) : a \in A$ can be identified with $A$ it self, meaning that is countable and his elements can be arranged in a row:

Also observe that since $A^{n-1}$ is countable as well, then, for each $a \in A$ the set of the pairs $(\alpha,a) : \alpha \in A^{n-1}$ is countable as well and it can be disposed in a column, obtaining a structure in which we can use the first Cantor diagonal method.

Allowing us to create the sequence:

Reidentificating the pair $(\alpha,a)$ with the tuple $(a_1,\ldots,a_n) \in A^n$, we have created a biyection from $\mathbb{Z}^+$ over the elements of $A^n$, so this is is countable as well.

Essentially observe that we reused the argument provided in the first point since $A^n$ is the countable union of the countable sets $\Set{(\alpha,a) : a \in A}$ for each $\alpha \in A^{n-1}$, so is countable.

$\mathbb{Q}$ is countable

Observe that we can again reuse the argument before. We did see that $\mathbb{Z}$ is countable so each set $E_m = \Set{n/m : n \in \mathbb{Z}}$ for some $m \neq 0$ is countable, thus:

Is countable.

Uncountable set. Second Cantor's diagonal method. $\mathbb{R}$ is uncountable.

This is an example of an infinity which is strictly greater than $\mathbb{N}$. Consider the set $A:=\Set{0,1}^\mathbb{N}$, of all the functions $s:\mathbb{N} \to \Set{0,1}$. Observe that each sequence is the indicator function of some subset of $\mathbb{N}$, for which is in biyection with $\mathcal{P}(\mathbb{N})$, hence the theorem is $card(\mathbb{N}) \leq card(\mathcal{P}(\mathbb{N}))$.

Take $E \subseteq A$ as countable consisting in the sequences $s_1,s_2,s_3,\ldots$. Then, take some sequence $s$ crafted as:

If the $n$-th digit in $s_n$ is $1$, then the $n$-th digit of $s$ is $0$ provoking that $s$ differs from any element of $E$ in at least one place so $s \notin E$ but $s \in A$ since $s$ is composed of $0$ and $1$ so $E \subset A$. Any countable subset of $A$ is a proper subset, thus $A$ can't be countable (or it would be its own countable subset) and is uncountable.

Laterly we will see that $\mathbb{R}$ admits a binary representation and this same result will apply to demonstrate that $\mathbb{R}$ is uncountable.

2.5. Summary.

Let's take a brief summary of section $2$. First, we introduce the relation and a concrete form or relation which we called function, which is a relation for which the image set for any element of the domain is a unary-non-empty set.

Then we introduced the biyection, or one-to-one relations and along with it the mathematically conception of count which is identify the elements of two sets through a biyection in the sense that we can attach each element with another unique item of other set, stablishing that both sets has the same number of elements, the same cardinality. The cardinality equivalence is a relation equivalence in which the canonical representant is the natural number subsets, $[n]$.

Later

3. Metric Spaces.

1. Presentation and Definition. $\mathbb{R}^k$
2. Metric Spaces.
   • 2.1. Metric definition. Metric spaces.
   • 2.2. Euclidean metric.
   • 2.3. $\mathbb{R}^k$ is a metric space with the euclidean metric.
3. Main properties of the norm in $\mathbb{R}^k$.

4. Summary.

   
   

1. Presentation and Definition. $\mathbb{R}^k$

A Euclidean space is an affine space over the reals such that the associated vector space is a euclidean vector space; a finite-dimensional inner product space over the real numbers.

Let's consider the affine space $(\mathbb{R}^k,V,+)$ where the associated $\mathbb{R}$-vector space is $V= (\mathbb{R},(\mathbb{R}^k,+),·)$, with the operations as follows:

Being $+: V \times \mathbb{R}^k \to \mathbb{R}^k $ a simply transitive action over $\mathbb{R}^k$

Over $V$, we define the following inner product, which we remember from the Complex Numbers post, on $6.3.1$.

An inner product is the object which injects geometry by defining how a vector expresses its own direction (norm), introducing subsequently the orthogonality. In this particular presentation, this inner product receives the particular name dot product.

The defined dot product gives rise to the norm; for each vector $x \in \mathbb{R}^k$ it is defined:

Which gives us a notion of "length" of the vector. It is worth saying that this is specifically true in $\mathbb{R}^k$ where intuitive geometry matches the formal geometry instituted through the dot product.

Thus, a euclidean space is the result of injecting geometry into an affine space through a real inner product space; a real vector space with geometry.

As a consequence of the geometry injected by the inner product, we can consider a notion of proximity (metric) between the points in the affine space which allows us to introduce the notion of limit and develop the analysis.

2. Metric spaces.

2.1. Metric definition. Metric spaces.

A metric abstracts the notion of distance. This is an important concept since from this all the analysis departs. From the distance notion comes proximity and then limits gain sense.

A distance tries to capture how far two points are from each other in a non-empty set $X$. Observe that this is not an algebraic structure or algebraic property; contrary to the norm, the distance doesn't need to sum or scale.

Let $X$ be a non-empty space, then a metric space is a pair $(X,d)$ where: $d : X \times X \to \mathbb{R}$ is an application called distance, verifying:

• $M1$: Symmetry, the distance is indifferent to the point of measurement:

  \[d(x,y)=d(y,x) \quad \forall x,y \in X\]

  
  

• $M2$: Gets null on the diagonal:

  \[d(x,y)=0 \iff x = y \quad \forall x,y \in X\]

  
  

• $M3$: Triangular inequality:

  \[d(x,z) \leq d(x,y) + d(y,z) \quad \forall x,y,z \in X\]

  
  

Observe some immediate consequences of the three axioms:

The distance is always positive as a consequence of the triangular inequality and the symmetry.

2.2. Euclidean metric.

Let's consider again the affine space $\mathbb{R}^k$ as defined above with the dot product. Then, $V$ is an inner product vector space and, consequently, a normed vector space. Let's see how we can develop a metric in $\mathbb{R}^k$ to state it as a metric space.

First, at this point, we will develop the metric in $\mathbb{R}^k$ through its norm (depending on the inner product) although we established this is not entirely necessary.

We define the metric in $\mathbb{R}^k$ as, for $x,y \in \mathbb{R}^k$, we have that:

This is, the distance between two points on $\mathbb{R}^k$ is the length of the vector that connects both points. Remember from our knowledge of affine space that $x + \overrightarrow{xy} = y$

2.3. $\mathbb{R}^k$ is a metric space with the euclidean metric.

Let's see that $(\mathbb{R}^k,d)$ as defined is a metric space:

• Symmetry:

  \[d(x,y) = \|x - y\| = \sqrt{\sum_i^k (x_i -y_i)^2} = \sqrt{\sum_i^k (y_i - x_i)^2} = \| y - x \| = d(y,x)\]

  
  

• Zero distance along the same point:

  \[d(x,y) = \|x - y\| = \sqrt{\sum_i^k (x_i -y_i)^2}\]

  Since it is a sum of positive terms, to be zero all the operands have to be zero, this is:

  \[d(x,y) = 0 \iff x_i =y_i \quad i=1,2,...,k \iff x = y \quad \forall x,y \in \mathbb{R}^k\]

  
  

• Triangle inequality: This result comes immediately from the inner product. Take $x,y \in \mathbb{R}^k$, then:

  \[\|x+y\|^2 = \|x\|^2 + \|y\|^2 + 2 \langle x,y \rangle\]

  Let's see that, by Cauchy-Schwarz: $|\langle x,y \rangle| \leq ||x|| ||y||$, thus

  \[\|x+y\|^2 = \|x\|^2 + \|y\|^2 + 2 \langle x,y \rangle \leq \|x\|^2 + \|y\|^2 + 2\|x\| \|y\| = (\|x\| + \|y\|)^2\]

  Since $||x+y|| \geq 0$ and $||x|| + ||y|| \geq 0$ is:

  \[\|x+y\|^2 \leq (\|x\| + \|y\|)^2 \implies \|x+y\| \leq \|x\| + \|y\|\]

  Giving place to the triangular inequality (which is always true in a real inner product space).

  
  

  Now, observe that:

  \[d(x,z) = \|x -z\| = \|x -y + y-z\| \leq \|x-y\| + \|y-z\| = d(x,y)+d(y,z)\]

  The inequality is guaranteed by the triangle inequality in the norm, preceded by Cauchy-Schwarz.

  
  

3. Main properties of the norm in $\mathbb{R}^k$

Let's evaluate some main properties of the norm of $\mathbb{R}^k$.

1. $||x|| \geq 0$, is immediate from the fact that is the square of a real number.
2. $||x|| = 0 \iff x = 0$, immediate from norm definition.
3. $||\alpha x || = (\langle \alpha x, \alpha x \rangle)^\frac{1}{2} = (\alpha^2 \langle x,x \rangle)^\frac{1}{2} = |\alpha| ||x||$
4. $||x · y|| \leq ||x|| ||y||$ Immediate from Cauchy-Schwarz.
5. $||x+y|| \leq ||x|| + ||y||$ Triangle inequality previously demonstrated.

6. $||x-z|| \leq ||x-y|| + ||y-z||$ An extension of the triangle inequality relative to the euclidean metric also demonstrated above.

   
   

4. Summary.

Thus, we've seen that $\mathbb{R}^k$ as an affine space with $\mathbb{R}^k$ as the associated vector space and the real inner product as the dot product; $x · y$ conforms a Euclidean space; an affine space with a geometry induced by the vector space which crystallizes in the metric $d(x,y) = ||\overrightarrow{xy}||$

5. Exercises.

5.1.

1. If $2r > d$ there are infinitely many $z \in \mathbb{R}^k$ such:

   \[|z - x| = |z - y| = r\]

   Observe that by the triangle inequation is:

   \[|x -y| \leq |x-z| + |z-y|\]

   Thus, the points requested by the exercise are those points satisfying:

   \[z \in \mathbb{R}^k: \begin{cases}|x-z| = |z-y|\\ d = |x -y| < |x-z| + |z-y| = 2r\end{cases}\]

   Note that the first condition give us a general caracterization of the points $z$, those whose coordinates satisfies the equation:

   \[\left(\sum_{i=1}^k(x_i - z_i)^2\right)^{1/2} = \left(\sum_{i=1}^k(y_i - z_i)^2\right)^{1/2} \iff \sum_{i=1}^k(x_i - z_i)^2 = \sum_{i=1}^k(y_i - z_i)^2\]

   Note that for each square is:

   \[(x_i - z_i)^2 = x^2_i -2x_iz_i + z^2_i = y^2_i -2y_iz_i + z^2_i = (y_i - z_i)^2\]

   The $z^2_i$ cancels from both sides and sorting:

   \[2z_i(y_i - x_i)=y^2_i - x^2_i\]

   Adding now the summatories we have:

   \[z·(y-x) = \frac{1}{2}(|y|^2 - |x|^2)\]

1. Introduction.
2. Subspaces.
   • 2.1. Definition and characterization.
   • 2.2. Example of subspaces.
   • 2.3. Combination of vector spaces.
     • 2.3.1. Intersection of subspaces.
     • 2.3.2. Spanned subspace by a set.
     • 2.3.3. Sum of subsets of vector spaces.
   • 2.4 Example of combinated vector spaces.
     • 2.4.1. Subspace defined by equations.
     • 2.4.2. Matrix.
     • 2.4.3. Row-spaces.
     • 2.4.4. Caracterization of the polynomial vectorspace.

     • 2.4.5. Subspaces exercises.

       
       

1. Introduction.

In the past post; Vector Spaces, we've introduced that a vector space $(K,V,·)$ is the algebraic structure resulting from using a field $K$ to weigh the compositions of an abelian group $V$, through a field action $·$. Meaning that a vector space is a triple $(K,V, ·)$ where:

• $K$ is a field.
• $V$ is an abelian (commutative) group.
• $· : K \times V \to V$ is a field action, which acts over $V$ using $K$'s elements to scale vectors, making families of proportional vectors.

The resulting compositions are what we call linear combinations, independent contributions of the group's elements mediated by the field's scalars:

We presented some examples of sets which, with proper operations involved, are examples of vector spaces:

• The $n$-tuples space: $F^n$
• The space of $m \times n$ matrices: $M_{m \times n}(F)$
• The space of functions from a set $S$ to a field $K$: $K^S$

• The space of polynomial functions over a field K

  
  

We also developed, surreptitiously, the affine space structure, which is the way in which we use vectors to study a non-empty set in a simply transitive way, and we defined the arrow concept as the segment that connects two points of the affine space with magnitude, direction and orientation, which captures the displacement from the point at the base of the segment to the one lying at the end of the arrow.

We also demonstrated that if you fix one point $O$ and consider the family of all possible arrows with base at $O$, then that family has the structure of a vector space, which leaves us with a geometric intuition of what a vector is.

Then, with this information as a starting point, let's develop the fundamental properties of vector spaces.

2. Subspaces.

2.1. Definition and characterization.

In this section we shall introduce some of the basic concepts in the study of vector spaces.

Let $V$ be a vector space over the field $F$. A subspace of $V$ is a non-empty subset $W$ of $V$ which is itself a vector space over $F$ with the operations of vector addition and scalar multiplication on $V$.

Let's observe that, from the axioms of vector spaces, if $W$ is a vector space such $W \subset V$, then:

• Closure relative to linear combinations:

  \[v,u \in W \implies (\alpha v + \beta u \in V \ \forall \alpha, \beta \in F)\]

• Contains zero vector: $0_V \in W$, or $0_W = 0_V$. Remember that $V$ being a vector space means that $(V,+)$ is an abelian group, so $(W,+)$ is an abelian subgroup of $V$ and it inherits $0_V$ from $V$ by the uniqueness of this element. (Observe that this also extends to associativity and inverses).

  
  

Let's take a characterization for any non-empty subset $W \subset V$ to be a vector space. If $V$ is a $K$-vector space, then:

Let's reconstruct the structure:

• $(W,+)$ is an abelian subgroup of $(V,+)$.

  Since $W \neq \varnothing$ (by the premise), we can consider $u,v \in W$, then: $(-1)u + v = v - u \in W$, so it is a subgroup of $V$. Let's see that $W$ also inherits commutativity from $V$ so it is an abelian subgroup.

  
  

• $· |_W: K \times W \to W$ is a field action:

  • Observe that $-1 u + u = 0 \in W$, thus $1 u + 0 = u \in W$ so the unit in $F$ doesn't change the vector and it falls inside $W$.

  • Take some $v \in W \implies \alpha v + 0 = \alpha v \in W \implies \beta (\alpha v) + 0 = \beta (\alpha v) \in W$. For the same reason $(\alpha \beta)v \in W$. Then observe that in $V$, by associativity, is $\beta (\alpha v) = (\beta \alpha)v$ so both elements are equal in $V$ and the same in $W$ (by unicity of the inverse), thus $\beta (\alpha v) = (\beta \alpha)v$

  • Take $(\alpha + \beta)v + 0 = (\alpha + \beta)v \in W \subset V$, then in $V$ is $(\alpha + \beta)v = \alpha v + \beta v$ so, for the same argument as above, is $(\alpha + \beta)v = \alpha v + \beta v$ in $W$.

  • Take $\alpha v + 0 \in W \implies \exists w = \alpha v \in W$, take now other $u \in W$, then $\alpha u + w \in W$, again in $V$ is $\alpha u + w = \alpha u + \alpha v = \alpha(u + v)$ and again is $ \alpha(u + v) = \alpha u + \alpha v$ in $W$.

  
  

2.2. Example of subspaces.

Let's consider some well-known examples of subspaces. Consider some $V$ a $K$-vector space, then:

1. $V$ is a subspace of $V$.

2. $\Set{0} \subset V$ is the zero subspace of $V$; $\alpha 0 + 0 = 0 \in \Set{0} \quad \forall \alpha \in K$ so $\Set{0}$ is a subspace of $V$.

3. $A:= \Set{x \in K^n \mid x_1 = 0}$ is a subspace of $K^n$, check that always $\alpha x + y \in A$, trivially.

   But let's observe that $B := \Set{x \in K^n \mid x_1 = 1 + x_2}$ (the solution of a non-homogeneous equation) does not satisfy the rule, take some $\alpha$ and observe that:

   \[\alpha x + y = (\alpha (1 + x_2) + 1 + y_2), \alpha x_2 + y_2,..., \alpha x_n + y_n)\]

   Then, $\alpha + 1 + \alpha x_2 + y_2 = 1 + \alpha x_2 + y_2 \iff \alpha=0$.

   So we have that $\forall \alpha \neq 0 \quad (x,y \in B \implies \alpha x + y \notin B)$

   
   

4. The space of polynomial functions over $K$;

   \[\operatorname{Pol}(K, K) := \left\{\, f \in K^K \ \middle|\ \exists n \in \mathbb{N}_0,\ \exists (\alpha_0, \ldots, \alpha_n) \in K^{n+1} : \forall s \in K,\ f(s) = \sum_{i=0}^{n} \alpha_i\, s^i \,\right\}\]

   Is a subspace of $K^K$.

   Observe that, if we get $f,g \in Pol(K,K)$, then is: $ f(s) = \sum_{i=0}^{n} \alpha_i\, s^i$, $g(s) = \sum_{i=0}^{m} \beta_i\, s^i$, consider is $n \leq m$, then we can expand $f(s)$ to $\sum_{i=0}^{m} \alpha_i\, s^i = \sum_{i=0}^{n} \alpha_i\, s^i +\sum_{i=n+1}^{m} 0\, s^i$ and then is:

   \[(\gamma f + g)(s) = \gamma f(s) + g(s) = \gamma \sum_{i=0}^{m} \alpha_i\, s^i + \sum_{i=0}^{m} \beta_i\, s^i =\] \[\sum_{i=0}^{m} \gamma\alpha_i\, s^i + \sum_{i=0}^{m} \beta_i\, s^i = \sum_{i=0}^{m} (\gamma\alpha_i + \beta_i) s^i = \sum_{i=0}^m \varphi_i \, s^i \in Pol(K,K)\]

   
   

5. Consider $S:=\Set{(a_{ij}){ij} \in M_n(K) \mid (a{ij}){ij} = (a{ij})_{ji}}$, the set of all symmetric matrices form a subspace of $M_n(K)$

   Take, $A,B \in S$, then let's see that $\alpha A + B = \alpha (a_{ij}){ij} + (b{ij}){ij} = (\alpha a{ij} + b_{ij})_{ij}$, then let's see that this matrix is in $S$ by observing that:

   \[A \in S \implies a_{ij} = a_{ji} \quad \forall i,j \implies \alpha a_{ij} = \alpha a_{ji} \quad \forall i,j \implies \alpha A \in S\]

   Now, observe that:

   \[A,B \in S \implies \begin{cases}a_{ij} = a_{ji} \quad \forall i,j \\ b_{ij} = b_{ji} \quad \forall i,j \end{cases} \implies a_{ij} + b_{ij} = a_{ji} + b_{ji} \quad \forall i,j \implies A+B \in S\]

   Ultimately concluding $\alpha A + B \in S$, so $S$ is a subspace of $M_n(K)$

   
   

6. Consider $M_n(\mathbb{C})$, then we say that $A \in M_n(\mathbb{C})$ is Hermitian or self-adjoint if $(a_{ij}){ij} = \overline{(a{ij})_{ji}}$, meaning that is equal to the transverse of the conjugate.

   Then, the set $H := \Set{A \in M_n(\mathbb{C}) \mid (a_{ij}){ij} = \overline{(a{ij}){ji}} }$ is not a subspace of $M_n(\mathbb{C})$ check that the diagonal elements impose $(a{ii}){ii} = \overline{(a{ii})_{ii}}$, since the entries are complex numbers, a complex number is equal with his conjugate when is real.

   Consider some $A \in H$ and observe that $iA \notin H$, since the diagonal is not real and is not an Hermitic matrix, so our criteria $\alpha A +B \in H \quad \forall A,B \in H, \alpha \in \mathbb{C}$ doesn't apply.

   How ever it does applies when $\alpha \in \mathbb{R}$, observe that $(\mathbb{R},(H,+), ·)$ is a $\mathbb{R}$-vector space.

   
   

7. The solution space of a system of homogeneous linear equations over $K$

   Let be $A \in M_{m\times n}(K)$, then consider $W := \Set{X \in M_{n \times 1}(K) \mid AX = 0}$ that's not empty, let's remember that:

   \[AX = \begin{pmatrix} a_{11} \cdots a_{1n} \\ \vdots \quad \quad \quad \vdots \\ a_{m1} \cdots a_{mn} \end{pmatrix} \begin{pmatrix} x_1 \\ \vdots \\ x_n\end{pmatrix} = \begin{pmatrix} \sum_{i=1}^n a_{1i}x_i \\ \vdots \\ \sum_{i=1}^n a_{mi}x_i \end{pmatrix}\]

   Then, $X \in W \implies \sum_{i=1}^n a_{ji}x_i = 0\quad \forall j \leq m$

   
   

   Let's now consider $\alpha X + Y$, then:

   \[A[\alpha X + Y] = \begin{pmatrix} a_{11} \cdots a_{1n} \\ \vdots \quad \quad \quad \vdots \\ a_{m1} \cdots a_{mn} \end{pmatrix} \begin{pmatrix} \alpha x_1 + y_1 \\ \vdots \\ \alpha x_n + y_n\end{pmatrix} = \begin{pmatrix} \sum_{i=1}^n a_{1i}(\alpha x_i + y_i) \\ \vdots \\ \sum_{i=1}^n a_{mi}(\alpha x_i + y_i) \end{pmatrix}\]

   Observe that, for being $X,Y \in W$:

   \[\sum_{i=1}^n a_{mi}(\alpha x_i + y_i) = \alpha \sum_{i=1}^n a_{ji} x_i + \sum_{i=1}^n a_{ji}y_i = 0 \quad \forall j \leq m\]

   So, $\alpha X +Y \in W$.

   
   

2.3. Combination of vector spaces.

Let's now see that the intersection of vector subspaces is a subspace, the spanned vector space of a subset of vectors is at the same time the intersection of all the subspaces that contains the subset and the set of all linear combinations of the vectors of the subset.

The the addition or sum of subspaces is a subspace

2.3.1. Intersection of subspaces.

Let $V$ be a vector space over the field $K$. The intersection of any collection of subspaces of $V$ is a subspace of $V$.

Consider $V$ a $K$-vector space, then $\Set{W_i}$ a family of subspaces of $V$, consider $\bigcap_i W_i$. Then, observe that for being each $W_i \leq V \implies 0 \in W_i \quad \forall i \implies 0 \in \bigcap_i W_i \neq \varnothing$. Then, consider:

And thus, $\bigcap_i W_i \leq V$

This result gains its importance from the fact that the subspace generated by a subset $S \subset V$ is the intersection of all the subspaces of $V$ that contains $S$, this is in fact the smallest subspace that contains $S$.

2.3.2. Spanned subspace by a set.

Spanned subspace definition

Consider again a $K$-space, $V$, and a subset $S$ of $V$.

Then, the subspace spanned by $S$ is the intersection of all those subspaces of $V$ which contains $S$, which, as we see above, is a vector space.

When $S$ is a finite set of vectors, $S:=\Set{\alpha_1,…, \alpha_n}$, we shall simply call $W$ the subspace spanned by the vectors $\alpha_1, \alpha_2, …, \alpha_n$,

Spanned subspace caracterization

Be $S$ a non-empty subset of a $K$-vector space $V$, then the spanned subspace of $S$, $W$ is the set of all the linear combinations of $S$'s vectors.

• First, let's see that obviously $W \neq \varnothing$: $S \neq \varnothing \implies \exists u \in S$, then $u = 1u \in W \implies W \neq \varnothing$. Lastly, just consider $u,v \in W$, then $\alpha u + v \in W$ so $W \leq V$.

  Let's make an observation, $\varnothing$ vacuosly satisfies the closure on linear combinations, there are no $u,v$ to test so is a null antecedent implication. The fact which separates $W$ from this degenerated case is the fact that $W \neq \varnothing$, meaning that for a vector space is unconditional to be non-empty, linear combination closure is not enough.

  
  

• Let now be $\Set{W_i \leq V \mid S \subseteq W_i}_{i \in I}$ the family of subspaces of $V$ that contains $S$ let's see that in fact is $W = \bigcap_i W_i$ by demonstrating each set contains the other.

  • $W \subset \cap_i W_i$

    If $w \in W \implies \exists u,v \in S : \alpha u + \beta v = w$. Since $S \subset \cap_i W_i \implies u,v \in \cap_i W_i$ and, since each $W_i \leq V$ and the intersection of vector spaces is a vector space then any linear combination is in $\cap_i W_i$, meaning $w = \alpha u + \beta v \in \cap_i W_i$

    
    

  • $\cap_i W_i \subset W$

    Let's observe $W$ is the subspace of all the linear combinations of $S$ elements, is contained in each $W_i$ because every vector space contains the linear combinations of the elements of $S$ for being vector spaces.

    Thus, observe that $W$ is the minimum element of the collection which, coincides by definition with his ínfimum which is by definition the spanned vector space this is, the intersection $\cap_i W_i$.

    
    

2.3.3. Sum of subsets of vector spaces.

If $S_1,S_2,\cdots, S_k$ are subsets of a vectror space $V$, then the set of all sums:

Now consider: $W_i \leq V : i = 1,2…,k$, then the set $\sum_{i=1}^k W_i$ is immediately a subspace. Observe that this spaces is the space spanned by the set $\bigcup_i W_i$.

2.4 Example of combinated vector spaces.

2.4.1. Subspace defined by equations.

Let $K$ be a subfield of the field $C$ of complex numbers. Then suppose:

Let's observe that the spanned vector space by $\Set{\alpha_1,\alpha_2, \alpha_3}$, $W$, is the set of al the lineal combinations of $\alpha_1, \alpha_2$ and $\alpha_3$. Hence, a vector

Observe that this subspace $W$ can be described using the equations as the solution set of the linear equation system:

2.4.2. Matrix.

Let be $K$ a subfield of the field $\mathbb{C}$ of complex numb ers, and let $V$ be the vector space of all $2 \times 2$ matrices over $K$.

Let consider be the subset of $V$ consisting of all matrices of the form:

Observe that: $V = W_1 + W_2$, or, in other terms, the linear combinations of the elements of the two spaces can generate any other element in $V$:

Also, the intersection subspace is:

2.4.3. Row-spaces.

Let $A \in M_{m \times n}(K)$. Then, the row vectors of $A$ are vectors $\alpha \in K^n$; $A := (\alpha_1, \ldots, \alpha_n)$.

In this context we call the row-space of $A$ to the spanned vector space, $W \leq K^n$ of the elements $\Set{\alpha_i \in K^n \mid A = (\alpha_1,\ldots, \alpha_n)}$.

2.4.4. Caracterization of the polynomial vectorspace.

Consider:

Then, consider now $ S \subset Pol(K,K)$ defined as:

Note that the spanned vector space of $S$ is $Pol(K,K)$.

2.4.5. Subspaces exercises.

2.4.5.1. Subspaces of $\mathbb{R}^n$.

1. $\Set{\alpha \in \mathbb{R^n} \mid a_1 \geq 0}$

   Naturally isn't a vector space. Observe that trivially $A \neq \varnothing$ and $-1·\alpha + 0 = -\alpha \notin A$, since $a_1 \geq 0 \implies -a_1 \leq 0$.

   
   

2. $\Set{\alpha \in \mathbb{R^n} \mid a_1 + 3a_2 =a_3}$

   We already covered this set in $2.2-3/7$, the solution set of any homogeneous linear equation system is a vector space.

   
   

3. $\Set{\alpha \in \mathbb{R^n} \mid a_2 =a^2_1}$

   Let's see that this equation is not linear, thus, it do not respects linear combinations and is not a subspace.

   Take for example $\alpha = (1,1,0)$ and consider the linear combination $\alpha + \alpha$ which not satisfies the condition.

   
   

4. $\Set{\alpha \in \mathbb{R^n} \mid a_1a_2=0}$

   Clearly is not a subspace, consider again: $\alpha = (1,0,1)$ and $\beta = (0,1,0)$ and his linear combination $\alpha + \beta$ which, again, do not satisfies the condition.

   
   

5. $\Set{\alpha \in \mathbb{R^n} \mid a_2 \in \mathbb{Q}}$

   Is not a subspace, take $\alpha = (0,1,0)$, then $\sqrt{2}\alpha = (0,\sqrt{2},0)$ do not satisfies the condtion.

   
   

2.4.5.2. Function subspaces.

Consider again $\mathbb{R}^{\mathbb{R}}:=\Set{f \in \mathcal{P}(\mathbb{R} \times \mathbb{R}) \mid \forall x \in \mathbb{R} \ \exists ! y \in \mathbb{R}: (x,y) \in f}$, with the operations:

Then, which of the following sets of functions are subspaces of $\mathbb{R}^{\mathbb{R}}$

1. $\Set{f \mid f(x^2) = f(x)^2}$

   Let's observe immediately that $(\alpha f)(x^2) = \alpha f(x^2)= \alpha f(x)^2 \neq (\alpha f)(x)^2 = \alpha^2 f(x)^2$ for any $\alpha \neq 0$, so is not a vector subspace.

   
   

2. $\Set{f \mid f(0) = f(1)}$

   Check that $(\alpha f + g)(0) = \alpha f(0) + g(0) = \alpha f(1) + g(1) = (\alpha f + g)(1)$, so is a subspace.

   
   

3. $\Set{f \mid f(3) = 1 + f(-5)}$

   Check that

   \[(\alpha f + g)(3) = \alpha f(3) + g(3) = \alpha (1 + f(-5))+ 1 + g(-5) = (\alpha f + g)(-5) + 1 + \alpha\]

   Thus, the condition is not meeted for any $\alpha \neq 0$.

   
   

4. $\Set{f \mid f(-1) = 0}$

   Check that $(\alpha f + g)(-1) = \alpha f(-1) + g(-1) = 0$, so is a subspace.

   
   

5. $\Set{f \mid f \text{ is continuous}}$

   Naturally yes.

   
   

2.4.5.3. Spanned vector.

Consider the vector $\alpha = (3,-1,0,-1)$, is spanned by the vectors $u = (2, -1, 3, 2), v = (-1, 1, 1, -3), w = (1, 1, 9, -5)$?

Consider $W$ the vector subspace spanned by the set $S := \Set{u,v,w}$. Then, the statement ask if $\alpha \in W$.

By $2.3.2$, we do know that the spanned vector subspace, $W$ is the intersection of all the subspaces that contains $S$ which coincides with the set of all the linear combinations of the vectors of $S$ which is a subspace it self.

Then, $\alpha$ is an element of $W$ only if is a linear combination of $u,v,w$, formally:

Taking coordinate to coordinate, we can form the following equation system on $\mathbb{R}$:

2.4.5.4. Set of spanned subspace.

Consider $W \leq \mathbb{R}^5$ of those $(x_1,x_2,x_3,x_4,x_5) \in \mathbb{R}^5$ satisfying the following equation system:

Find a finite set of vector that spans $W$.

We have to find some set $S$ such the set of the linear combinations of his elements coincides with $W$. Leveraging the exercise before, giving some $\alpha \in \mathbb{R}^5$, we have to find a finite some of vectors $u_1,u_2, \ldots, u_n$ such the equation system on the escalars $t_1,\ldots, t_n \in \mathbb{R} : \alpha = \sum_{i=1}^n t_i u_i$ is equivalent to the one gived by the exercise.

First, let's consider the $RREM$ form of the matrix previous matrix:

Which leave us with the system:

Since $M$ and $M'$ has row-equivalent asociated matrix, we do know that both are equivalents; $M \equiv M'$, so both shares the same solution set, $W$.

Ultimately, let's solve it, the solution of $M'$ is:

Calling; $x_3 = \alpha,\ x_4 = \beta,\ x_5 = \gamma$, then:

Let's observe that:

Hence, any vector of $W$ is a linear combination of $(−2/3​,0,1,0,0), (0,−1,0,1,0), (1,2,0,0,1)$

In summary, the solution set of a homogeneous linear system is a vector subspace. Writing the solution in its parametrized form yields the vectors that span the subspace.

2.4.5.5. Matrix subspaces.

Let be $M_{n \times n} (K) : n \geq 2$, which of the following sets $\Phi_n$ are subspaces?

1. $ \Set{A \in M_{n \times n}(K) \mid \exists A^{-1}}$

   Let's observe that if we consider $A \in \Phi_n \implies -A \in \Phi_n$ but $A + -A = 0 \notin \Phi_n$, hence is not a subspace.

   Observe that, we could quicker check that isn't a subspace because is not a vector space since it not contains $0$.

   
   

2. $\Set{A \in M_{n \times n}(K) \mid \nexists A^{-1}}$

   Let observe that we can consider:

   \[A = \begin{pmatrix}1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}, \quad B = \begin{pmatrix}0 & 0 & 0\\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}\]

   Both of them aren't invertible since they are already their $RREM$ which not coincides with $I_3$ but $A+B = I_3 \in \Phi_n$

   
   

3. $\Set{A \in M_{n \times n}(K) \mid AB = BA}$, for some fixed $B$.

   Observe that taking $A, C \in \Phi_n$ then:

   \[(\alpha A + C)B = \alpha AB + CB = B(\alpha A) + BC = B(\alpha A + C)\]

   Thus, it conforms a subspace

   
   

4. $\Set{A \in M_{n \times n}(K) \mid A^2 = A}$

   Observe that if $A,B \in \Phi_n$, then:

   \[(\alpha A + B)^2 = \alpha^2 A^2 + B^2 + \alpha AB + B \alpha A = \alpha^2A + B + \alpha AB + \alpha BA \neq \alpha A + B\]

   
   

2.4.5.6. Subspaces of $\mathbb{R}$.

1. Prove that the only subspaces of $\mathbb{R}$ are $\mathbb{R}$ and the zero subspace.

   That $\mathbb{R} \leq \mathbb{R}$ and $\Set{0} \leq \mathbb{R}$ comes immediately from $2.2.$

   Note directly that if there is some subspace $V \leq \mathbb{R}$, then $V$ contains all the linear combinations of himself, so consider some $a \in V \implies 1 = (a/a) \in V$, then since we can multiply $1$ by any scalar, we can reach any element of $\mathbb{R}$ inside of $V$ so $V = \mathbb{R}$.

   
   

2. Prove that a subspace of $\mathbb{R}$ is $\mathbb{R}^2$, or the zero subspace, or consists of all scalar multiples of some fixed vector in $\mathbb{R}^2$.

   First, again that $\mathbb{R} \leq \mathbb{R}$ and $\Set{0} \leq \mathbb{R}$ comes immediately from $2.2.$

   Next, the lines on $\mathbb{R}^2$ are subspaces is also clear, take some not null $v \in R^2$ then any scaled element of $v$ is a linear combination of $v$ so the line contains all the linear combinations of his elements.

   Lastly, let's suppose two elements $u,v \in \mathbb{R}^2: \nexists \alpha \in \mathbb{R} : u = \alpha v$. Then, this two elements doesn't belong to the same line. Suppose $u = (a,b), v = (c,d)$, hence observe that for some $\alpha, \beta \in \mathbb{R}$ is $\alpha u + \beta v = (\alpha a+ \beta c,0) + (0, \alpha b+ \beta d)$, observe that we can form the following equation system:

   \[\begin{cases} \alpha a + \beta b = 1 \\ \alpha c + \beta d = 1\end{cases}\]

   Solving it we get:

   \[\alpha=\dfrac{d-c}{ad-bc},\qquad \beta=\dfrac{a-b}{ad-bc}\]

   Observe that the solution has sense, since $u,v$ are no proportionals, $a,c$ and $b,d$ can't be zero at the same time so $ad - bc \neq 0$

   And:

   \[\dfrac{d-c}{ad-bc} u + \dfrac{a-b}{ad-bc} v = (1,0) + (0,1) = e_1 + e_2\]

   Observe that (without the need to invocate orthogonality or basis), we can ensure that, being $w = (x,y) \in \mathbb{R}^2$, then is:

   \[w = \dfrac{x(d-c)}{ad-bc} u + \dfrac{y(a-b)}{ad-bc} v \quad \forall w \in \mathbb{R}^2\]

   Meaning, with two non-proportional vectors, we can reach any other element in $\mathbb{R}^2$ thus the spanned subspace $V$ of the subset $S := \Set{u,v \mid \nexists \alpha \in \mathbb{R} : u = \alpha v} \subset \mathbb{R}^2$ is $\mathbb{R}^2$ it self.

   Thus, observe that for any $W \leq V$, then if $W$ contain at least two non-proportional vectors then $W = \mathbb{R}^2$, since a vector space contain all the linear combination of his vectors, if do not then is either a line or the zero subspace.

   
   

3. Can you describe the subspaces of R3?

   Extending the argumentation in $2.$ is the zero subspace, those subspaces for which exists at least three independant vectors (one vector which is not linear combination of other two) which spans $\mathbb{R}^3$, then those subspaces with at least two non-proportional vectors which are planes and those with all his vectors proportionals which are lines.

   
   

2.4.5.7. Union of subspaces.

Let $W_1, W_2 \leq V$ be subspaces of a vector space such that the set-theoretic union of $W_1$ and $W_2$ is also a subspace. Prove that one of the spaces $W_i$ is contained in the other.

Let's consider some interesting visual approach about the union of subspaces. Consider $\mathbb{R}^2$, then consider two non proportional vectors and the spanned subspaces of each of them, which are two distinct lines crossing in the origin. Is easy to see that a linear combination of the vectors of each line can fall out of each line since the subspace containing two non-proportional vectors coincides with $\mathbb{R}^2$. This means that the union of the two lines doesn't contains all his linear combinations so is not a subspace. For the union to be a subspace, each linear combination should end up as vector of one of the lines wich is pretty much to say that is proportional to one of the vectors of the linear combination but observe that this also implies that the third vector is also proportional so to the two lines are the same.

Then, generalizing:

Then, suppose $w \in W_i \implies v = w - u \in W_i$, observe that this means that we can't consider simulatenously $u \in W_i \setminus W_j$ and $v \in W_j \setminus W_i$ since the last afirmation would be false.

2.4.5.8. Even and odd function subspaces. Direct Sum.

Consider $\mathbb{R}^\mathbb{R}$ and the following subsets:

\(A := \Set{f \in \mathbb{R}^\mathbb{R} \mid f(-x) = f(x)}\) \(B := \Set{f \in \mathbb{R}^\mathbb{R} \mid f(-x) = -f(x)}\)

Then, prove that:

1. Both subsets are subspaces of $\mathbb{R}^\mathbb{R}$

   Observe that:

   \[(\alpha f + g)(-x) = \alpha f(-x) + g(-x) = \begin{cases} \alpha f(x) + g(x) = (\alpha f + g)(x) \quad \forall f,g \in A \\ -\alpha f(x) - g(x) = -(\alpha f + g)(x) \quad \forall f,g \in B \end{cases}\]

   So both contains the linear combinations of his own elements so their are subspaces.

   
   

2. $A + B = \mathbb{R}^\mathbb{R}$

   Let's suppose that $f \in \mathbb{R}^\mathbb{R}$, then let's suppose also $g \in A$ and $h \in B$ such $f = g + h$, observe that this functions verifies at the same time:

   \[\begin{cases}f(x) = g(x) + h(x) \\ f(-x) = g(-x) + h(-x) = g(x) - h(x) \end{cases}\]

   In this context is, combining both equatlities we get:

   • $g(x) = \frac{1}{2}[f(x) + f(-x)]$
   • $h(x) = \frac{1}{2}[f(x) - f(-x)]$

   So is:

   \[f(x) = \frac{1}{2}[f(x) + f(-x)] + \frac{1}{2}[f(x) - f(-x)]\]

   
   

3. $A \cap B = \Set{0}$

   Observe that this is almost immediate, suppose some common function from $A \cap B$, then:

   \[f(x) = \frac{1}{2}(f(-x) - f(-x)) = 0 \quad \forall x \in \mathbb{R}\]

   
   

Observe that we have demonstrated that two disjoint subspaces $A,B \leq \mathbb{R}^\mathbb{R}$ sum the total space:

Observe that this allows to dispose every vector of $\mathbb{R}^\mathbb{R}$ in terms of two componentes of $A$ and $B$.

2.4.5.9. Property of direct sum.

Let $W_1, W_2$ subspace such are direct sum of $V$. Prove that for each vector $\alpha \in V$ there are unique vectors $\alpha_1 \in W_1$ and $\alpha_2 \in W_2$ such that $a = \alpha_1 + \alpha_2$.

Take $a \in V$ and suppose that is $a = \alpha_1 + \alpha_2 = \beta_1 + \beta_2 : \alpha_i, \beta_i \in W_i$ with $i = 1,2$, then observe that it cannot be $\alpha_i - \beta_j = 0$ since that would imply $\alpha_i \in W_j$ or $\beta_j \in W_i$ contradicting the premise that both subspaces do not share not-null vectors, so it can only be $\alpha_i - \beta_i = 0 : i = 1,2$, so both vectors are the same exact vector.

3. Bases and Dimension.

3.1. Conceptal introduction.

Let's first introduce what a basis is, a basis is the minimum set of a vector space which contains all the information the vector space is capable to express. Meaning that is the minimum set of vectors that spans the vector space; any vector is reachable trough a linear combination of the elements of a basis and at the same time this set is minimum, any subset of a basis is uncapable to span the vector space.

The dimension of the vector space is the number of "degrees of freedom" that a vector space has; the number of independent vector required to specify an arbitrary vector materialized in the cardinal number of the basis set.

This two concepts has a fundamental importance; the dimension is the fundamental invariant that clasifies vector spaces, and the basis what transform linear algebra (operation with vectors) in matrix computations.

3.2. Linearly dependent/independent.

Let first introduce the concept of dependant and independant linearity, which basically determines wheter a vector is owned by the spanned vector space of a set of vectors.

Let be $V$ a $K$-vector space and $S \subset V$. Then:

• $S$ is said to be linear dependant if and only if:

  \[\exists \alpha_1,\ldots, \alpha_n \in S \text{ distinct}, \exists c_1,\ldots, c_n \in K \text{ not all 0} : \sum_{i=1}^n c_i\alpha_i = 0\]

  
  

• $S$ is said to be linear independant if and only if is not dependant, negating the above we get

  \[\forall \alpha_1,\ldots, \alpha_n \in S \text{ distinct}, \forall c_1,\ldots, c_n \in K \text{ not all 0} : \sum_{i=1}^n c_i\alpha_i \neq 0\]

  Observe that we can convert the statement by quitting the "not all 0" obtaining:

  \[\forall \alpha_1,\ldots, \alpha_n \in S \text{ distinct}, \forall c_1,\ldots, c_n \in K : \sum_{i=1}^n c_i\alpha_i = 0 \iff c_i = 0 \quad \forall i \in [n]\]

  
  

Let's observe this more carefully, check something interesting, a subset of vectors of a vector space is said to be dependant when it can reference the zero vector, $0$ through a finite, non-trivial, linear combination.

Take for example two vectors, $u,v \neq 0 \in V$, then if $\exists \alpha,\beta\neq 0 \in K : \alpha u + \beta v = 0 \iff u = \frac{-\beta}{\alpha}v$, both are proportionals and we can retrieve one from the other without involve any other vector, so we can say they contains the same information, both are the same type of vector so to speak. Thus, let's suppose that $u,v$ can't reference null vector and consider $w = tu + lv \neq 0 : t,l \neq 0 \in K$, then observe that this three vectors can again reference the zero vector, $tu + lv -w = 0$ through a finite non-trivial linear combination, qwe can think in $w$ as a proportional vector to a combination of $u,v$, thus $w$ and $u,v$ contains the same information despite scaling and observe that the fact that three vectors can reference through a non-trivial linear combination states the same, each of them is proportional to a combination of the other ones.

Thus, information redundancy is the essense of linear dependence, a set of vectors is said to be linear dependent when at least one of them carries complete redundant information, in the sense that the whole set can express a part and the counterpart leading to a cancelation on the zero vector. Linear independence is just the negation of this statement, a set of vectors is said to be linear independent when all vectors contribute with some non-replicable information, or, in other terms, substract any vector in the set implies to loose information.

Let's see some easy consequences of the definition.

1. Any set which contains a linearly dependent set is linearly dependent.
2. Any subset of a linearly independent set is linearly independent.

3. Any set which contains the 0 vector is linearly dependent; for $1 · 0 = 0$.

   
   

3.2.1. Examples.

Consider $K$ a field, then in $K^3$ the vectors:

Satisfies:

So are linear dependant. Note that $\Set{\alpha_1, \alpha_2, \alpha_3}$ are linear dependant as well, hence, any set containing this vectors is also linear dependant.

The vectors:

Are linear independant.

3.3. Basis definition and examples.

Let $V$ be a vector space. A basis, $\mathcal{B}$, of $V$ is a linearly independent set of vectors in $V$ which spans the space $V$. The space $V$ is finitedimensional if it has finite basis.

Let's observe here that we are giving continuity to the dependence/independence frame but we are adding generators sets in the process. A basis is nothing more that the combination of two objects:

• A generator set, this is; a set which contains all the information that the vector space can express.

• A independent set, a set from which you can't free any element without loose information.

This two objects gives us a compress notion; a basis is the minimal generator set of a vector space; substracting a vector make a piece of information to get lost, so the span doesn't hold. This will be explain in a later theorem.

3.3.1. Standard basis of $K^n$.

Let $K$ be a field and in $K^n$, consider the subset:

At first, observe that for any $x = (x_1,x_2,\ldots,x_n) \in K^n$ is:

So $\mathcal{B}$ spans $K^n$ and also is linear independent as we see in the particular case with $K^3$. So $\mathcal{B}$ is a basis of $K^n$, particularly called standard basis.

3.3.2. Invertible matrix and basis.

Let's consider some $P \in M_n(K)$ invertible. We do know that $P \simeq I_n$, which means that no row in $P$ can be zeroed through a linear combination of the other rows in $P$ so the set of the rows in $P$ forms a linear independent set. Observe that, in more simple terms, be $X \in K^{n\times 1}$, then, by $6.3.3$ in Linear Equations:

Thus, $PX = \sum_{i=1}^n x_iP_i = 0 \iff x_i = 0 \quad \forall i \in [n]$, meaning that $\Set{P_1, \ldots, P_n}$ is a linear independent set.

And also, as we see above, it spans any column on $K^{n \times 1}$ so $\Set{P_1, \ldots, P_n} \subset K^{n \times 1}$ is in fact a basis of $K^{n \times 1}$.

Let's observe that we are saying that in $K^n$, a basis and an invertible matrix is literally the same object. Is the naturall continuity to the conception of matrix seen as linear-information codified packeges viewed in Linear Equations.

The columns of any invertible matrix become a basis of the same matrix-dimension tuple vector space, in a way that the coordinates of any vector in terms of that basis becomes the solution of a linear equation system. Again from $6.3.3$ in the post above the solution is unique so:

3.3.3. Basis of non-squared matrix vector spaces.

Let $A \in M_{m \times n}(K)$ and let $S := \Set{X \mid AX = 0}$ the solution on the homogeneous system. Take the $RREM$, $R \simeq A$, then, $RX=0$ share the space solution.

Observe that since $R$ is $RREM$, it has $r$ non-zero rows, which allow to clear $r$ unknowns in terms of $n-r$ unknowns, be $J$ the set of the index of the uncleared unknowns, then $RX = 0$ is equivalent to the system:

All solution are retrieved by giving values to the dependent unknowns assigning arbitrary values to the independent unknows, those with and index in $J$.

Observe then that, be $j \in J$ then consider $E_j$ the solution by giving $x_j=1$ and $x_i=0$ for any other $i \in J$. Observe since the solutions are indeed linear combinations of the independent unknowns, then any arbitrary solution are is a linear combination of the family $\Set{E_j}_{j \in J}$ as described above and also is clear that are linearly independant so is a basis.

3.3.4. Infinite Basis.

Let's note first that an infinite basis do not introduce infinite sums, each vector gets obtained through a finite linear combination. Which is infinite is the generator engine, the basis it self which, algebraicly speacking, means that there is only no finite basis at all. The richness of infinite-dimensional theory (functional analysis, operators) lies not in the vector space structure itself, but in the additional structure superimposed upon it.

Take some subfield $W \subset \mathbb{C}$, and consider $Pol(W,W)$, remember:

Note then that, for any $f \in Pol(W,W)$, it takes the form:

Let's call $f_n(x) = x^n$, then the family $\Set{f_n}_{n \in \mathbb{N}}$ is a basis over $Pol(W,W)$.

First, is clear that it can generate any element of $Pol(W,W)$, but let's also see that they are independent. We do know that a set is independent if any linear combination of his elements is $0$, it would be sufficent to proove that for any $n$, the finite family $\Set{f_0,f_1,\ldots,f_n}$ is independent, then, for some $n$ is:

We assume that the reader knows that a polynomial of degree $n$ with complex coefficients cannot have more than $n$ distinct roots. It follows that $\alpha_0 = \cdots = \alpha_n = 0$.

3.5.

1. Definition and operations.

2. $\mathbb{C}$ as a field.

3. $\mathbb{R}$ in $\mathbb{C}$.

4. Real formulation of $\mathbb{C}$. Imaginary and Real part.

5. Geometric representation of $\mathbb{C}$.
   • 5.1. Axis.
   • 5.2. Absolute Value. Polar Form.
   • 5.3. Geometric Intuition of the operations in $\mathbb{C}$.
     • 5.3.1. Addition as a torsor.
     • 5.3.2. Multiplication as a rotation.
   • 5.4. Conjugate.
6. Important properties of Complex Numbers.
   • 6.1. Conjugate properties.
   • 6.2. Absolute Value properties.
   • 6.3. Cauchy-Schwarz inequality.
     • 6.3.1. Appendix: Inner product. Norm. Orthogonality.
     • 6.3.2. Real definition and conceptual explanation.
     • 6.3.3. Induced Norm.
     • 6.3.4. Proper inner product expression.
     • 6.3.5. Cauchy-Schwarz and angle between two vectors.
   • 6.4. Rudin's 1.35 theorem.

1. Definition and operations.

A complex number is an ordered pair of real numbers. By definition it is:

Observe that this means that:

We define two operations in the complex numbers, addition and multiplication as follows. Be $z_1,z_2 \in \mathbb{C}$, then:

2. $\mathbb{C}$ as a field.

These definitions of addition and multiplication turn the set of all complex numbers into a field in an algebraic sense, with $0^\ast = (0, 0)$ and $1^\ast = (1, 0)$.

Meaning that $(\mathbb{C},+)$, $(\mathbb{C}\setminus \Set{(0,0)}, \ ·)$ are compatible abelian groups.

It is worth mentioning the form of the inverse of a $z \in \mathbb{C}$:

It is easy to prove that $zz^{-1} = 1 \iff z \neq 0$ by operating.

3. $\mathbb{R}$ in $\mathbb{C}$.

Observe that if we take $z_1,z_2 \in \mathbb{C} : z_1 = (a,0), z_2 = (b,0)$, then $z_1,z_2$ behave exactly as real numbers:

Thus, we can identify each $\alpha \in \mathbb{R}$ with the complex $(\alpha,0) \in \mathbb{C}$.

Observe we can consider $\varphi : \mathbb{R} \to \text{Re}$ such that $\varphi(\alpha) = (\alpha,0)$. Note that $\varphi$ is a field homomorphism:

• $\varphi(\alpha + \beta) = (\alpha + \beta,0) = (\alpha,0) + (\beta,0) = \varphi(\alpha) + \varphi(\beta)$

• $\varphi(\alpha \beta) = (\alpha \beta,0) = (\alpha,0) (\beta,0) = \varphi(\alpha) \varphi(\beta)$

  
  

Also, it maintains the order; we define in $\text{Re} \subset \mathbb{C}$ the order: $(x,0) < (y,0) \iff x < y$.

Also, it is clear that $\varphi$ is bijective, thus it is an isomorphism of ordered fields, and we can identify $\mathbb{R}$ and $\text{Re}$ or, in other words: $\mathbb{R} \simeq \text{Re} \subset \mathbb{C}$ and $\mathbb{C}$ contains $\mathbb{R}$.

4. Real formulation of $\mathbb{C}$. Imaginary and Real part.

We define $i := (0,1) \in \mathbb{C}$, observe immediately that:

Or, making an abuse of notation: $i^2 = -1 \in \mathbb{R}$

Let's also observe that there is a real reformulation for any complex number using $i$ as a toehold.

Be $z = (x,y) \in \mathbb{C}$, then:

Or simply: $z = (x,y) = x + yi$

5. Geometric representation of $\mathbb{C}$.

5.1. Axis.

Let's observe a phenomenon: we had defined $\mathbb{C}$ basically as an extension of $\mathbb{R}$ to two dimensions, precisely:

Thus, let's try to represent this set in a coordinate system. A few minor considerations:

• Since we already saw that $\text{Re} := \Set{(x,0) \mid x \in \mathbb{R}} \simeq \mathbb{R}$, the abscissa axis must be the real line.

• Then, $\text{Im}:= \Set{(0,y) \mid y \in \mathbb{R}}$ is what we call the imaginary axis, and it is not a strict copy of $\mathbb{R}$ as $\text{Re}$ is; let's observe that it is not closed under the product:

  \[(0,y_1)(0,y_2) = [(y_1,0)(y_2,0)][(0,1)(0,1)] = (y_1y_2,0)(-1,0) = (-y_1y_2,0) \notin Im\]

Both axes form:

5.2. Absolute Value. Polar Form.

Let's take some $z$ as in the image above and observe that this is a two-dimensional vector, thus we can identify each $z := (x,y)$ with the vector $\overrightarrow{z}=(x,y)$ and leverage vector space properties and the trigonometric relations of the object.

• First, we call as absolute value of $z$ to the real number:

  \[|z| := (z\overline{z})^{1/2}\]

  
  

• Also, we can consider the angle of the vector with the abscissa axis, $\theta$:

  \[\theta := \arctan(y/x)\]

Observe that $z$ gets completely characterized by these two elements in what we call the polar form of the complex numbers:

5.3. Geometric Intuition of the operations in $\mathbb{C}$.

In other parts of this same post we asserted that the real numbers were positions on a line, and that this obliges us to interpret the operations of the field with a geometric nuance.

Thus, we stated that in $\mathbb{R}$ the addition was a translation and the product was a homothety. Let's now see how the operations we just defined act in the $Re \times Im$ plane.

5.3.1. Addition as a torsor.

In this case, the abelian group $(\mathbb{C},+)$ behaves in $Re \times Im$ making it a torsor. (It is desirable to take a look at the following post to understand the explanation below: vector spaces).

Recapitulating, we know that an affine space is the algebraic structure resulting from using vectors to study a non-empty set in a simply transitive way. Then, a torsor is something prior, more primitive; a torsor is a regular group action of (in this case an abelian) group over a non-empty set, basically an affine space that has forgotten the field's action over the vectors so these can't be scaled and, consequently, you can't scale distances.

We define the pair $(Re \times Im, +)$ being:

Let's see this is a regular group action:

• $A1$, taking $0 \in \mathbb{C}$, then: $(x,y) + 0 = (x,y) \quad \forall (x,y) \in Re \times Im$. So there is an identity element.

  
  

• $A2$, the associativity is immediate from the fact that $(\mathbb{R},+)$ is an abelian group:

  \[((x,y) + z)+t = (x+z_1,y+z_2) + t = ([x + z_1] +t_1,[y+z_2]+t_2) =\] \[(x + [z_1 +t_1],y+[z_2+t_2]) = (x,y) + (z + t)\]

  
  

• Free and Transitivity

  Let's take two points $x,y$ on $Re \times Im$, asking whether $\exists! z \in \mathbb{C} : x + z = y$ is equivalent to ask if the following equation in $\mathbb{R}$, given by the coordinates, has a unique solution:

  \[x_i + z_i = y_i \quad i = 1,2\]

  Since $(\mathbb{R},+)$ is an abelian group, we can assert that $z_i = y_i - x_i \in \mathbb{R} \quad i=1,2$ is a solution for each equation and is unique.

  In other words, we have that $\forall P,Q \in Re \times Im \ \exists ! z \in \mathbb{C} : P + z = Q$, so the group action $+$ is also regular and the structure $(Re \times Im, +)$ is a torsor.

  
  

Intuitively, we can think that any complex $t \in \mathbb{C}$ is a two-dimensional vector $\overrightarrow{t}$, then it can be identified with an arrow over a plane by selecting an origin, in this case; $0 \in \mathbb{C}$. Thus, leveraging the equivalence class of free vectors, doing $z + t$ is, geometrically, as if we take $\overrightarrow{t}$ and translate its base from $0$ to $z$, and then the point referenced by $z + \overrightarrow{t}$ in an affine space and $z+t$ are the same.

5.3.2. Multiplication as a rotation.

Let's recapitulate and see that we've defined $\mathbb{C}$ as an $\mathbb{R}$-vector space, a two-dimensional vector space, taking $\mathcal{B}:=\Set{e_1 = (1,0), e_2 = (0,1)}$ as the basis; basically we are referring to the fact that $\mathbb{C} = \text{span}(\mathcal{B})$ and any complex $z$ is a linear combination of these two vectors:

This is nothing new and we already worked with this concepts before.

Let's now fix some $w \in \mathbb{C}$ and consider $M_w(z) = wz$, observe that this application is linear, thanks to the commutative and distributive properties of $\mathbb{C}$:

So, as we know, considering $\mathcal{B}$, there exists some $2 \times 2$ matrix that characterizes $M_w$. Take that if $z = (z_x,z_y)_{\mathcal{B}} = z_x + iz_y \in \mathbb{C}$, then:

So, the columns of our matrix $T$ are where the product with $w$ sends $1$ and $i$, but we can get even further, taking $w = (w_x,w_y)_{\mathcal{B}}$, then:

Thus, $M_w(z) = T_{\mathcal{B}}·z = \begin{pmatrix}w_x &-w_y \ w_y & w_x \end{pmatrix}\begin{pmatrix}z_x \ z_y\end{pmatrix} $, but let's be careful for a moment and take a closer look. We do know, from the polar form, that for any complex number $z_x = |z|\cos\theta$ and $z_y = |z|\sin\theta$, meaning:

Observe that, using trigonometric identities:

Meaning that, essentially:

Observe that, to the original $z = |z|(\cos\theta_z + i · \sin\theta_z)$, the product $w·z$ has scaled it by $|w|$ and added the angle $\theta_w$ to $\theta_z$.

Thus the product of complex numbers is basically a homothety centered at $0$ with ratio $|w| \in \mathbb{R}$ composed with a rotation of $\theta_w$ about $0$.

5.4. Conjugate.

The conjugate is one of the most fundamental symmetries in $\mathbb{C}$. Given $z \in \mathbb{C}$, we define the conjugate of $z$, $\overline{z}$ as the complex number obtained by reflecting it across the real axis, $Re$:

According to the definition we have two equivalent representations. Given: $z \in \mathbb{C}$, then:

The crucial importance of the conjugate is that it captures the invariance for complex solutions to real polynomial equations, this is, if a complex number $z$ is a solution for some real polynomial, then $\overline{z}$ is also a solution.

6. Important properties of Complex Numbers.

We have introduced some important notions about complex numbers, let's mix them up in order to present important results about complex numbers:

6.1. Conjugate properties.

Let's start by saying that, although an algebraic proof will be provided, most of the properties of the conjugate take place by understanding that the real axis acts as a mirror reflecting what happens on each side of the axis onto the other side.

Thus, the conjugate of the addition is the addition of the conjugates and the conjugate of the product is the product of the conjugate:

• $\overline{z}+\overline{w}=(z_x - iz_y) + (w_x -iw_y) = (z_x + w_x) - i(z_y+w_y) = \overline{z+w}$

• $\overline{z} · \overline{w} = |z|_{-\theta_z}|w|_{-\theta_w} = |zw|_{(-\theta_z + - \theta_w)} = |zw|_{-(\theta_z + \theta_w)} = \overline{z·w}$

  
  

Also, observe that adding a complex to its conjugate basically collapses the value onto the real line, so:

• $z + \overline{z} = (z_x + z_x) + i(z_y - z_y) = 2Re(z)$

And, by the same argument, the subtraction eliminates the real part:

• $z - \overline{z} = z + - \overline{z} = (z_x - z_x) + i(z_y + z_y) = 2Im(z)$

  
  

Also, let's consider $z\overline{z}$, intuitively, we have that we are making over $\overline{z}$ a rotation that negates its own angle, thus it ends up in the positive part of the real axis:

• $z \neq 0 \implies z\overline{z} \in \mathbb{R}^+$

  Since, $z\overline{z} = |z\overline{z}|_{\theta -\theta} = |z\overline{z}|_0$

  
  

  

  
  

  Lastly, observe also that $z\overline{z} = |z|^2 \implies |z| = \sqrt{z\overline{z}}$.

  Thus $z \in \mathbb{R} \iff z = \overline{z}$

  \[|z| = \begin{cases} z \quad z \geq 0 \\ -z \quad z < 0\end{cases}\]

  
  

6.2. Absolute Value properties.

Note that, we have the following properties of the absolute value:

• $|z| \geq 0 \wedge (|z| = 0 \iff z = 0)$

• $|z| = |\overline{z}|$

• $|zw| = |z| |w|$

The three properties are immediate as we just see until now.

• Observe $-1 \leq \cos\theta \leq 1 \iff -|z| \leq |z| \cos\theta \leq |z| \implies | Re(z) | \leq |z|$

  
  

Lastly, let's consider two points $z,w \in \mathbb{C}$ and consider $z+w$. Then, draft $\triangle(0,z,z+w)$ and observe that the relation between $|z+w|$ and $|z|+|w|$ given by the cosine theorem is:

Meaning that $|z+w|$ is $|z|+|w|$ plus an angle correction; let's argue then:

Since, again $|\cos\theta| \leq 1$, then:

observe that this implication can be asserted since we are always dealing with the square of positive quantities; $|z| \geq 0 \quad \forall z \in \mathbb{C}$

6.3. Cauchy-Schwarz inequality.

6.3.1. Appendix: Inner product. Norm. Orthogonality.

Before defining the Cauchy-Schwarz inequality, let's talk a bit about inner products, which are, essentially, the structure that transforms linear algebra into geometry.

We do know, from this post that vectors from vector spaces are weighted elements from an abelian group through a field's action. They can compound among themselves to generate other vectors and they can be dilated as much as is convenient using the field's scalars, but they can't say much about themselves.

Observe that, through the notion of affine space we acquired a geometric notion about vectors as arrows; the family of all possible vectors emerging from a single origin has vector space structure. Then, intuitively as vectors, these arrows can reference any other arrow in the family and, furthermore, they can also serve as models for other arrows emerging from other points (free vector class) but they can't talk about properties relative to their own relative disposition or what properties remain invariant after a transformation (despite being the subject of these questions) without appealing to geometric statements living in the euclidean space. The only thing that two vectors can say about themselves is whether they are proportional or not, a binary geometrical statement.

This is where the inner product enters; it is the minimal mathematical object that injects this geometric kit onto vector spaces.

6.3.2. Real definition and conceptual explanation.

Consider $V$ an $\mathbb{R}$-vector space, an inner product over $V$ is an application: $\langle\cdot,\cdot\rangle:V\times V\to\mathbb{R}$, satisfying:

• $I1$ Linearity on first argument: $\langle x+y,z\rangle = \langle x,z\rangle + \langle y,z\rangle$ and $\langle \alpha x,z\rangle = \alpha\langle x,z\rangle$ such $\alpha \in \mathbb{R}$.
• $I2$ Symmetry: $\langle x,y\rangle = \langle y,x\rangle$, observe that this means that the linearity also applies to the second argument, it is bilinear.
• $I3$ Positive definite $\langle x,x\rangle\geq 0$ and $\langle x,x\rangle = 0 \iff x = 0$

Let's observe how a geometric structure distills from these axioms.

6.3.3. Induced Norm.

First, we consider that the root of the inner product of a vector with itself behaves in a special way, we define it as the norm of a vector:

Consider two proportional vectors; $x,y \in V: x = \alpha y, \ \alpha \in \mathbb{R}$, from the vector space perspective, $x$ is a deformed version of $y$, then we have that:

Meaning that the quotient of the norms of two proportional vectors expresses their proportionality factor. Let's consider now:

Thus, we can consider a proportional vector $e$ to $x$ with norm $1$ and say that the norm of any vector expresses the absolute value of the dilation factor relative to a proportional vector of unit norm:

How amplified or narrowed $x$ is expresses the same information as $e$. And this is what we have agreed to call "magnitude".

6.3.4. Proper inner product expression.

Let's now consider $x,y \in V$ non nulls, and consider the norm of $x+y$, then, the inner product $\langle x,y \rangle$ acquires the following form:

Concluding:

This is, in terms of magnitude, the inner product measures the defect of the squared magnitude of the composition relative to the addition of each individual squared magnitude of the combined vectors.

Let's explain this idea more deeply. Consider the subspace spanned by $x,y$ which is a real subspace of dimension $2$ and isometric to $\mathbb{R}^2$ (in which euclidean geometry applies) and consider some inner product.

Then, select $u,v \in \mathbb{R}^2 : \langle u,v \rangle = 0$ and observe that:

So, identifying the norm of the vector with its geometric length, by Pythagoras, this is equivalent to asserting that $u \perp v$. The inner product introduces geometry in vector spaces by introducing the notion of disjoint direction; orthogonality is algebraic perpendicularity.

Let's explain it a little more, recapitulating; the inner product gives rise to the norm of a vector which is a quantitative factor of how much it dilates the direction in which the inner product situates it (along with all its proportional vectors). Then, it uses this substratum to understand the relation between the vectors' directions by measuring how much each vector's norm factor contributes to the norm of the combined vector.

If each norm is an expression of its vector's direction, direction interference would affect the norm of the combined vector as the inner product expresses:

Identifying the norm with the length in the euclidean plane, perpendicular directions satisfy the Pythagorean theorem as we indicated above. Considering overlapped directions, the same orientation would be expressed as an expansion, $\langle u,v \rangle > 0$, opposite orientation would appear as a retraction $\langle u,v \rangle < 0$ and perpendicular directions will make $\langle u,v \rangle = 0 \iff u \perp v$

It is important to note that what the norm is, is defined by the inner product itself; the geometry inserted (this disjoint notion) depends entirely on how the norm of a vector gets measured, meaning that, depending on the inner product selected, non-perpendicular vectors can still be mathematically orthogonal (extend through disjoint directions) from the product's perspective.

Thus, it is enough to say at this point that $\langle x,y \rangle$ measures how much direction both share or simply how aligned they are in terms of direction and norm.

6.3.5. Cauchy-Schwarz and angle between two vectors.

With a basic notion about what inner product is, let's check what Cauchy-Schwarz asserts.

There are two presentations of the same statement, both are mathematically equivalent but admit distinct immediate readings:

Let $x,y \in V$, and $\alpha$ a scalar of the field, then:

According to how we understand the inner product, the Cauchy-Schwarz inequality has a direct meaning; the maximum expression of shared direction is proportionality.

Observe that:

Directly derived from above we have the following expression:

We finally state that the angle between two vectors is;

6.4. Rudin's 1.35 theorem.

Observe that the $1.35$ theorem in Baby Rudin basically takes $\mathbb{C}^n$ as a vector space and an inner product as:

Then, our previous expression takes the form:

Let's see now a demonstration. The demonstration is a little tricky, so to speak. The motivation is an orthogonal projection from which denominators have been subtracted. Essentially, the Cauchy-Schwarz inequality asserts the Pythagorean theorem.

Consider $a,b \in \mathbb{C}^n$ and:

Now, let's ask ourselves a question: what is the exact part of $b$ in $a$? Or, in other words, what is that $t$ such that $\langle a -tb,b \rangle = 0$? Check that:

This quotient measures the common direction of $a,b$ in terms of the direction of $b$, or in other words, how much $b$'s magnitude occupies in the shared direction of $a$ with $b$.

Check then, calling $r = a - \frac{C}{B}b$:

Which means:

Now, just changing each part:

7. Exercises.

7.1. Prove that no order can be defined in the complex field that turns it into an ordered field.

Let's make a brief reminder about what an ordered field is. An ordered field is a field which is also an ordered set and an ordered se which is a non-empty set in which there is defined a binary relation satisfying reflexivity, antisymmetry and trychotomy.

Let's observe a simple fact, given any order $< \subseteq \mathbb{C}^2$, then since $\mathbb{C}$ is a field it satisfies the axiom of the product with the order; this is positive order closure, meaning that the product of any pair of positive numbers should be positive but observe that $i$ simply doesn't fit on it.

It doesnt care how we define it, always this invariance get's transgresed:

• $i>0 \implies i^2 >0$, but by definition $i^2 = -1 <0$

• $i< 0 \implies -i>0 \implies (-i) (-i) = i^2 >0$, but again $i^2 = -1 <0$

Thus, $\mathbb{C}$ can't be an ordered field because $i$ simply don't fit on it.

Let's take a closer look on this fact. The real field is a total ordered set, we discussed that total orders allows to dispose all elements in a sequence along a line. We talk that a representation of this order is the positiveness and negativeness of a number which determines on what side of the $0$ this number lies. Then, we stablished that the multiplication had a geometric nuance as an homothety; in the sense that is an operation with a fixed point from which it dilitates each other point in the set. We defined that an homothety of positive ratio respects the sign and negative ratio reverses it, as a consecuence, any squared number is always positive; if the number it self is positive it stills positive and if not, then his own negativeness make his square positive and this is precisely the rule that $i$ brokes as we saw above.

This is because in $\mathbb{C}$ the product is a homothety combined with a rotation which takes number out of the line, breaking the total order.

7.2. Lexicographic order.

Take $z=(a,b),w=(c,d) \in \mathbb{C}$, then we define the lexicographic order as:

Does $(\mathbb{C}, <)$ have the $LUB$ property?

Let's remember that $LUB$ property stays that for any non-empty upperbounded subset exists the least upperbound element in the universe set.

We can think in the set $A := \Set{z =(a,b) \in \mathbb{C} : a < 1}$, obviously is upperbounded since any $(1,b) \in \mathbb{C}$ is greater than any other element in $A$ so it is upperbounded, but there is not least upper bound, since for any upperbound $(p,q) : p \geq 1$, $(p,q-1)$ is also an upperbound smaller than the first one.

Let's observe that $\mathbb{C}$ is often presented without an order, not only because with the order it can't be an ordered field but also because there even total orders doesn't make it complete.

7.3. Dual complex roots.

Suppose $z = a +bi$, $w = u + iv$ such:

1. Prove that $z^2 = w$ if $v \geq 0$

2. Prove that $\overline{z}^2 = w$ if $v \leq 0$

3. Conclude that every complex number (with one exception!) has two complex square roots.

First, check that:

7.4. If $z$ is a complex number, prove that there exists an $r > 0$ and a complex number $w$ with $|w| = 1 : z= rw$. Are $w$ and $r$ always uniquely determined by $z$?

Take the polar form of $z$ and call:

Observe that; $r\geq 0$, $|w| = \sqrt{cos^2\theta + sin^2\theta} = \sqrt{1} = 1$ and $z = rw$. Observe also that to compute both elements $z$ is only needed, so both are uniquely determined by $z$.

7.5. Generalization of the triangle inequality.

If $z_1,…,z_n \in \mathbb{C}$, prove that:

Let's proof this result by induction.

• First, for $n=2$, if we take $z_1,z_2 \in \mathbb{C}$, then, the real inner product given by the absolute value of a complex number states that:

  \[|z_1 + z_2|^2 = |z_1|^2 + |z_2|^2 + 2\langle z_1,z_2 \rangle \leq |z_1|^2 + |z_2|^2 + 2|\langle z_1,z_2 \rangle|\]

  And by Cauchy-Schwarz is:

  \[|z_1|^2 + |z_2|^2 + 2|\langle z_1,z_2 \rangle|\leq |z_1|^2 + |z_2|^2 + 2 |z_1||z_2| = (|z_1| + |z_2|)^2\]

  Thus, since the absolute value is always non-negative :

  \[|z_1 + z_2|^2 \leq (|z_1| + |z_2|)^2 \iff |z_1 + z_2| \leq |z_1| + |z_2|\]

  And we prove the result for $n=2$.

  
  

• Let's consider it true for some $n$. Consider $z_1,…,z_n \in \mathbb{C}$, $w = \sum_{i=1}^n z_i$ and some $z \in \mathbb{C}$, then:

  \[|z_1 \cdots + z_n + z| = |w + z| \leq |w| + |z| = |z_1 \cdots + z_n | + |z| \leq |z_1| \cdots + |z_n| + |z|\]

  
  

7.6. If $x,y \in \mathbb{C}$, prove that:

• First, take $|x| = |x -y + y| \leq |x -y| + |y| \iff |x| - |y| \leq |x-y|$.

• Second $|y| = |y-x + x| \leq |y - x| + |x| \iff |y| - |x| = - (|x| - |y|) \leq |y -x| = |x - y|$

Ultimately:

1. Conceptual Approach. Algebra, Algebraic Structures.

1.1. Algebra as a discipline.

Let's start by introducing what Algebra is. Algebra is the mathematical discipline in charge of the study of what operating with the elements of a set means: how items of a set are combined through an operation and what properties emerge from that context. It doesn't study the elements (like arithmetic would do) but the properties of the operations between them.

1.2. Algebraic Structures.

An algebraic structure is a set $(S, \Omega)$ where:

• $S \neq \varnothing$
• $\Omega$ is a collection of operations over $S$ (and maybe other sets as we will see) along with a sequence of rules called axioms that these operations satisfy.

This is the minimal basic item of study in algebra; a finite number of operations over a set defined by axioms. Our task is to comprehend how these operations behave through the axioms, applying classical logic.

In the previous section we already provided an introduction to algebraic structures through the number sets and then introduced a solid idea of the group and then the field. Our objective was to present a formalization about linear equations over a field.

We can review and develop the ideas presented in the Linear Equation section to understand the natural progression and the algebraic structures towards the vector space.

1.2.1. One operation algebraic structures.

Let $G := (S, \Omega) : \Omega := \Set{\star}$, where $\star : S \times S \to S$ is an internal operation (a function).

Then we define:

• $G$ (or $S$ by default) is said to be a Magma if $\star$ only verifies the closure property:

  • $a \star b \in S \ \ \forall a,b \in S$ (Closure)

  Observe that this structure is the minimal one, and his only property is forced by $\star$ definition.

  
  

• $G$ is a semigroup if it also verifies associativity:

  • $a \star b \in S \ \ \forall a,b \in S$ (Closure)
  • $(a \star b) \star c = a \star (b \star c) \ \ \forall a,b \in S$ (Associativity)

  
  

• A monoid is a semigroup that also has the identity element $e \in S$ relative to $\star$ verifying:

  • $a \star b \in S \ \ \forall a,b \in S$ (Closure)
  • $(a \star b) \star c = a \star (b \star c) \ \ \forall a,b \in S$ (Associativity)
  • $ \exists e \in S \ \ \forall a \in S : e \star a = a \star e = a$ (Identity)

  
  

• Now we enter the comfort zone; a Group is a monoid which verifies the existence of an inverse for each item in $S$:

  • $a \star b \in S \ \ \forall a,b \in S$ (Closure)
  • $(a \star b) \star c = a \star (b \star c) \ \ \forall a,b,c \in S$ (Associativity)
  • $ \exists e \in S \ \ \forall a \in S : e \star a = a \star e = a$ (Identity)
  • $\forall a \in S \ \ \exists a' \in S : a \star a' = a' \star a = e$ (Inverse)

  
  

Let's observe that each axiom lets us do something new in the structure; each new rule is a new capability based on the defining of invariants along the structure;

• The closure guarantees internal compositions; the belonging of any composition to $S$ remains constant.

• The associativity guarantees order-independence as long as the direction in the composition remains the same, meaning that $(a\star b) \star c$ and $a\star (b \star c)$ are the same object in the sense that we can talk about $a\star b \star c$ to refer to any of them, no matter what you compose first.

• The existence of the identity guarantees the existence of a way to operate that does nothing, concreted in an element of $S$ called the identity. The composition of $e$ with any item of $S$ in any direction remains constant and is the very same item.

• The existence of the inverse makes use of the identity to guarantee a way to operate that lets you reverse the operation. The composition of any item with its inverse in any direction remains constant and is the identity.

Observe that these are the main invariants in operations because these are the smallest subset of rules that allow you to define constraints about some elements (equations) and comfortably operate to isolate them in order to fully understand what that constraint is equivalent to and what a valid item is.

Suppose there is an interest in finding the relation between two elements $a,b \in S$, this is, $b$ is the composition of $a$ with some other element $x \in S$ which we are going to call the unknown: $a \star x = b$. Now, let's observe:

Step by step, the closure axiom asserts that $a^{-1} \star (a \star x) = a^{-1} \star b \in S$ and the proposal makes sense; the associativity establishes that $a^{-1} \star (a \star x) = a^{-1} \star a \star x= (a^{-1} \star a) \star x$, or that we can choose the order of the composition as we please. Then, the identity and the inverse cooperate together to simplify the expression; the inverse makes $a$ disappear, it substitutes it for the identity and the identity composed with any other item is that other item, thus:

These four invariants are summarized in the idea of group.

This idea of group can be extended by introducing the commutativity which defines the composition under direction as an invariant:

This allows reorganizing, making the simplification of expressions even more comfortable.

1.2.2. Two operations algebraic structures.

Until now, we've defined an algebraic structure that contemplates only one operation; let's now introduce algebraic structures with two operations.

Rings

First, the rings; a ring is a triple $(K,+,\ ·)$ such:

• $(K,+)$ is a group and introduces the four properties seen above.
• $(K, \ ·)$ is a monoid, it drops the existence of the inverse for any element of $R$.

• $+$ and $·$ relates themselves by the distributive law:

  \[a(b+c) = ab + ac \wedge (b+c)a = ba + ca\]

  We say both operations are compatible.

  
  

The ring introduces the second operation in a soft way; hanging on the existence of the inverse allows it to fit in many well-studied structures such:

• Integers ring; $(\mathbb{Z},+, \ ·)$, where for example $\forall a ( a \in \mathbb{Z} \wedge a \neq \pm 1 \implies a^{-1} \notin \mathbb{Z})$.

  
  

• Polynomial ring; with coefficients in a field $F$; $F[x]$ is a ring but not a field. For instance, $x$ has no multiplicative inverse:

  $\forall q \in F[x] \setminus {0} \ \deg(x \cdot q) = 1 + \deg(q) \geq 1 \neq 0 = \deg(1)$

  So the equation $x \cdot q = 1$ has no solution in $F[x]$

  
  

• Matrix ring; we've already seen that $(M_n(F)\setminus 0, \ ·)$ has no inverse for every element of $M_n(F)$; remember that we already saw that not every matrix is invertible, only those row-equivalent to the identity $I_n$

  
  

The mathematical world is full of structures with two operations where division fails.

Field

Then the concept of the field can be thought as an extension of the ring where each element has an inverse for the product operation except the zero element.

However, we want to present another perspective of a field.

Above we built step by step from the smallest algebraic structure Magma to the Group and his four axioms and we conclude the importance of such structure was that it allows to present and solve equations:

Then, the field $K$ aims to extend the same idea for two operations departing from the group structure. This means that a field is a triple $(K,+, \ ·)$ such:

• $(K, +)$ is an abelian group.
• $(K \setminus \Set{0}, \ ·)$ is an abelian group.
• $+$ and $·$ are compatible.
• $1 \neq 0$

This basically means that the field intends to be the structure in which equations can be defined and solved in two operations simultaneously; the motivation was to build an algebraic framework for common arithmetic operations in $\mathbb{R}, \mathbb{Q},…$ and other common fields.

2. Vector spaces.

Before, we've seen a set of axioms that rule how a set of operations $\Omega$ behaves in a non-empty set $K$. We extend this concept to two sets $(K,V)$ where $K$ is a field and $V$ is a group and both are related by a field-action.

Let's dive into this.

2.1. Actions. Algebraic Actions.

2.1.1. Definition.

First, let's define what an action is. Let $A$ and $S$ be two sets, then we define an action as a function:

\(\varphi: A \times S \to S\) \(\ \ \ \ \ \ \ (a,s) \to s\)

It grabs two elements, one from $A$ and another from $S$ and it maps it to a third element from $S$.

As defined, this function doesn't have any interest at all; it becomes interesting when $\varphi$ respects the structure of $A$; this way it is said that an action transforms $S$ using $A$'s algebraic structure.

2.1.2. Group's Action.

Group's action

Consider again $(G,S,\varphi)$ such that $\varphi : G \times S \to S$, let's suppose now that $G$ is a group for some internal operation $\star : G \times G \to G$.

In this context, we impose two rules $A1$, $A2$ over $\varphi : A \times S \to S$. In total, the triple $(G,S,\varphi)$ satisfies:

• Closure: $\varphi(a,s) \in S \ \ \forall a \in G, s \in S$, forced by $\varphi$ definition.

  
  

• (A1) Identity: Be $e \in G$ the identity on $G$ then we require: $\varphi(e,s) = s \ \ \forall s \in S$

  Meaning that the $G$'s identity doesn't move anything through $\varphi$.

  
  

• (A2) Associativity: $\varphi(a,\varphi(b,s)) = \varphi(a \star b,s) \ \ \forall a,b \in G, s \in S$

  Observe that this resembles the order invariant we talked about in groups; as long as the direction of the composition remains still ($a \to b \to s$) it doesn't matter the order in which you perform these compositions.

  Composing $b$ with $s$ through $\varphi$ and then this with $a$ is the same that "multiply" $a \star b$ in $G$ and then compose it with $s$ through $\varphi$. Check that if we write $a \star b = ab \wedge \varphi(a,s) = a\ ·s$ then, the rule above states that:

  \[a \ ·(b \ ·s) = (ab)\ ·s \ \ \forall a,b \in G, s \in S\]

  And it appears more familiar.

  
  

• Inverse: Observe this is not labeled as an axiom because it can be deduced from the two above but is included from pedagogic reasons.

  The idea of the inverse on a group intends to formalice the idea of revert a composition or go back. Thus, we are looking to demonstrate that, calling $\varphi_a(s) = \varphi(a,s)$

  \[\forall \varphi_a \ \exists \varphi^{-1}_{a} \ \forall s \in S: \varphi_a(\varphi^{-1}_{a}(s)) = \varphi^{-1}_{a}(\varphi_a(s)) = s\]

  Let's take $x = a^{-1}$ and observe that applying $A2$, $A1$ and the existence of the inverse in $G$ we get:

  \[a^{-1} \ ·(a \ · s) = (a^{-1}a) \ · s = e \ · s = s\] \[a \ · (a^{-1} \ · s) = (aa^{-1}) \ · s = s\]

  Meaning that $\varphi^{-1}_{a}= \varphi_{a^{-1}} \ \ \forall a \in G$.

  
  

Thus, this four properties makes that the triple $(G,S,\varphi)$ behaves like a group in the sense that we can define and solve equations over the elements of $S$ which remember, initially wasn't a group:

Meaning that we can treat the elements of $S$ as elements over which we can define constraints, relations, etc.

Although, it is worth remembering that $S$ is not a group and neither is the triple $(G,S, \varphi)$; what $S$ has acquired through $\varphi$ is a family of reversible parametrized transformations by the group $G$. It is a way to use $G$ in order to study $S$.

2.2. Field's action. Vector Space.

2.2.1. Definition.

Now consider again a triple $(K,V,\ ·)$, where $K$ is a field and $V$ is an abelian group and $·: K \times V \to V$ is a field's action.

As we said before this action respects the algebraic structure of $K$, which remember; is a combined abelian group over two different compatible operations, by forcing the following axioms:

• M1: $1_K \ · v = v$. The identity over the product doesn't transform anything.
• M2: $\alpha \ · (\beta \ · v) = (\alpha \beta) \ · v \ \ \forall \alpha, \beta \in K, v \in V $. associativity resemblance, order-independence while direction stays still.
• D1: $\alpha \ · (u + v) = \alpha \ · u + \alpha \ · v \ \ \forall \alpha \in K, u, v \in V $. Compatibility between $·$ and $+$ in $V$.
• D2: $(\alpha + \beta) \ · v =\alpha \ · v + \beta \ · v \ \ \forall \alpha, \beta \in K, v \in V$. Compatibility between $·$ and $+$ in $K$.

This four axioms plus the four axioms of the group $V$ gives us the eight axioms of the vector space $(K,V, \ ·)$.

2.2.2. Conceptually Approach.

This time, a field's action over a group doesn't intend to give an algebraic structure to $V$; instead it provides, through $K$, a uniform deformation mechanism. The arithmetic system of $K$ as a field is used to perform richer algebraic manipulations on $V$ as a group.

$·: K \times V \to V$ gives to $V$ a parametric scaling mechanism through $(K,+, \ ·)$ which, ultimately, allows a coherent way to perform linear combinations on the elements of $V$.

Let's dive into "scaling", "linear" and "linear combination" terms in order to understand what this introduction means.

Scaling.

Scaling is about relating $V$'s elements using $K$'s elements as mediators. The term "scalar" comes from scale; changing the size without changing the essential nature of the object.

Then, being $u,v \in V$ and $\alpha \in K : \alpha \neq 0$, when we say that $u = \alpha \ · v$, we are really saying $u$ is obtained from $v$, keeping its structural information since, thanks to $K$ being a field and $·$ respecting $K$'s properties, we can revert this operation and go back from $u$ to $v$. Let's see that by $M2$ and $M1$ we can guarantee the existence of some $\beta \in K$ such that $\beta \ · u = v$. Take $\beta = \alpha ^{-1}$ and:

In this context we say that $u$ and $v$ are proportional. Specifically, there exists an entire family of elements to which $u$ is proportional: $\Set{\alpha v : \alpha \in K \wedge \alpha \neq 0} \subseteq V$.

Linear & Linear Combinations.

Linear refers to independent contributions of several parts which do not interfere between them.

In this case, a linear combination is the minimal operation that takes place in the triple $(K,V, \ ·)$; independent contribution of proportional components of $V$.

Is the most general way to combine while respecting the principle of independent contributions:

Observe that $D1$ and $D2$ guarantee that $K$ and $V$ respect this object on $(K,V,\ ·)$.

Vector Space

Thus, $(K,V, \ ·)$ is the algebraic environment where a set of objects can be combined by independent contributions weighted by a field. A field's action is the mechanism that imports the arithmetic of $K$ as a system of intensities applicable to the elements of $V$.

A vector space is nothing but, conceptually, a functional linear combination system and his elements are called vectors.

Vectors

Thus, vectors are elements of an algebraic structure called vector space whose elements relate through linear combinations verifying the eight axioms presented before.

Being the vector space an algebraic structure means that vectors (and vector spaces) are nothing; as we said at the start of these very notes, algebra cares about how to operate and the properties that emerge from some defined operable structure.

In other words; vectors are the elements of an abelian group $V$ on which a field $K$ acts through a "deformation" mechanism. This action, the combination or composition of these deformed elements of $V$, is what we call linear combinations: weighted sums where each element of $V$ contributes with an intensity set by $K$.

The corollary from this assertion is that what really matters is that there are objects (matrices, polynomials, arrows, etc.) that, with the right operations, verify the vector space axioms; thus, they are vectors and all the machinery defined around them works also for these objects.

2.3. Examples.

Let's see a few mathematical objects that algebraically behave as vector spaces.

2.3.1. The n-tuple space.

Consider $F$ a field, then the algebraic structure $(F^n,+)$ with the addition behaving as:

Observe that the properties of the abelian group $(F,+)$ expand to $(F^n,+)$ so this is also an abelian group.

Thus, defining the function defined as:

\(· : F \times F^n \to F^n\) \((\alpha, \begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix} ) \to \begin{pmatrix}\alpha x_1 \\ \vdots \\ \alpha x_n\end{pmatrix}\)

Verifies the field's action axioms:

• M1: $1 \ · \begin{pmatrix}x_1 \ \vdots \ x_n\end{pmatrix} = \begin{pmatrix}1x_1 \ \vdots \ 1x_n\end{pmatrix} = \begin{pmatrix}x_1 \ \vdots \ x_n\end{pmatrix} \ \ \forall x \in F^n$

  
  

• M2: $\alpha \ ·( \beta \ · \begin{pmatrix}x_1 \ \vdots \ x_n\end{pmatrix}) = \alpha \ · \begin{pmatrix}\beta x_1 \ \vdots \\beta x_n\end{pmatrix} = \begin{pmatrix}\alpha \beta x_1 \ \vdots \\alpha \beta x_n\end{pmatrix} = (\alpha \beta) \ · \begin{pmatrix}x_1 \ \vdots \ x_n\end{pmatrix} \ \ \forall \alpha, \beta \in F, x \in F^n$

  
  

• D1:

  \[\alpha \ · (\begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix} + \begin{pmatrix}y_1 \\ \vdots \\ y_n\end{pmatrix}) = \alpha \ · \begin{pmatrix}x_1 +y_1 \\ \vdots \\ x_n + y_n\end{pmatrix} = \begin{pmatrix}\alpha (x_1 + y_1) \\ \vdots \\ \alpha (x_n + y_n)\end{pmatrix} = \begin{pmatrix}\alpha x_1 + \alpha y_1\\ \vdots \\ \alpha x_n + \alpha y_n\end{pmatrix}\] \[= \begin{pmatrix}\alpha x_1 \\ \vdots \\ \alpha x_n\end{pmatrix} + \begin{pmatrix}\alpha y_1 \\ \vdots \\ \alpha y_n\end{pmatrix} = \alpha \ · \begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix} + \alpha \ · \begin{pmatrix}y_1 \\ \vdots \\ y_n\end{pmatrix} \ \ \forall x,y \in F^n, \alpha \in F\]

  
  

• D2:

  \[(\alpha + \beta)\ · \begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix} = \begin{pmatrix}(\alpha + \beta)x_1 \\ \vdots \\ (\alpha + \beta)x_n\end{pmatrix} = \begin{pmatrix}\alpha x_1 + \beta x_1 \\ \vdots \\ \alpha x_n + \beta x_n\end{pmatrix} = \begin{pmatrix}\alpha x_1 \\ \vdots \\ \alpha x_n\end{pmatrix} + \begin{pmatrix}\beta x_1 \\ \vdots \\ \beta x_n\end{pmatrix}\] \[= \alpha \ · \begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix} + \beta \ · \begin{pmatrix}x_1 \\ \vdots \\ x_n\end{pmatrix}\]

  
  

Thus, the triple $\big(F,(F^n,+), \ · \big)$ satisfies:

• $F$ being a field.
• $(F^n,+)$ an abelian group.
• $(\ ·)$ is a field's action (an action satisfying $M1,M2,D1,D2$ axioms).

And is a vector space.

Observe that, as a special case, $(\mathbb{R},(\mathbb{C}^n,+), \ ·)$ is a vector space.

2.3.2. The space of m x n matrices.

Let's consider the set $M_{m \times n}(F) = F^{m \times n}$ with

Observe that again addition in $F^{m \times n}$ is extended from $(F,+)$ which is an abelian group so $(F^{m \times n},+)$ is also an abelian group, the argument is the same, addition in $F^{m \times n}$ are $mn$ additions on $F$.

Also, consider the action:

\(· : F \times F^{m \times n} \to F^{m \times n}\) \((\alpha, (a_{ij}) ) \mapsto (\alpha a_{ij})\)

Observe that:

• M1: $1 \ · (a_{ij}) = (1 a_{ij}) = (a_{ij}) \ \ \forall (a_{ij}) \in F^{m \times n}$

  
  

• M2: $\alpha \ · (\beta \ · (a_{ij})) = \alpha \ · (\beta (a_{ij})) = (\alpha \beta (a_{ij})) = (\alpha \beta) \ · (a_{ij}) \ \ \forall \alpha, \beta \in F, (a_{ij}) \in F^{m \times n}$

  
  

• D1: \(\alpha \ · ((a_{ij}) + (b_{ij})) = \alpha \ · \big((a_{ij} + b_{ij})_{ij} \big) = \big(\alpha (a_{ij} + b_{ij})_{ij} \big) = \big(((\alpha a_{ij}) + (\alpha b_{ij}))_{ij} \big)\)

  \[= (\alpha a_{ij}) + (\alpha b_{ij}) = \alpha \ · (a_{ij}) + \alpha \ · (b_{ij}) \ \ \forall \alpha \in F, (a_{ij}),(b_{ij}) \in F^{m \times n}\]

  
  

• D2:

  \[(\alpha + \beta) \ · (a_{ij}) = (((\alpha + \beta)a_{ij})_{ij}) = ((\alpha a_{ij} + \beta a_{ij})_{ij}) = (\alpha a_{ij}) + (\beta a_{ij}) \ \ \forall \alpha, \beta \in F, (a_{ij})\in F^{m \times n}\]

  
  

This way, $F$ is a field, $(F^{m \times n},+)$ is an abelian group and $( \ ·)$ is a field's action of $F$ over $(F^{m \times n},+)$.

Thus, $\big(F,(F^{m\times n},+), \ · \big)$ is a vector space.

Let's observe that, despite we already touched some operation properties from the addition and product of matrix, the fact that this set is a vector space finish refining the details of both operations and how they related them selves.

2.3.3. The space of functions from a set to a field.

Consider now some field $K$ and some set $S \neq \varnothing$, then, remember that being $f : S \to K$ a function, this gets identified with its graph $f \subseteq S \times K : (\forall s \in S \ \exists ! \alpha \in K \mid (s,\alpha) \in f)$, meaning that a function is a relation in which every element of the domain has a unique image through $f$.

Then, the set of all the relations from $S$ to $K$ that are applications is:

In this context, we agree on the notation $f(s) = \alpha \iff (s,\alpha) \in f$, then let's call:

This is the set of all the functions from $S$ to $K$ and define an addition over $K^S$ elements as:

Observe that, due to the closure in $(K,+)$ we have that $f(s) + g(s) \in K$, so $f+g \in K^S$. Or simply, abreviate it as the formulation: $(f+g)(s) = f(s) + g(s)$.

The addition definition again gets defined directly in terms of the addition in $K$, so $(K^S,+)$ is an abelian group. Being the neutral element and the inverse defined as:

Also, define the action:

And check that satisfies the field's action axiom:

• M1: $1 \ · f = \Set{(s,1 f(s)) \mid s \in S} = \Set{(s,f(s)) \mid s \in S} = f$

  
  

• M2:

  \[\alpha \ ·(\beta \ ·f ) =\alpha \ · \Set{(s,\beta f(s)) \mid s \in S} = \Set{(s,\alpha \beta f(s)) \mid s \in S}\] \[= (\alpha \beta) \ · \Set{(s, f(s)) \mid s \in S} = (\alpha \beta) \ ·f\]

  
  

• D1:

  \[\alpha \ · (f + g) = \alpha · \Set{(s, f(s) + g(s)) \mid s \in S} = \Set{(s,\alpha f(s) + \alpha g(s)) \mid s \in S}\] \[= \Set{(s,\alpha f(s)) \mid s \in S} + \Set{(s,\alpha g(s)) \mid s \in S} = (\alpha \ · f) + (\alpha \ · g)\]

  
  

• D2:

  \[(\alpha + \beta) \ · f =(\alpha + \beta) \ · \Set{(s, f(s)) \mid s \in S} = \Set{(s,(\alpha+\beta) f(s)) \mid s \in S}\] \[=\Set{(s,\alpha f(s)+\beta f(s)) \mid s \in S}= \Set{(s,\alpha f(s)) \mid s \in S} + \Set{(s,\beta f(s)) \mid s \in S}\] \[= (\alpha \ · f)+ (\beta \ · f)\]

  
  

Thus, $( \ ·)$ is a field's action and $(K,(K^S,+),\ · )$ is a vector space.

2.3.4. The space of polynomial functions over a field K.

2.3.4.1. Introduction to polynomials.

A polynomial is, elementally, the simplest algebraic expression that can be formed by combining an indeterminate with itself along with scalars of a field $K$ using addition and multiplication operations.

The concept of polynomial function emerges when you try to evaluate the expression on entities from an evaluation domain.

Note that we deliberately used indeterminate instead of unknown, since this last one resembles the scalars of a field, but in a polynomial function, usually, the domain can be more than scalars, it can be tuples, matrices, or even other polynomials.

In summary, a polynomial is a generic presentation of a finite sequence of operations over a single object with itself along with two predefined operations.

Vector space of the polynomial functions over a field K

Then let's consider a field $K$ and the set:

Let's consider the same operations $+$ and $·$ than in the example before. Let's observe that:

• $\boldsymbol{0} \in Pol(K,K)$, since the function $f (s) = 0 = \sum_{i=0}^n 0 s^i\ \ \forall s \in S$ have a polynomial form.

• $f(s) + g(s) = (f+g)(s) = \sum_{i=0}^n (\alpha_i + \beta_i)s^i \in Pol(K,K) \ \ \forall f,g \in Pol(K,K)$, so the addition is closed on $Pol(K,K)$.

• $ \forall f (-f \in Pol(K,K))$, just consider: $f(s) = \sum_{i=0}^{n} \alpha_i\, s^i$ other $g(s) = -(f(s)) = - (\sum_{i=0}^{n} \alpha_i\, s^i)$, then, is obvious that $(f + g)(s) = (g + f)(s) = \boldsymbol0$

  
  

Thus, we've just demonstrated that $(Pol(K,K),+)\leq (K^K,+)$; the abelian condition comes from commutativity on $(K,+)$, so $(Pol(K,K),+)$ is an abelian group.

Observe that $\alpha \ · f \in Pol(K,K) \ \ \forall f \in Pol(K,K)$, meaning that it does have sense to define the action $· : Pol(K,K) \to Pol(K,K)$ as a subfunction of $· \in \mathcal{P}(K^K \times K^K)$ defined above. Thus the properties of the field's action $·$ described above already applies to $Pol(K,K)$ since is a subset of $K^K$.

Thus, the triple $(K,Pol(K,K),\ ·)$ is a vector space.

2.4. Immediate Properties from vector spaces.

Let's check some important properties from the vector spaces that are immediately derived from the axioms. From now on, let's consider $(K,V, \ ·)$ as a vector space.

• Let be: $\boldsymbol{0} \in V, \alpha \in K$ then:

  \[\alpha \ · \boldsymbol{0} = \alpha \ · ( \boldsymbol{0} + \boldsymbol{0} ) = \alpha \ · \boldsymbol{0} + \alpha \ · \boldsymbol{0} \iff \alpha \ · \boldsymbol{0} = \boldsymbol{0} \ \ \forall \alpha \in K\]

  
  

• Also, be $0 \in K, v \in V$, then:

  \[0 \ · v = (0 + 0) \ · v = 0 \ · v + 0 \ · v \iff 0 \ · v = \boldsymbol{0} \ \ \forall v \in V\]

  
  

• Consider now $\alpha \in K : \alpha \neq 0$ and $v \in V$ such $\alpha \ · v = \boldsymbol{0}$, then observe $\alpha \neq 0 \implies \exists \alpha ^{-1} \in K$, thus

  \[\alpha \ · v = \boldsymbol{0} \iff \alpha^{-1}(\alpha \ · v) = (\alpha ^{-1} \alpha) \ · v = 1 \ · v = v = \alpha ^{-1} \ · \boldsymbol{0} = \boldsymbol{0}\]

  So, $(\alpha \ · v = \boldsymbol{0} \wedge \alpha \neq 0) \implies v = \boldsymbol{0}$

  Observe that essentially: $\alpha \ · v = \boldsymbol{0} \implies \alpha = 0 \vee v = \boldsymbol{0} \ \ \forall \alpha \in K, \forall v \in V$

  
  

• Observe that: $0 = 0 \alpha = (1 - 1) \alpha = \alpha + (-1) \alpha \implies -\alpha = (-1)\alpha$

  
  

• Lastly observe that, although we've defined linear combinations over two elements, the associative and distributive properties of the vector space allow us to think about a linear combination of $n$ vectors.

  Be $v, u_i \in V \ \ \forall i \in \mathbb{N}$ and $ \alpha_i \in K : \alpha_i \ \ \forall i \in \mathbb{N}$, satisfying:

  \[v = \sum_{i = 1}^n \alpha_i u_i\]

  Then, we say that $v$ is a linear combination of the $u_1,u_2, \ldots,u_n$ vectors.

  
  

3. Vector Spaces and Geometry.

Before concluding this introductory section on vector spaces, we shall consider the relation of vector spaces with geometry or the geometric intuition of vector spaces.

Although we haven't presented this result yet, it is true that for any $K$-vector space $V$ with a finite dimension there exists some $n \in \mathbb{N}$ for which $K^n$ and $V$ are isomorphic; $V \simeq K^n$, meaning that both are the same vector space and share the same specific properties; this will be completely acknowledged by the reader at the end of this post.

Thus, we can understand the geometric structure of any $K$-vector space by alluding to the geometric construction of $K^n$.

3.1. Analytic Geometry: The affine space.

3.1.1. Definition.

Let's say that $K = \mathbb{R}$, then $\mathbb{R}^n$ is what we call an affine space. In this section we are going to explain briefly what an affine space is and how it diverges from the vector space and why this idea is geometrically interesting.

Let $V$ be a $K$-vector space and $\mathcal{A} \neq \varnothing$, whose items we will call points. Then, we define as an affine space over $V$ a pair $(\mathcal{A},+)$ where $+ : \mathcal{A} \times V \to \mathcal{A}$ is a simply transitive group action.

This means that $+$ satisfies group's action axioms:

• $A1$ (Identity): $P + 0_V = P \ \ \forall P \in \mathcal{A}$
• $A2$ (Associativity): $(P + u) + v = P + (u + v) \ \ \forall P \in \mathcal{A}, \forall u,v \in V$

Observe that at this point, as we discussed above with the group actions, $+$ offers a family of reversible parametrized transformations by the vector space $V$ that allow us to study the points of $\mathcal{A}$ through the vectors of $V$.

But also this group action is free and transitive (the combination gives us the simply transitive property):

• Transitivity: $\forall P, Q \in \mathcal{A} \ \exists v \in V : P + v = Q$

  Every point on $\mathcal{A}$ is reachable by any other point through a $V$'s item. Thus, there are no distinguished, isolated or privileged points on $\mathcal{A}$.

  
  

• Free: $\forall P \forall v (P + v = P \implies v = 0)$

  This essentially means that no point gets translated over itself; each non-zero vector moves one point $P$ to another point $Q$ such that $P \neq Q$

  
  

Observe that the Transitivity and Free properties can be collapsed into:

Let's observe that, if we consider $u,v \in V : P + u = P + v$, then $(P + u) + -v = (P + v) + -v$, by $A2$ is $(P + u) + -v = P + (u - v) = (P + v) + -v = P + (v - v) = P + 0_V$ and then, by $A1$, $P + (u - v) =P$, then by the Free property $P + (u - v) =P \implies u - v = 0_V$, thus $u = v$, so both properties imply the one above.

Also observe that if we consider the statement above as true, obviously the transitivity property applies but also the uniqueness of $v$ implies the Free property since by $A1$ the $0_V$ already satisfies $P + 0_V = P$, thus $\forall v \in V \ (P + v = P \implies v = 0_V)$. So both statements co-imply each other and can be substituted.

Thus, essentially, an affine space is the object resulting from applying vectors to study points of a non-empty set through a simply transitive group action.

3.1.2. Isomorphism by origin.

Let's now check some interesting property of affine spaces. The transitive property asserts that, from the $V$ perspective, all points in $\mathcal{A}$ are equal; as we said, there are no privileged or distinguished points.

Now, let's take some $O \in \mathcal{A}$, then, we define the application:

Obviously it is a bijection since, fixing $O$, only one vector corresponds to one point $P$.

Let's also consider the following operations over the points of $A$ being $P,Q \in \mathcal{A}$ and $\lambda$ a scalar:

Basically, we operate with the vectors assigned by $\Phi_O$ and then return the resulting point of $A$ (note that since $\Phi_O$ is bijective, we can consider the inverse $\Phi_O^{-1}$).

Let's see that $(\mathcal{A},+_O, ·_O)$ is a vector space:

• $(\mathcal{A},+_O)$ is an abelian group:

  • Closure: For any $P, Q \in \mathcal{A}$, $\Phi_O(P), \Phi_O(Q)$ are defined as vectors of $V$ and since $V$ is a vector space, the linear combination $\Phi_O(P) + \Phi_O(Q)$ is guaranteed to be in $V$ and thus, since $\Phi_O$ is bijective, there exists a point $P' \in \mathcal{A} : \Phi_O(P') = \Phi_O(P) + \Phi_O(Q)$, so $P' = \Phi_O^{-1}(\Phi_O(P) + \Phi_O(Q)) = P +_O Q \in \mathcal{A}$

    
    

  • Associativity:

    Let's observe that:

    \[P +_OQ =\Phi_O^{-1}(\Phi_O(P)) +_O \Phi_O^{-1}(\Phi_O(Q)) = \Phi_O^{-1}\big(\Phi_O(P) + \Phi_O(Q)\big)\]

    Then, applying the same scheme, we have:

    \[(P +_OQ) +_OT = \Phi_O^{-1}\big(\Phi_O(P) + \Phi_O(Q)\big) +_O \Phi_O^{-1}(\Phi_O(T))\] \[= \Phi_O^{-1}\big([\Phi_O(P) + \Phi_O(Q)] + \Phi_O(T)\big)\]

    Thus, by the associativity in $(V,+)$ is:

    \[(P +_OQ) +T = P +(Q +_OT)\]

    
    

  • Identity: Let's check that $O$ satisfies $\Phi_O(O) = 0 \in V$ (by $A1$ and the simple transitivity property):

    \[P +_O O = O +_O P = P \ \ \forall P \in \mathcal{A}\]

    
    

  • Inverse: Check that since $(V,+)$ is an abelian group, for each $v \in V$ there exists one and only one $u \in V$ such that $v + u = u + v = 0$ and we say $u = -v$.

    Thus since $\Phi_O$ is bijective, for each $P \in \mathcal{A} : \Phi_O(P)=v $ there exists a unique $Q$ such that $\Phi_O(Q) = -v$. And is:

    \[P +_O Q =\Phi_O^{-1}(v + (-v)) = \Phi_O^{-1}(0) = O\]

    The same $Q +_O P$.

    
    

  • Commutativity: Immediate from the commutativity in $(V,+)$.

    
    

• $·_O:K \times A \to A$ is a field's action.

  • M1: $1 ·_O P = \Phi_O^{-1}(1 · \Phi_O(P)) = \Phi_O^{-1}(\Phi_O(P)) =P$
  • M2: $\alpha ·_O (\beta ·_O P) = \alpha ·_O\Phi_O^{-1}(\beta · \Phi_O(P)) = \Phi_O^{-1}(\alpha\beta · \Phi_O(P)) = (\alpha \beta) ·_O P$

  • D1: $\alpha ·_O (P +_O Q) = \alpha ·_O \Phi_O^{-1}(\Phi_O(P) + \Phi_O(Q)) = \Phi_O^{-1}(\alpha · \Phi_O(P) + \alpha · \Phi_O(Q)) =$

    And by the same property announce in the associativity point, is:

    $= \Phi_O^{-1}(\alpha · \Phi_O(P)) + \Phi_O^{-1}(\alpha · \Phi_O(Q)) = \alpha ·_O P +_O \alpha ·_O Q$

  • D2: $(\alpha + \beta) ·_O P = \Phi_O^{-1}((\alpha + \beta) · \Phi_O(P)) = \Phi_O^{-1}(\alpha·\Phi_O(P) + \beta·\Phi_O(P)) = \alpha ·_O P +_O \beta ·_O Q$

  
  

And by the bijection $\Phi_O$ it is isomorphic to $V$. This means that whenever we select an origin $O$ in which to define $\Phi$ we automatically instantiate (through $+_O,·_O$) a copy of $V$. An affine space acquires a vector space structure when you fix a point.

3.2. Vectors as geometric objects: Arrows.

Thus, let's conserve this conception about an affine space. Again, an affine space is what results from using vectors to study a non-empty set in a simple and transitive way. At this point we have introduced the idea of a vector as an algebraic item, but we now can introduce its natural geometric counterpart.

3.2.1. Bound arrows. Vectors as arrows.

Let's consider the affine space $(\mathcal{A},+)$ over the $K$-vector space $V$ and also, fixed an origin $O \in A$, define $\Phi_O, +_O,·_O$ as above, and subsequently we get the $K$-vector space: $(K,(\mathcal{A},+_O),·_O)$ isomorphic to $V$.

In this context, let's get two points $P,Q \in \mathcal{A}$; we subtly introduced before the idea of the arrow. When we said in $(A,+)$ there is some $v \in V: P + v = Q$ we were introducing the idea about $v$ being a "displacement" from $P$ to $Q$:

Let's identify $v$ with $P$ and $Q$ through the following application:

Observe that $v$ is unique for each pair and that there are a few properties that $\delta$ satisfies that are released from the axioms of the affine space:

• $\overrightarrow{PP} = 0_V$​, from $A1$ and simple transitivity.

  
  

• Chasles Relation: $\overrightarrow{PQ} + \overrightarrow{QR} = \overrightarrow{PR}$

  Let's quickly observe that: $P + \overrightarrow{PQ} = Q \wedge Q + \overrightarrow{QR} = R$

  Thus, using $A2$ we get: $R = Q + \overrightarrow{QR} = (P + \overrightarrow{PQ}) + \overrightarrow{QR} = P + (\overrightarrow{PQ} + \overrightarrow{QR})$, thus:

  \[P + (\overrightarrow{PQ} + \overrightarrow{QR}) = R \iff \overrightarrow{PQ} + \overrightarrow{QR} = \overrightarrow{PR}\]

  
  

• $\overrightarrow{QP} = -\overrightarrow{PQ}$​. This result is immediate from above: $0_V = \overrightarrow{PP} = \overrightarrow{PQ} + \overrightarrow{QP} \iff \overrightarrow{QP} = -\overrightarrow{PQ}$

  
  

In this context, we say that the pair $(P,Q)$ or simply $\overrightarrow{PQ}$ is a "bound arrow", with its base in $P$ and its end in $Q$. We formalize this idea applying the $\delta$ construction.

A bound arrow is a pair $(P,v) \in \mathcal{A} \times V$ often related as $(P,\overrightarrow{PQ})$ and interchangeable with $(P,Q)$ since two elements define the third.

Let's pull back briefly to the vector space $(K,(\mathcal{A},+_O),·_O)$.

Let's observe that having fixed $O$, then we can collapse $\delta: A \times A \to V$ to $\delta_O: \mathcal{A} \to V$, verifying:

Since the simple transitivity of $+$ imposes a unique displacement from one point to another, fixing one of those points allows us to identify each point on $A$ with a unique "position relative to $O$" vector in $V$.

This means that we can treat each point on $A$ as an arrow from $O$, the identity of the abelian group $(A,+_O)$. In other words, the collection of arrows from an origin has a structure of vector space. This allows us to think of vectors as arrows in a line, plane, space and so on.

3.2.2. Equipolence class: Free vectors.

Let's observe that bound vectors, conceptualized as a displacement, have magnitude, direction and an orientation in the direction. Thus, despite the point of application, two bound vectors are equal if both match these three features.

This leads to the construction of the notion of the free vector as an equivalence class of those bound vectors that share these three features.

3.2.2.1. Appendix: Equivalence Class.

An equivalence class is the set of all elements in a set that we consider "equal" under a certain criterion of equality. The idea is that when we declare certain objects to be interchangeable for our purposes (even if they are not literally the same object), we group together all those that share that quality. That group is the class.

Their importance is structural: equivalence classes are the mechanism that allows us to quotient a set, that is, to create a new set whose points are precisely those packages.

Formally, let $X$ be a non-empty set; an equivalence relation is a binary relation $\sim \ \subseteq X\times X$ satisfying:

• Reflexivity: $\forall x \in X \quad x \sim x$
• Symmetry: $\forall x,y \in X \quad x \sim y \implies y \sim x$
• Transitivity: $\forall x,y,z \quad x \sim y \wedge y \sim z \implies x \sim z$

In this conditions, this equivalence relation defines our criterion to build the equivalence classes $[x]$, formally:

We say that $x$ is the representative of the class. Observe that the following immediate properties are satisfied by the equivalence class:

• $x \in [x] \ \ \forall x \in X$
• $x \sim y \iff [x] = [y]$

• $\forall x,y \in X \ (x \sim y \oplus x \not \sim y \implies [x] = [y] \oplus [x] \cap [y] = \varnothing)$

  This means that two elements of $X$ are either equivalent or not equivalent at all; following the second property, this means that their equivalence classes are disjoint or are the same.

• $\displaystyle\bigcup_{x \in X} [x] = X$

  
  

Lastly, we define the quotient set as:

This is the set whose elements are the groups of elements that are qualitatively different according to the $\sim$ criterion.

3.2.2.2. Vector equipolence.

Now, let's say that $X = A \times A$, then we define in $A \times A$ the relation:

Observe that reflexivity, symmetry and transitivity are immediate from $=$ in $V$. Thus, two bound vectors are equal as long as their displacement is the same in magnitude, direction and orientation despite the application point.

This criterion allows us to define the quotient set:

Whose equivalence classes are $[(P,Q)]$. Let's now observe carefully that this structure has a vector space structure.

First, let's fix an origin in $\mathcal{A}$, $O$ and consider the vector space $(K,(\mathcal{A},+_O),·_O)$ and observe that for each equivalence class $[(P,Q)]$, there is some $T \in \mathcal{A} : (O,T) \in [(P,Q)] \iff [(O,T)] = [(P,Q)]$, meaning that we can define:

This function is bijective since $O$ is fixed and we can transport $(+_O, \cdot_O)$ to the quotient set $\mathfrak{F}(\mathcal{A})$ by operating with the canonical representative of each class.

So $(K, (\mathfrak{F}(\mathcal{A}),+_O),·_O)$ is a vector space and this ultimately means that, in an affine space, whenever we treat a vector $v$ we catch the canonical representative of its equivalence class relative to an origin $O$, and work with it as if both were the same vector.

As an example, observe that this matches some known operations with points and vectors. Consider $P,Q \in \mathcal{A}$, we can identify each point with its vector from the origin, $\overrightarrow{OP}, \overrightarrow{OQ}$. Then, by Chasles:

Meaning that subtracting $P$ from $Q$ (from any arbitrary point) gives us the displacement from $P$ to $Q$ formalized in $\overrightarrow{PQ}$, and the thing is that we would not work directly with $\overrightarrow{PQ}$ but with the canonical representative of its equivalence class, $\overrightarrow{OT} : \overrightarrow{OT} \sim \overrightarrow{PQ}$

4. Summary.

As a brief summary of the post, we have defined Algebra as a mathematical discipline that deals with the study of the properties of operations. In this context, we developed an algebraic definition of the vector space as an algebraic structure in which a deformation mechanism acts on an abelian group, standardizing the "linear combinations" as the natural operation of this space, understood as independent and field-weighted contributions of elements from the abelian group.

Later, we explored the geometric intuition behind the vector spaces. For that, first we defined the affine space as the object resulting from using a vector space in a simple and transitive way to study a non-empty set. We also saw that the affine space is a "point-uniform" structure and whenever you fix one point as a source of vectors, the affine space acquires a vector space.

Then, we built the concept of "bound vector" as a displacement between two points in the affine space and identified such bound vector with the mathematical object we call "arrow". This way, with an origin fixed, we defined a bijective application that identifies any point with a certain vector from the origin. Thus, the collection of all possible arrows that stem from a point in an affine space has the structure of a vector space and that is the precise visual representation we were searching for.

Ultimately we formalized the bound vector as a "free vector", this being the equivalence class of all the bound vectors that share magnitude, direction and orientation. The set of all free vectors has also a vector space structure which allows us to identify any bound vector on the affine space with its canonical class representative and operate with it as if both were the same vector.

5. Exercises.

5.1. If $F$ is a field, verify that $F^n$ (as defined in Example 1) is a vector space over the field $F$.

Done in $2.3.1.$

5.2. If $V$ is a vector space over the field $F$, verify for all vectors $a_i \in V : i =1,2,3,4$ that:

Observe that since $(V,+)$ is an abelian group the exercise is immediate, it is just generalizing the associativity and commutativity for more than two elements.

5.3. If $C$ is the field of complex numbers, which vectors in $C^3$ are linear combinations of $(1, 0, -1), (0, 1, 1), (1, 1, 1)$?

Those resulted in being the composition of any proportional vector of the given ones, formally verifying:

Doing the algebraic simplification, we get the family of vectors:

Let's note an interesting observation, if we think about the forms of the vectors in order to extract a restriction, this is, when, for a vector $(x,y,z)$ does not exists $\alpha, \beta, \gamma \in \mathbb{C}:$

Ultimately; $\alpha = y-z, \beta = -x+2y-z, \gamma = x−y+z$, meaning the solution always exists and is unique, so in fact our span set is $\mathbb{C}^3$.

5.4. Let $V$ be the set of all pairs $(x, y) \in \mathbb{R}^2$ of real numbers. Is $(R, V, ·)$ a vector space?. Defined:

The addition $+$ is the classic addition operation in $\mathbb{R}^2$ so it is clear that $(V,+) = (\mathbb{R}^2,+)$ is an abelian group. Now, let's take a closer look at the field's action.

Observe that there is something weird; $0 · (x,y) = (0,0) \iff y = 0$ which should not be.

In a proper vector space the ordinary proof of the statement asserts that:

And the second equality comes from the axiom $D2$, that states:

But this is not satisfied by the operation:

Which is only true when $2y = y \iff y = 0$, so the structure provided does not verify the axioms of the action's field and is not a vector space.

5.5. On $\mathbb{R}^n$, which of the axioms for a vector space are satisfied by $(\mathbb{R}^n, \oplus, · )$ defined:

Observe that $\alpha \oplus \beta = \alpha - \beta \neq \beta - \alpha = \beta \oplus \alpha$, so $(\mathbb{R}, \oplus)$ is not an abelian group. Also, the application is not a field action, since the identity does transform the vector; $1\alpha = - \alpha$. Thus, the structure as presented is not a vector space.

5.6. Let $V$ be the set of all complex-valued functions $f$ on the real line such that $\forall t \in \mathbb{R} \quad f(-t) = \overline{f(t)}$, then show that $(V, +,·)$ is a $\mathbb{R}$-vector space with:

Observe that $V := \Set{f \in \mathbb{C}^{\mathbb{R}}\mid f(-t) = \overline{f(t)}}$, thus, let's first check $(V,+)$ is an abelian group:

• Associativity and Commutativity are immediate from Associativity and Commutativity of addition in $(\mathbb{C},+)$.

  Let's check closure: observe that

  \[f,g \in V \implies (f+g)(-t) = f(-t) + g(-t) = \overline{f(t)}+\overline{g(t)} = \overline{f(t)+g(t)}= \overline{(f+g)(t)}\]

  Let's think about the identity element. We have to demonstrate the existence of some element $i \in V$ such that

  \[(i+f)=(f+i)=f \quad \forall f \in V\]

  Let's consider $i \in \mathbb{C}^{\mathbb{R}}: i(t) = 0 \quad \forall t \in \mathbb{R}$, note in first place that $0 = \overline{0} \implies i \in V$, and it satisfies:

  \[(i+f)(t)=i(t) + f(t)=f(t)+i(t) = f(t) \quad \forall f \in V\]

  Lastly, consider some $f \in V$, we have to think in some $-f \in V: (f+-f)=(-f + f)=i$, or, what is the same: $(f+-f)(t) = (-f +f)(t) = 0$.

  Let's consider $-f \in \mathbb{C}^{\mathbb{R}}: -f(t) = -(f(t))$, observe that:

  \[-f(-t) = -(f(-t))=-(\overline{f(t)}) = \overline{-f(t)} \implies -f \in V\]

  Then, $(V,+)$ is an abelian group.

  
  

Let's check now that $·:\mathbb{R} \times V \to V$ is a field's action:

• Identity; Taking $1 \in \mathbb{R}$, is immediate.
• Associativity: Immediate from associativity in $(\mathbb{R}\setminus\Set{0}, ·)$.
• Compatibility in $V$; Immediate from compatibility in $\mathbb{C}$
• Compatibility in $\mathbb{R}$; Immediate from compatibility in $\mathbb{C}$

Thus, $(\mathbb{R},(V,+),·)$ is a vector space.

As an example of a function in $V$ not real-valued is $f(t) = it$.

5.7. Take the following operations in $\mathbb{R}^2$. Is $(\mathbb{R},(\mathbb{R}^2,+),·)$ a vector space?

Observe that there is no identity in $(\mathbb{R}^2,+)$ so it is not an abelian group

1. Multiple Input Features.

2. Vectorization.

   • 2.1. Mathematical concept.
   • 2.2. Vectorization with python.
     • 2.2.1. Python and C.
     • 2.2.2. Numpy.
     • 2.2.3. Numpy basic usage.
       • 2.2.3.1. Importing Numpy.
       • 2.2.3.2. Vectors and Numpy arrays.
       • 2.2.3.3. Operation on vectors.
       • 2.2.3.4. Matrix and operation with matrices.

3. Multiple Linear Regression.

• 3.1. Defining the problem.
• 3.2. Writing our code.

1. Learning Models.

1.1. Introduction. Problem setup.

In the first section we talked about a model to find an approximation of an unknown relation between two variable sets $\mathcal{X}$ and $\mathcal{Y}$.

This model consists of a template about this procedure based on an iterative process to find the local minimum of the function that measures the error of the hypothesis over the training set. Minimizing the error results in a "fair", reliable approximation of the sought relation.

This template's components are:

1. An input space $\mathcal{X}$ and output space $\mathcal{Y}$. This is where the relation lives and where the predictions want to be made.

2. The training set $\mathcal{D}$, this is what the model consumes to optimize the approximation.

3. A hypothesis set $\mathcal{H} = {h_\theta \mid \theta \in \Theta \subseteq \mathbb{R}^p}$. The family of functions which determines the model's expressiveness; what patterns it can and cannot represent.

4. A loss function $L : \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$. How do we measure error.

5. A cost function $J(\theta)$. How the loss function gets aggregated across the training data. This determines what is optimized and what gets parametrized by $\theta$ and decides the behavior of the model.

6. An optimization algorithm.

   
   

Every distinct decision over these parameters results in what we call a learning model. The learning models are in fact instantiations of the template we described in the post before. Most of the changes affect components $1$, $3$ and $4$; the cost function and the gradient descent remain the same over the models we will see in this blog.

1.2. Learning Model. Learning Paradigms.

Essentially, what we presented above is the problem setup, the parts of the problem we want to solve.

A problem setup is a tuple:

With fixed data, this is, given $(\mathcal{X}, \mathcal{Y}, \mathcal{D})$, then we decide the learning model, this is, how the model predicts and how we calibrate how well it predicts (the hypothesis set and the loss function).

Thus, a learning model is a pair:

This way, linear regression model means; learning model of affine transformation hypothesis and squared loss function: $M_{lr}:=(H_{at},L_s)$

2. Supervised Learning.

1.2.1.1. Definition.

Supervised Learning is one of the three best-known learning models, which trains a program to infer a relation between two data sets (inputs and outputs).

This training model relies on the fact that the program has a collection of pairs of inputs and outputs related by an unknown rule: $(x,y) \in \mathcal{X} \times \mathcal{Y} : y = f(x)$, and $f$ remains unknown (this function is typically an unknown joint distribution $P(X,Y)$, a stochastic relation; the function is a concrete case where all noise vanishes).

In this sense, supervised learning refers to the fact that during training (algorithm execution), every input $x$ comes with a label $y$; a correct answer that "supervises" the learning process.

In this sense, the task of supervised learning is, given the training set: ${ (x^{(i)}, y^{(i)}) }_{i=1}^{n} \subset \mathcal{X} \times \mathcal{Y}$ find some hypothesis $h : \mathcal{X} \to \mathcal{Y}$ of some hypothesis class $\mathcal{H}$ that minimizes the expected risk. However, taking again that $f$ is a simplification of $P$, in general, what is measured is the empirical risk instead which we call:

which basically measures the risk over the gathered sample instead of the entire population (which often is inaccessible in real problems).

Let's observe that this empirical risk is a template with three factors:

• The hypothesis class $h$ that makes the prediction against the labeled data.
• The loss function $\ell$ which measures the error between each pair $(h(x)^{(i)},y^{(i)})$
• A convenient scaling $\frac{1}{n}$; for convenience we can scale the empirical risk to simplify the result.

Later we will select each one of the three factors to instantiate the empirical risk into the cost function $J$.

1.2.1.2. Linear Regression: The simplest Supervised Learning Model instantiation.

Linear Regression is the simplest instantiation of the Supervised Learning Model.

Let's take a moment to introduce the affine concept. Linear and Affine are ways to apply transformations over the data on a set of points which satisfy specific conditions.

• Linearity applies a scalar transformation over the data, in linear algebra we did learn that a linear transformation $f$ satisfies:

  \[f(\alpha u + \beta v) = \alpha f(u) + \beta f(v) \implies 0 \to 0\]

  
  

• Affinity can be described as a linear transformation incorporating a translation, being $f: V \to W$ a linear transformation between two vector spaces, then we can consider an affine transformation $t: V \to W$ as:

  \[t = f + b : b \in W\]

  Where $b$ takes the role of the translation.

  
  

Now, suppose it is: $\mathcal{X} = \mathbb{R}^d \wedge \mathcal{Y} = \mathbb{R}$; the problem, having fixed these sets, is to predict a real-valued output from a vector of features. The core assumption of linear regression is that the relationship between input and output is affine, meaning that the hypothesis class is:

The reason for which we force the hypothesis to be an affine transformation is that for many types of relation between a set of parameters and the data, a linear application simply doesn't fit; for example a professional with $0$ years of experience doesn't receive a salary of $0$; the relation between the experience in a professional industry and the salary exists and is documented but is not linear.

Thus, suppose we have $n$ observations, each observation is a pair $(x^{(i)},y^{(i)}) \in \mathcal{X} \times \mathcal{Y}$ where the output is roughly determined by the input, with some noise on top: $y^{(i)} = f(x^{(i)}) + \varepsilon^{(i)}$ (this is the affine relation we talked about above), then the linear regression is the decision to search only among affine maps as candidates for $f$:

1.2.1.3. Classification Models.

Before, we've introduced Regression models with Linear Regression but we will cover that topic more extensively later.

Now we are going to cover the Classification Model; we've already internalized the regression setup: you have a hypothesis $h : \mathcal{X} \to \mathcal{Y}$ that predicts a continuous response; $h$ is continuous at any point which only makes sense if $\mathcal{Y}$ is isomorphic to some subset of $\mathbb{R}^n : n \in\mathbb{N}$. Classification arises when this hypothesis is no longer continuous, in the sense that $\mathcal{Y}$ is a discrete set.

In regression we talked about an error approximation through the loss function $\ell$ which tries to predict how far the prediction is from the desired output in a geometric "metric" sense; you care about the magnitude of the error, how far you are. In classification, being $\mathcal{Y}:= \Set{1,…,k}$ there is no meaningful notion of "distance"; when you predict a wrong class in $Y$, there is no notion of nearness to the correct answer in the sense that class $1$ isn't any closer or much better than class $7$ when the desired output is class $3$; there are only right or wrong answers.

This collapses $\ell$ to the 0-1 loss, which counts misclassifications:

In regression you minimize empirical risk with a continuous loss (squared error, absolute error), and the hypothesis class consists of functions $h: \mathcal{X} \to \mathbb{R}$. In classification you minimize empirical risk with a discrete loss (0-1 loss), and the hypothesis class consists of functions $h: \mathcal{X} \to \Set{1,…, k}$

Then, basically, classification models force the machine to recognise previously labeled classes (by minimizing the error function) using the hypothesis $h$ over the input $x$ data and identifying it as some value of the discrete codomain $y$.

1.2.2. Unsupervised Learning. Clustering.

The idea on a high level is that, in contrast to supervised learning, the labeling of the output $y$ disappears; there are no wrong answers when applying the hypothesis on the data. The task for the algorithm is to find structures in the data; clusters.

In Classification there is a training set $\Set{(x_i​,y_i​)}_{i=1}^n$​ where the labels $y_i$ were given by an oracle, in the sense that it is the desired output. Then, the task is to find a map that approximates $x \to y$ as much as possible. Observe that the categories exist before the algorithm starts the distinction.

Clustering inverts this relationship. The training set consists only of $\Set{x_i​}_{i=1}^n$ (without labels). The task is to discover that there are groups at all, and to assign each point to one. The categories are not predefined; they emerge from the geometric or distributional structure of the data itself. The fundamental assumption behind clustering is that the data is not uniformly spread across $\mathcal{X}$, but concentrates around certain regions. Clustering algorithms formalize different notions of what "concentrate" means:

• Centroid-based; structure means proximity to a representative point.

• Density-based; structure means regions of high density separated by regions of low density.

• Distributional; structure means the data was plausibly generated by a mixture and inference on the latent component assignments.

Each of these is a different inductive bias about what a "cluster" is.

The comparison is more subtle than "one has labels, the other doesn't."

The problem is ill-defined in a way classification is not.

• In classification, given a loss function and a hypothesis class, the optimal classifier is a well-defined object (the Bayes classifier, the ERM solution, etc.).

• In clustering, there is no unique "correct" partition, the answer depends on your notion of similarity, the number of clusters $k$ (often a hyperparameter you must choose), and the scale at which you look. Two entirely different clusterings of the same data can both be "valid."

2. Regression Model.

2.1. Linear Regression Overview.

Now, we are going to dive into the Linear Regression Model; this is the most widely used learning algorithm and many of the concepts used to explain it apply to other training algorithms.

Let's make clear that Linear Regression is an instantiation of what we call Supervised Learning, which is a training model that consists of first training the machine with previously labeled data $(x,y)$, where $x$ is called the input feature data and $y$ is the output target variable (the expected output). Also it is important to know that we have a finite number of training pairs $n$.

When you finish the training, you let the machine predict outputs for new unlabeled data.

Each training pair is reflected in a Cartesian coordinate system forming a cloud of points. As we said before, linear regression consists of the assumption that the relation between each pair is an affine transformation, or simply a line:

The notation $(x^{(i)},y^{(i)}) \in \mathcal{X} \times \mathcal{Y}$ refers to the fact that we are treating the i-th pair in the sample. Also, each pair is reflected in a table opposing $x$ and $y$ in an $n$-row table:

2.2. Training the machine. Linear regression.

Remember that the training algorithm's job is to form a function $f$ with the provided training set such that the transformation of an input feature $x$ by $f$, the prediction; $f(x)=\hat{y}$, is approximated enough to the target $y$; the expected output; $\hat{y} \to y$.

As we said before, linear regression assumes that $f$ is represented as a straight line in the graph of the form:

It is worth noting that these parameters tell us how $\hat{y}$ is built from $x$:

• $w$ represents how much $x$ contributes to $\hat{y}$; this will be clearer when the linear regression model for multiple variables gets introduced; $x$ and $w$ are vectors related in $f$ by the dot product. This way each $w_i \in w$ tells us how much the $x_i \in x$ feature contributes to the value of $\hat{y}$.

• $b$ as well represents how much $y$ is independent of $x$ since it is a constant value not related to the input. Observe that when $x = 0$, then $\hat{y} = b$ exactly, showing that this is the value that separates $\hat{y}$ from any $wx$.

2.3. Cost Function.

2.3.1. Squared Error.

Until now, we established that linear regression is about a finite training set that is represented in a coordinate system. We want a line such that $y = wx + b$ represents well enough the relation between the points.

Let's now dive into what well enough means. Well enough means that it minimizes the error in the prediction; let's now see what we call error and how to measure it.

The most natural thing to measure is the residual: how far off is the prediction from the truth:

So in intuitive terms, the thing we want to measure is $\displaystyle \sum_{i=1}^{n}e^{(i)}$, but positive and negative terms can cancel out and erase part of the information.

The two natural candidates to solve the problem, $\vert e^{(i)} \vert$ and the square $(e^{(i)})^2$; the reason why we decide to square is because it is differentiable at any point (we will understand this later); $\displaystyle \sum_{i=1}^{n}(e^{(i)})^2$

Now we take the average (so the cost doesn't scale arbitrarily with dataset size):

This is the Mean Squared Error. It gives you a single number that summarizes how badly the line misses the data on average and is essentially what we are measuring.

In practice you'll almost always see a factor of $\frac{1}{2}$ thrown in front:

This doesn't change which $(w,b)$ minimizes $J$ (it's just a positive constant scaling). It's there purely for convenience: when you take the derivative, the exponent $2$ comes down and cancels with the $\frac{1}{2}$​, giving you cleaner gradient expressions.

2.3.2. Interpretation. Optimization Problem.

Let's observe that $J(w,b)=0$ means that every single squared residual is zero, which forces each prediction to coincide with the labeled value; $f_{w,b}(x^{(i)}) = y^{(i)} \ \ \forall (x^{(i)},y^{(i)}) \in P(x,y)$

As $J$ increases, the line deviates more. This way, $J$ defines an ordering on parameter pairs $(w,b)$ lower $J$ means a better-fitting line. Finding the best line is now a clean optimization problem; minimize $J$ over $(w,b) \in \mathbb{R}^2$.

One thing worth noting: $J$ being zero is typically impossible (or undesirable) with real data, since the data has noise. But the key property holds $J$ is non-negative, equals zero only at perfect fit, and increases monotonically with the aggregate deviation of predictions from targets.

3. Gradient Descent.

We just saw that, in linear regression model we assume that exists an affine transformation $f_{w,b}(x) := wx + b$ that makes a solid prediction over the labeled points $(x,y) \in \mathcal{X} \times \mathcal{Y}$. We also said that a way to measure how "solid" this line is, is to build our Cost Function, represented as: $J(w,b) = \frac{1}{2m} \sum_{i=1}^{m} \bigl(f_{w,b}(x^{(i)}) - y^{(i)}\bigr)^2$ and we establish it as an optimization problem over the $(w,b) \in \mathbb{R}^2$ that minimizes the $J$ value as much as possible.

Now, Gradient Descent taught us a way to consistently search for $(w,b)$ pairs.

3.1. Introducing the problem.

We have the Mean Squared Error; $J(\theta)$, which measures the average distance-error between the predicted values $\hat{y}$ and the labeled values $y$, and we want to find that parameter $\theta^*$ that minimizes that error.

For a simple system, for example a continuous one-variable function on a real segment, we would calculate $\nabla J(\theta) = 0$:

But in the moment the parameter space grows, that closed-form solution either doesn't exist or becomes computationally prohibitive. An iterative method is needed.

3.2. Gradient Descent Algorithm.

Gradient Descent exploits one fundamental fact from calculus: the gradient $\nabla J(\theta)$ points in the direction of steepest ascent of $J$ at $\theta$, meaning that to "decrease" $J$ implies moving $\theta$ towards the opposite direction of $\nabla J$:

Where $\alpha$ is the learning rate and controls the step size.

This way, computing $\nabla_\theta J$ gives you a concrete vector that tells you how each parameter contributes to the error. In the sense that each component $\frac{\partial J}{\partial \theta_j}$​ tells you how $J$ (the error) changes, let's keep in mind the image above, $\theta$ always advances from left to right, this means that:

• If $\frac{\partial J}{\partial \theta_j} > 0$, then the function is ascending and is moving away from the minimum, $\theta$ has overshot the minimum to the right so subtracting a positive value $\alpha · \frac{\partial J}{\partial \theta}$ from $\theta$ nudges $\theta$ to the left, hence $J$ towards the minimum.

• If $\frac{\partial J}{\partial \theta_j} < 0$, then the function is descending and is approaching the minimum, $\theta$ has the minimum in front, to the right, so subtracting a negative value $\alpha · \frac{\partial J}{\partial \theta}$ is like adding a positive one, nudging $\theta$ forward and hence $J$ towards the minimum.

So, the sign always corrects the direction in both cases.

It is worth noting that convexity matters. For linear regression, $J$ is convex, there's a single global minimum and gradient descent will find it (given a reasonable $\alpha$). For neural networks, $J$ is non-convex with many local minima and saddle points.

It is also worth noting that, as $J$ gets closer to the minimum, the derivative term gets smaller in absolute terms, meaning that the term subtracted from $\theta$ is smaller, so $J$ advances more slowly to the minimum as it gets closer through Gradient Descent.

Observe that applying this to our linear regression model:

Where:

1. Multiple Input Features.

In our previous notes, our input was a single feature $x \in \mathbb{R}$ and the linear regression imposes the hypothesis to be an affine transformation: $f_{w,b}(x):=wx + b$, with two parameters.

Now suppose each observation has $d$ input features instead of one. The $i$-th training example is no longer a scalar $x^{(i)}$, but a vector:

Then, $w$ is also another vector, this can be seen as the complements of the features, so the affine transformation gets the form:

Let's observe that it is easier now to observe that each $w_i$ tells us how much $x_i$ contributes to $\hat{y} = f_{w,b}$ and how $b$, remaining a scalar, keeps maintaining the distance between $w · x$ and $\hat{y}$.

Let's now extend the vectorial product to simplify the expression by forming:

And defining $f_{\theta}(\tilde{x}) = \theta^\top \tilde{x} = w^\top x + b$

Then, the cost function is:

And the gradient is:

Where:

So conceptually, multivariate linear regression is identical to univariate — you just have more components in the gradient. Every intuition from the previous post (sign correction, convexity, learning rate behavior) carries over unchanged.

2. Vectorization.

2.1. Mathematical concept.

Vectorization is a technique to compute the gradient of a multivariable regression model efficiently. At the cost function there are $n$ training examples, each a vector in $\mathbb{R}^{d+1}$. The direct implementation is a double loop over $n$ examples, and for each, loop over $d+1$ components to compute $\theta^\top \tilde{x}^{(i)}$. This is computationally terrible.

The key observation is that the entire training set can be packed into a single matrix.

And thus, the vector of all predictions can be defined as the following matrix product:

This way, the $i$-th entry $X \theta$ is exactly: $(x^{(i)})^\top \theta$, which is the prediction for the $i$-th example.

The cost function becomes:

The entire gradient computation is, this way, expressed as two linear algebra operations. This is particularly important in computation, since when you write a Python loop to compute $\sum_{i=1}^m (\theta^\top x^{(i)} - y^{(i)}) x_j^{(i)}$ for each $j$, every iteration has interpreter overhead, cache misses, and no parallelism.

When you write X.T @ (X @ theta - y) in NumPy, the actual work is dispatched to BLAS (Basic Linear Algebra Subprograms), highly optimized Fortran/C routines that exploit CPU vector instructions (SIMD: single instruction, multiple data), cache-aware memory access patterns, and sometimes multiple cores. The algorithmic complexity is the same $O(md)$, but the constant factor can differ by orders of magnitude.

2.2. Vectorization with python.

2.2.1. Python and C.

Python makes no assumptions at parse time about what type a variable holds. The price of that flexibility is paid per operation, every time. It is an interpreted, dynamically-typed language meaning that at runtime, every operation on every value requires the interpreter to inspect the object's type, resolve the appropriate method, perform type coercion if needed, and manage reference-counted memory, all before the actual arithmetic happens.

Also, Python represents every value as a heap-allocated object with a type pointer, a reference count, and the actual payload. When you store n floats in a Python list, you have a contiguous array of pointers, each pointing to a separate heap-allocated float object scattered across memory. The data is fragmented by design.

These two problems together mean that iterating over n numerical values in Python incurs $O(n)$ interpreter overhead, $O(n)$ pointer dereferences with poor cache locality, and $O(n)$ dynamic type dispatches, none of which contribute to the actual mathematical computation.

C (and Fortran) operate under a fundamentally different model. Types are resolved at compile time, values are stored inline in contiguous memory without metadata wrappers (often in the stack), and the compiled machine code operates directly on raw bytes with no interpreter mediating each instruction.

A loop over n doubles in C touches a flat buffer sequentially, which is exactly the access pattern modern CPUs and their cache hierarchies are optimized for. Furthermore, decades of work in numerical linear algebra (BLAS, LAPACK) have produced C/Fortran libraries that exploit hardware-level parallelism, SIMD instruction sets that process multiple floats in a single clock cycle. These libraries represent the practical floor of how fast a given computation can run on a given CPU.

2.2.2. Numpy.

NumPy stands for Numerical Python, is a C-backed Python library that provides an n-dimensional array object (ndarray) and a large collection of mathematical operations that act on these arrays. The key insight is that the ndarray is a contiguous block of homogeneous, typed memory (not a python list of python objects).

When you write np.array([1.0, 2.0, 3.0]), you get a flat buffer of 64-bit floats laid out sequentially in memory, with shape/stride metadata on top. This is essentially the same memory layout a C or Fortran program would use.

Python does have a math module, but it solves the function resolution problem not efficient computation. NumPy resolves this by moving the inner loop from Python to C. It provides a Python object (ndarray) that internally holds a flat, typed, contiguous buffer, the same memory layout a C program would use. When you call a NumPy operation, Python dispatches once to a compiled C routine, which then iterates over the raw buffer with no interpreter involvement per element. The cost model changes from $O(n)$ Python operations to $O(1)$.

2.2.3. Numpy basic usage.

2.2.3.1. Importing Numpy.

We can import Numpy to our repository with:

import numpy as np    # it is an unofficial standard to use np for numpy
import time

2.2.3.2. Vectors and Numpy arrays.

Vectors are ordered arrays of numbers. The number of elements in the array is often referred to as the dimension. The vector shown has a dimension of $n$.

The elements of a vector can be referenced with an index. In math settings, indexes typically run from $1$ to $n$. In computer science indexing will typically run from $0$ to $n-1$.

NumPy's basic data structure is an indexable, n-dimensional array containing elements of the same type (dtype). Let's observe that in maths and computation, the dimension is not the same.

The dimension or rank of a vector $v$ refers to the number of elements of the basis of the vector space $V : v \in V$ which essentially refers to the $n$ in $\mathbb{R}^n : V \simeq \mathbb{R}^n$ (informally speaking, the number of 'elements' of the vector).

In computer science, dimension refers to the number of indices in the array; the array has a shape, which refers to the number of elements per index, and a dimension, which is the number of indices. This way, an $n$-dimensional vector coincides with a $1$-D array of shape $n$.

numpy.array

Create an array.

In [28]: print(np.array([1,2,3,4]))
[1 2 3 4]

numpy.zeros

numpy.zeros, according to the documentation; return a new array of given shape and type, filled with zeros.

numpy.zeros(shape, dtype=None, order='C', *, device=None, like=None)

Parameters:

• shape: int or tuple of ints

  Shape of the new array, e.g., (2, 3) or 2.

• dtypedata-type, optional

  The desired data-type for the array, e.g., numpy.int8. Default is numpy.float64.

• order{‘C’, ‘F’}, optional, default: ‘C’

  Whether to store multi-dimensional data in row-major (C-style) or column-major (Fortran-style) order in memory.

• devicestr, optional

  The device on which to place the created array. Default: None. For Array-API interoperability only, so must be "cpu" if passed.

In order to show some examples, the ipython3 tool is being used:

In [16]: print(np.zeros(4)); print(np.zeros(4).shape);  print(np.zeros(4).dtype)
[0. 0. 0. 0.]
(4,)
float64

numpy.random.rand

Random values in a given shape. Create an array of the given shape and populate it with random samples from a uniform distribution over [0, 1).

In [25]: print(np.random.rand(4))
[0.62305938 0.22321443 0.71399    0.09377264]

In [28]: print(np.random.rand(4,2))
[[0.77512097 0.76052161]
 [0.32290935 0.05235408]
 [0.04702614 0.95376066]
 [0.27955549 0.92774067]]

numpy.arange

Return evenly spaced values within a given interval.

>>> print(np.arange(4))
[0 1 2 3]

In [16]: print(np.arange(2,4))
[2 3]

In [17]: print(np.arange(0,100,10))
[ 0 10 20 30 40 50 60 70 80 90]

2.2.3.3. Operation on vectors.

Let's explore some operation with the vectors.

Indexing

Elements of vectors can be accessed via indexing and slicing. NumPy provides a very complete set of indexing and slicing capabilities.

In [18]: print(np.arange(0,100,10)[2])
20

If we pass a 'minus' index, we access the slots from the opposite side:

In [18]: print(np.arange(0,100,10)[-2])
80

Slicing

Slicing creates an array of indices using a set of three values (start:stop:step). A subset of values is also valid.

In [23]: print(np.arange(100)[10:])
[10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99]

In [22]: print(np.arange(100)[10:20])
[10 11 12 13 14 15 16 17 18 19]

In [26]: print(np.arange(100)[10:50:5])
[10 15 20 25 30 35 40 45]

Unary vector operations

Let's see how to operate with unary operations:

In [29]: print(-np.array([1,2,3,4]))
[-1 -2 -3 -4]

In [33]: print(np.sum(np.array([1,2,3,4]))) # Sum the components.
10

In [34]: print(np.mean(np.array([1,2,3,4]))) # Get the mean.
2.5

In [36]: print(np.array([1,2,3,4])**2) # Power to 2 the items of the list.
[ 1  4  9 16]

Vector element-wise operations

Most of the NumPy arithmetic, logical and comparison operations apply to vectors as well. These operators work on an element-by-element basis. For example:

In [2]: print(np.array([1,2,3,4])+ -np.array([1,2,3,4]))
[0 0 0 0]

for this to work correctly, the vectors must be of the same size.

In [3]: print(5*np.array([1,2,3,4]))
[ 5 10 15 20]

Vector dot product

Being $a,b \in V$, then we define the dot product of $a$ and $b$ as:

The dot product multiplies the values in two vectors element-wise and then sums the result. Vector dot product requires the dimensions of the two vectors to be the same.

Without it we would have to implement a for loop to first perform $n$ products and then do the summation, but NumPy has the .dot method:

In [7]: print(np.dot(np.array([1,2,3,4]),np.array([1,2,3,4])))
30

Let's also perform the same operation through the two methods and see which of the two spends more time computing the code and verify if the efficiency of NumPy is worth it.

First, take a look over the following test file:

import numpy as np
import time

def my_dot(a, b): 
    """
   Compute the dot product of two vectors
 
    Args:
      a (ndarray (n,)):  input vector 
      b (ndarray (n,)):  input vector with same dimension as a
    
    Returns:
      x (scalar): 
    """
    x=0
    for i in range(a.shape[0]):
        x = x + a[i] * b[i]
    return x


np.random.seed(1)
a = np.random.rand(10000000)  # very large arrays
b = np.random.rand(10000000)

tic = time.time()  # capture start time
c = np.dot(a, b)
toc = time.time()  # capture end time

print(f"np.dot(a, b) =  {c:.4f}")
print(f"Vectorized version duration: {1000*(toc-tic):.4f} ms ")

tic = time.time()  # capture start time
c = my_dot(a,b)
toc = time.time()  # capture end time

print(f"my_dot(a, b) =  {c:.4f}")
print(f"loop version duration: {1000*(toc-tic):.4f} ms ")

del(a);del(b)  #remove these big arrays from memory

The produced output is:

$ python3 test.py
np.dot(a, b) =  2501072.5817
Vectorized version duration: 14.1015 ms
my_dot(a, b) =  2501072.5817
loop version duration: 1158.4625 ms

2.2.3.4. Matrix and operation with matrices.

Matrices are two-dimensional arrays. The elements of a matrix are all of the same type.

In math settings, numbers in the index typically run from 1 to n. In computer science and these labs, indexing will run from 0 to n-1.

NumPy's basic data structure is an indexable, n-dimensional array containing elements of the same type (dtype). These were described earlier. Matrices have a two-dimensional (2-D) index [m,n].

Indexing

Matrices include a second index. The two indexes describe [row, column]. Access can either return an element or a row/column:

In [4]: a = np.arange(6).reshape(-1, 2) #Reshape allow to expand dimension
        print(f"a.shape: {a.shape}, \na= {a}")
a.shape: (3, 2),
a= [[0 1]
 [2 3]
 [4 5]]

3. Multiple Linear Regression.

Let's now extend the previously visited concepts and NumPy methods to build a Multi-Variable Linear Regression.

3.1. Defining the problem.

We will use the motivating example of housing price prediction. The training dataset contains three examples with four features (size, bedrooms, floors and age) shown in the table below.

3.2. Writing our code.

First, we start by importing the necessary libraries:

import copy, math
import numpy as np
import matplotlib.pyplot as plt # Visualize 2D,3D functions
np.set_printoptions(precision=2)  # reduced display precision on numpy arrays

Now, we introduce our training set $\mathcal{X} \times \mathcal{Y}$, being:

• $\mathcal{X}\subset Size \times NBedrooms \times NFloors \times Age: \mathcal{X} := \begin{cases} (2104, & 5, & 1, & 45) \\ (1416, & 3, & 2, & 40) \\ (852, & 2, & 1, & 35) \end{cases}$
• $\mathcal{Y} \in Price^3: \mathcal{Y}:= (460, 232, 178)$

This data is introduced making use of following arrays:

X_train = np.array([[2104, 5, 1, 45], [1416, 3, 2, 40], [852, 2, 1, 35]])
y_train = np.array([460, 232, 178])

Let's observe that we built the following structure from our data:

Let's remember that our goal is to find $f: \mathcal{X} \to \mathcal{Y}: f(x) = y \ \ \forall (x,y) \in \mathcal{X} \times \mathcal{Y}$.

The linear regression model assumes that there exists an affine transformation that approximates $f$ closely enough, in other words $\exists (w,b) \in \mathbb{Q^4} \times \mathbb{Q}$ such that:

Thus, $w \in \mathbb{Q}^4$ is a vector of $4$ elements $(w_1,w_2,w_3,w_4 )$ where $w_i$ determines how much $x_i$ of $x \in \mathbb{Q}^4$ affects the $f_{w,b}(x)$ value and $b \in \mathbb{Q}$ is a scalar parameter. Thus, we can reformulate $f_{w,b}$ as:

This function depends on parameters $w$ and $b$.

For demonstration, $w$ and $b$ will be loaded with some initial selected values that are near the optimal.

b_init = 785.1811367994083
w_init = np.array([ 0.39133535, 18.75376741, -53.36032453, -26.42131618])

In order to perform the operation we can use the .dot method of the NumPy module instead of crafting a for loop as we saw before:

def f(x, w, b): 
    """
    single predict using linear regression
    Args:
      x (ndarray): Shape (n,) example with multiple features
      w (ndarray): Shape (n,) model parameters   
      b (scalar):             model parameter 
      
    Returns:
      p (scalar):  prediction
    """
    p = np.dot(x, w) + b     
    return p

Now, let's model the cost function which measures the prediction error of $f_{w,b}$ for certain $w,b$ given by $\hat{y}^{(i)} := f_{w,b}(x^{(i)}) \simeq y^{(i)}$:

We can compute it by:

def J(X, y, w, b): 
    """
    compute cost
    Args:
      X (ndarray (m,n)): Data, m examples with n features
      y (ndarray (m,)) : target values
      w (ndarray (n,)) : model parameters  
      b (scalar)       : model parameter
      
    Returns:
      cost (scalar): cost
    """
    m = X.shape[0]
    cost = 0.0
    for i in range(m):                                
        f_wb_i = np.dot(X[i], w) + b           #(n,)(n,) = scalar (see np.dot)
        cost = cost + (f_wb_i - y[i])**2       #scalar
    cost = cost / (2 * m)                      #scalar    
    return cost

We want to find $\theta := (w,b) : \displaystyle\min_{\theta \in \Theta} \ J(\theta)$ and we go through the Gradient Descent algorithm. We remember that this algorithm tries to find a local minimum for $J$ by leveraging one fundamental fact: $\nabla J$ points to the direction which maximizes $J$, thus, we have to perform a finite sequence of steps of the form:

Where $\alpha$ is our learning rate.

Applying the formula above to our model, let's first unravel what $\nabla J$ is.

Being $\theta := (w_1,w_2,w_3,w_4,b)$, then, we define:

In order to compute the gradient, first, take the shape and craft the vector variable for each partial $\frac{\partial J}{\partial \theta}$:

m,n = X.shape           #(number of examples, number of features)
dj_dw = np.zeros((n,))
dj_db = 0.

Observe that in python m,n = X.shape is unpacking the dimension of the matrix $X$; also we are "initializing" to zero the partial vectors $dw$ and $db$.

Now, we feed each element of the vectors by mathematically computing the error and adding it to the vector entry, then divide the entry by $m$.

for i in range(m):                             
    err = (f(X[i], w, b)) - y[i]   
    for j in range(n):                         
        dj_dw[j] = dj_dw[j] + err * X[i, j]    
    dj_db = dj_db + err                        
dj_dw = dj_dw / m                                
dj_db = dj_db / m  

The whole code would be:

def nablaJ(X, y, w, b): # Gradient.
    m,n = X.shape           #(number of examples, number of features)
    dj_dw = np.zeros((n,))
    dj_db = 0.

    for i in range(m):                             
        err = (f(X[i], w, b)) - y[i]   
        for j in range(n):                         
            dj_dw[j] = dj_dw[j] + err * X[i, j]    
        dj_db = dj_db + err                        
    dj_dw = dj_dw / m                                
    dj_db = dj_db / m                                
        
    return dj_db, dj_dw

Also, this gradient computation must be inserted within a routine to become an algorithm:

def gradient_descent(X, y, w_in, b_in, cost_function, gradient_function, alpha, num_iters):
    
    # An array to store cost J and w's at each iteration primarily for graphing later
    J_history = []
    w = copy.deepcopy(w_in)  #avoid modifying global w within function
    b = b_in
    
    for i in range(num_iters):

        # Calculate the gradient and update the parameters
        dj_db,dj_dw = gradient_function(X, y, w, b)   

        # Update Parameters using w, b, alpha and gradient
        w = w - alpha * dj_dw               
        b = b - alpha * dj_db               
      
        # Save cost J at each iteration
        if i<100000:      # prevent resource exhaustion 
            J_history.append(cost_function(X, y, w, b))

        # Print cost every at intervals 10 times or as many iterations if < 10
        if i% math.ceil(num_iters / 10) == 0:
            print(f"Iteration {i:4d}: Cost {J_history[-1]:8.2f}   ")
        
    return w, b, J_history #return final w,b and J history for graphing

Observe that the code above is just assembling the pieces we already crafted in a step-by-step mechanism that resembles our gradient descent algorithm. By parts:

• First, we create some arrays to store the results and the modified data:

    J_history = []
    w = copy.deepcopy(w_in)  #avoid modifying global w within function
    b = b_in
  

• Then, we establish a number of iterations for our gradient descent algorithm which will determine how close we get to the local minimum and inside the loop we perform the approximation of the algorithm:

    for i in range(num_iters):
  
    # Calculate the gradient and update the parameters
    dj_db,dj_dw = gradient_function(X, y, w, b)   
  
    # Update Parameters using w, b, alpha and gradient
    w = w - alpha * dj_dw               
    b = b - alpha * dj_db               
      
    # Save cost J at each iteration
    if i<100000:      # prevent resource exhaustion 
        J_history.append(cost_function(X, y, w, b))
  
    # Print cost every at intervals 10 times or as many iterations if < 10
    if i% math.ceil(num_iters / 10) == 0:
        print(f"Iteration {i:4d}: Cost {J_history[-1]:8.2f}   ")
  

  
  

We can find the whole code in the following repository.

4. Problems with gradient descent.

4.1. Introducing the problem.

Sometimes, having a cost function $J(\theta)$ and applying a correct learning rate $\alpha$ doesn't conclude in a smooth convergence on the local minimum. In practice, the gradient descent applies a single learning rate $\alpha$ to update all parameters simultaneously; as we know, the algorithm performs the assignments as:

Let's observe that later, these new parameters, which are being updated in the same "proportion" $\alpha$, contribute to the next prediction, leading to the risk of creating disproportions between the features since each lives in its own range of values.

For example; in certain conditions, $x_1 \in[0,1] \wedge x_2 \in [0,50000] \implies \frac{\partial J}{\partial w_2} » \frac{\partial J}{\partial w_1}$ not because $w_2$​ is further from its optimum, but simply because $x_2$ has a larger numerical range. So the same $\alpha$ is simultaneously too large for $w_2$​ (causing oscillation or divergence) and too small for $w_1$​ (causing painfully slow progress).

You're forced into a compromise: pick a tiny $\alpha$ that prevents divergence along the sensitive direction, at the cost of crawling along every other direction.

The fundamental issue is a mismatch between the structure of the problem (features at wildly different scales) and the structure of the algorithm (a single scalar learning rate applied uniformly). The algorithm has no mechanism to treat each parameter according to its own geometry.

4.2. Solution to the problem; Feature Scaling.

Feature scaling is a preprocessing step that normalizes the range of input features before running gradient descent. The core reason is geometric: it reshapes the loss surface to make optimization dramatically more efficient.

Let's see some forms to actually do this.

4.3.1. Feature Scaling method.

If we have a $n$ number of features in each range of values as:

Then a solution would be to divide each feature between the maximum value of the range before operating with it:

Suppose for example that $x_1 \in [300,2000],x_2 \in [0,5]$, then $x_1' \in [0.15,1], x_2 \in [0,1]$. Geometrically, we would pass from ellipses to something more circular, allowing the path to the local minimum on the surface to be more accessible.

Note that the local minimum is accessible because the mapping $x \to x'$ we are applying here is an affine transformation; $x:= x'M +m$

4.3.2. Mean normalization.

There is also a variant of feature scaling called mean normalization; which is a feature scaling technique that transforms each feature so that it's centered around zero. Taking $x_i \in [m_i, M_i]$ then, mean normalization is the mapping given by:

Where $u_i$ is the sample mean, the mean of the feature $i$ of all the samples $x^{(j)}$ in the training set.

The result is that each feature ends up roughly in $[-0.5,0.5]$ (assuming a reasonably uniform distribution), with mean approximately zero.

4.3.3. Z-Score (Standardization).

The Z-score normalization (also called standardization) transforms each feature as:

Where $u_i$ still being the sample mean and $\sigma_i$ is the standard deviation which measures how spread out the values in a dataset are around their mean:

For each data point you compute how far it is from the mean $(x_i^{(j)} - \mu_i)$, square that distance (so negative and positive deviations don't cancel), average all those squared distances, and then take the square root to bring the result back to the original units of the feature.

The core difference is in the denominator. Mean normalization divides by the range $(\max - \min)$, while Z-score divides by $\sigma$ which is more robust because it maps based on how far each feature is from the mean instead of simply generalizing the range for every feature. Geometrically, both aim at the same thing: making the level sets of $J(\theta)$ more isotropic (closer to spherical) so that $\nabla J$ points more directly toward the optimum. But Z-score does this more reliably because $\sigma$ is a better measure of the "typical spread" of a feature than the range is.

Thus; if the features are reasonably well-behaved (no extreme outliers, roughly symmetric), both methods give you similar results, the contours get reshaped about equally well, and gradient descent converges in roughly the same number of steps. The difference becomes significant when the data has heavy tails or outliers, where Z-score keeps the scaling stable while mean normalization can degrade.

5. Gradient Descent efficiency. Loss curve & Learning rate.

5.1. Introducing the concept.

Let's now suppose we have a good gradient descent algorithm and we want to run it and, as it runs, we want to be sure when the cost function $J$ converges to the local minimum.

For that purpose we build what we call the loss curve or training curve, which is simply the plot of your objective function $J(\theta^{(t)})$ (which in our case gets instantiated as the cost function) evaluated at each iterate $\theta^{(t)}$ against the iteration index $t$.

It's the most direct tool to determine whether gradient descent is converging, how fast it's converging, and whether your hyperparameters are well-chosen. As the picture shows, a healthy loss curve is monotonically decreasing and flattening toward an asymptote (our local minimum).

5.2. Formal setup.

Let $J : \mathbb{R}^n \to \mathbb{R}$ be your loss function and the gradient iteration:

being $\alpha \in \mathbb{R}$ the learning rate. Then, we define the loss curve as the sequence:

Assuming that $J$ is $\beta$-smooth, meaning $\nabla J$ is $\beta$-Lipschitz:

Conceptually, a function is β-Lipschitz when it has a bounded rate of change, it cannot "jump" too fast. Observe that if the gradient represents the "change ratio" of the function, how the function changes between two points is no greater than the metric between those same points (up to parameter factors). The change ratio grows (or decreases) proportionally per unit of distance in the domain (up to parameter factors).

Then, by the descent lemma it is:

The descent lemma gives you a guaranteed upper bound on how much a β-smooth function can deviate from its linear approximation. It's the formal reason why gradient descent works, it tells you that if you take a step in the direction of $-\nabla J(x)$, you can guarantee the function decreases, provided your step isn't too large.

Thus, this basically guarantees that the sequence we called loss curve smoothly decreases at each step until $\nabla J =0$

5.3. Choosing the learning rate.

As we set before, given a good gradient descent algorithm (a well-chosen learning rate $\alpha$) the results above tell us that the gradient descent algorithm should make the $J$ function decrease smoothly (descent lemma).

Whenever we see that our loss curve does not behave like that then mathematically we can ensure that the gradient descent algorithm is not well-formed.

A recommended good practice is to choose a small $\alpha$ and then increase it slowly. If this does not fix the loss curve, maybe there is a bug in the code.

1. Machine Learning
2. Formal Framing
3. Learning Algorithm
   • 3.1. Mathematical Prerequisites
     • 3.1.1. Functions of Several Variables
     • 3.1.2. Partial Derivatives
     • 3.1.3. Directional Derivatives
     • 3.1.4. Gradient
     • 3.1.5. Taylor Expansions
   • 3.2. Formal Development
     • 3.2.1. Hypothesis Sets
     • 3.2.2. Measuring the Error: Cost Function
   • 3.3. Gradient Descent Algorithm
     • 3.3.1. Optimization. Classical Approach
     • 3.3.2. Optimization. Iterative Approach
     • 3.3.3. Explaining the Algorithm
     • 3.3.4. Descent Mechanism

• 1. Summary

  
  

1. Machine Learning.

In conventional programming, explicit rules are taught to programs in order to establish solid logic predicates about constraints or bifurcations over the execution flow of the program.

Machine Learning is a family of algorithms that learn patterns from data instead of following explicitly programmed rules. In other terms, Machine Learning is a field of study that gives computers the ability to learn without being explicitly programmed.

2. Formal Framing.

Let's understand what "learning from data" actually mathematically means; the objective is to find a function that captures the regularity of an observed phenomenon that produces data in order to make predictions on new, unseen inputs of the same nature.

Formally, we have the following pieces:

• An input space, the set of all possible inputs. For instance $\mathcal{X} = \mathbb{R}^n$, thus each input $x$ is a vector of $n$ features.

• An output space $\mathcal{Y}$, which is the set of all possible outputs. This depends on what is being modeled; in classification problems the output set can be a finite subset of the natural numbers which are the clusters, $\mathcal{Y} = [n]$; for regression, which wants to predict a continuous value, it is $\mathcal{Y} = \mathbb{R}$.

• The target function, the ideal mapping $\mathcal{X} \to \mathcal{Y}$ that relates each input $x \in \mathcal{X}$ to the desired output $y \in \mathcal{Y}$ which is abstracted in the application $f : \mathcal{X} \to \mathcal{Y}$ and remains unknown. This deterministic presentation is purely pedagogic and more complex in reality, the precise form would be to say that the data pairs arise from an unknown joint probability distribution $P$ over $\mathcal{X} \times \mathcal{Y}$.

• The training set; a finite subset $\mathcal{D} \subset \mathcal{X} \times \mathcal{Y}$ defined as $\mathcal{D} := \Set{(x_i,y_i)}_{i=1}^m : y_i = f(x_i)$

The goal of ML is to use $\mathcal{D}$ to find a mapping $g_{\mathcal{D}}:\mathcal{X} \to \mathcal{Y} : g_\mathcal{D} \approx f$. Observe that $g_\mathcal{D}$, by definition, allows us to make predictions along all the $\mathcal{X} \times \mathcal{Y}$ space, this is, not only for the inputs from $\mathcal{D}_\mathcal{X}$ but for any $x \in \mathcal{X}$.

We don't search across all possible functions $f: \mathcal{X} \to \mathcal{Y}$; that set is unimaginably large and the problem would be ill-defined. Instead, we choose a hypothesis set; $\mathcal{H}$, which is a restricted family of candidate functions. A learning algorithm selects one of the candidates of $\mathcal{H}$ as $g_\mathcal{D}$.

For example, the linear regression learning model is a supervised learning model (we will see what this is later) in which we assume that $\mathcal{H}$ is the set of all the affine transformations $\mathbb{R}^n \to \mathbb{R}$.

3. Learning Algorithm.

As we set in the previous section, we need to develop a mechanism that allows us to select the best candidate from a family of applications $\mathcal{H}$ that maps correctly - matching our needs - the input $\mathcal{X}$ with the outputs $\mathcal{Y}$ using only the training set $\mathcal{D} \subset \mathcal{X} \times \mathcal{Y}$ established before as a reference. Overall, this selection is based on the question of which $h \in \mathcal{H}$ minimizes as much as possible the noise or incorrectness in the mapping $\mathcal{X} \to \mathcal{Y}$ done by $h$.

The learning algorithm is the procedure that actually performs this search. In practice, for the vast majority of modern ML, this reduces to a continuous optimization problem: minimize a real-valued function (the loss function; the one that measures the error over the predictions) over a space of parameters.

3.1. Mathematical Prerequisites.

3.1.1. Functions of Several Variables.

A function of $n$ real variables is a map:

Where:

• $S$ is the evaluation domain
• $\mathbb{R}$ is the codomain

Then, the input is a vector with $n$ coordinates $x := (x_1,\cdots,x_n) \in \mathbb{R}^n$ and the image of $x$ through $f, f(x) \in \mathbb{R}$ is a real scalar.

It is worth visualizing that for $n = 2$, the plot of $f(x,y)$ on $x,y$ is a surface given by the parameters $(x,y,f(x,y)) \in \mathbb{R}^3$, $f(x,y)$ is the height in the plane formed by the coordinates $(x,y) \in \mathbb{R}^2$. Now, if $n >3$ the geometric visualization breaks, while the math holds in the sense that $f(x)$ is the height at each point of $\mathbb{R}^n$.

In ML, the models we work with depend on a vector of adjustable parameters $\boldsymbol{\theta} \in \mathbb{R}^p$ where $p$ depends on the model architecture and can range from a handful to billions. Training these models requires understanding how each parameter affects the output, which demands the machinery of functions of several variables.

3.1.2. Partial Derivatives.

Let $ f: S \to \mathbb{R}, \quad S \subseteq \mathbb{R}^n$ be now a multivariable function; let's choose a point $\mathbf{a}:=(a_1,\ldots,a_n) \in S$.

Then, we define the partial derivative of $f$ respect to $x_i$ at $\mathbf{a}$ as (provided this limit exists):

Where $e_i$ is the $i$-th standard basis vector.

It is important to note the partial derivative is just the extension of the single-variable derivative applied to a multivariable function on one single feature $x_i$, meaning that the scalar $t \in \mathbb{R}$ perturbs only the $i$-th coordinate (literally an ordinary single-variable derivative applied to a function where all variables except one have been frozen to constants).

A formal treatment is to define $g_i: \mathbb{R} \to \mathbb{R}$ from $f$ by:

Then, the partial derivative acquires another representation:

A simple exchange $g_i \leftrightarrow f$ returns to us the original definition, but this shape gives a more intuitive approach since we can understand the partial derivative from our understanding of the single-derivative concept with which we are already familiar.

The formula above tells us that the partial derivative tells us how $f$ changes under a minimal (differential) change of the $i$-th feature, maintaining the rest of the variables as constants.

Geometric Interpretation

Let's take the case $n=2$, the graph of $f : \mathbb{R}^2 \to \mathbb{R}$ is a surface on $\mathbb{R}^3$; $(x,y,f(x,y)) \in \mathbb{R}^3$.

In this context, the function $g_x:\mathbb{R} \to \mathbb{R} \ \vert \ g_x(t) := f(t,y_0)$ describes the curve of $f$ in the plane $(x,f(x,y_0)) \in \mathbb{R}^2$ which is the section given by the plane $y=y_0$ on the surface $(x,y,f(x,y)) \in \mathbb{R}^3$

This way, $\frac{\partial f}{\partial x}(a) = g_x'(a)$ is the slope of the tangent line to that curve at the point $(a_1,a_2,f(a_1,a_2))$. The geometric interpretation of the single-variable derivative carries over directly: you are just reading the slope on a specific slice of the surface:

In general, the graph of $f:\mathbb{R}^n \to \mathbb{R}$ is a hypersurface in $\mathbb{R}^{n+1}$, freezing all coordinates except $x_i$ defines a line in $\mathbb{R}^n$, the image of that line under the graph map $x \mapsto (x, f(x))$ is a curve on the hypersurface, and $D_i​f(\mathbf{a})$ is the slope of the tangent line to that curve.

In summary, partial derivatives measure how $f$ changes when we adjust one parameter while keeping all others fixed. This is needed to form the gradient which basically is a vector that tells us the movement tendency of $f$ at a point in $\mathbb{R}^n$

3.1.3. Directional Derivatives.

Definition

Let's barely introduce what the directional derivative is.

Observe that the partial derivatives we've just presented above measure the change ratio along the coordinate that isn't frozen. In the example we proposed, the curve is the intersection of the surface with the plane $y = y_0$, thus, the partial derivative explore how $f$ changes along the $x$ coordinate axis.

Thus, partial derivatives measure rates of change along coordinate axes, now directional derivatives introduce how to measure this change along virtually any direction. The directional derivative generalizes partial derivatives to arbitrary directions.

Let $E \subseteq \mathbb{R}^n$ be open, and let $f: E \to \mathbb{R}$ be differentiable at $\mathbf{a} \in E$. Then, we define the directional derivative of $f$ at $\mathbf{a}$ in the direction of $\mathbf{u}$:

Where $\mathbf{u}$ is a unit vector, a vector with direction but without disturbing magnitude (this is, so to speak, the modulus is $\vert \vert u \vert \vert = 1$, thus it does not disturb the magnitude of any other vector). Let's take a closer look.

If is $\mathbf{u} := (u_1,\ldots,u_n) \in \mathbb{R}^n : \vert \vert u \vert \vert = 1$, then $t\mathbf{u}$ is a scalar multiplication on $\mathbf{u}$,

Every coordinate gets perturbed simultaneously, each by the amount $tu_i$​. The vector $\mathbf{u}$ controls the direction of the perturbation, and the scalar $t$ controls how far you go in that direction.

Geometric Interpretation

This means that the geometric interpretation remains the same, but now the plane contains the line passing through the point $\mathbf{a}$ with the direction of the unit vector $\mathbf{u}$. This plane is no longer parallel to any coordinate axis but extends in the direction of $\mathbf{u}$.

As a visual example, for $f:\mathbb{R}^2 \to \mathbb{R}$ the intersection of this plane and the surface gives us a curve containing $(\mathbf{a},f(\mathbf{a}))$, then $D_uf(\mathbf{a})$ is the slope of the tangent line to the curve at $\mathbf{a}$

3.1.4. Gradient.

Definition

Consider now $f: S \subset \mathbb{R}^n \to \mathbb{R}$ and a point $\mathbf{a} \in \mathbb{R}^n: \exists \partial_if(\mathbf{a}) \ \forall i \in [n]$. We define as the gradient of $f$ at $\mathbf{a}$ to the vector:

Observe that the gradient lives in the domain of $f$, $\mathbb{R}^n$, not in the hypersurface $\mathbb{R}^{n+1}$.

Geometric Interpretation

Now, lets recover the directional derivative definition:

Observe that, by the chain rule (assuming $f$ is differenciable at $\mathbf{a}$), we have the following result:

Observe that $\nabla f(\mathbf{a})\ · \mathbf{u} = \Vert \nabla f(\mathbf{a}) \Vert \Vert \mathbf{u} \Vert \cos\theta$, which means that $D_\mathbf{u} f(\mathbf{a})$ is maximum when $\mathbf{u}$ and $\nabla f(\mathbf{a})$ have the same direction $\cos\theta = 1 \implies \theta = 0 \pmod {2\pi}$.

Thus, in general, the gradient $\nabla f(\mathbf{a})$ always points towards the direction in which the slope of the tangent line to the curve on the surface ($D_\mathbf{u}f(\mathbf{a})$) is maximum, this is what we call the steepest ascent direction.

Then $-\nabla f(\mathbf{a})$ always points towards the direction in which the slope is minimum, the steepest descent direction. This geometric property justifies the gradient descent algorithm we will see later.

3.1.5. Taylor Expansions.

Taylor formulas solve a fundamental problem; locally approximating a complicated function by a polynomial, which is the most tractable object in analysis.

The result asserts that, if a function is smooth enough (differentiable several times) at a point $\mathbf{a}$, then its behavior near $\mathbf{a}$ is captured, with controlled error, by a polynomial whose coefficients are the successive derivatives at $\mathbf{a}$.

Taylor Theorem (with Lagrange Remainder)

Let $f: [a, b] \to \mathbb{R}$ with $f^{(n)}$ continuous on $[a,b]$ and $f^{(n+1)}$ existing on $(a,b)$. Then:

Observe that, despite being an exact equality (the function is its Taylor polynomial plus an error term with an explicit form), the $c$ term in the Lagrange remainder remains unknown.

Nevertheless, the result is powerful since we are predicting information about $f$ in an interval using only local information at a single point.

Expansion: Peano's Remainder

There is another form of the remainder that is weaker but often more useful for optimization arguments. Under the same hypothesis:

where $o(|x-a|^n)$ means that $\displaystyle\frac{R_n(x)}{|x-a|^n} \to 0$ as $x \to a$ and at some point where, $x$ becomes close enough to $a$, the error factor becomes negligible.

Definition: Taylor's multivariate extension and Geometric Intuition

For $f: \mathbb{R}^d \to \mathbb{R}$ differentiable on $\mathbf{a}$, the first-order Taylor expansion around a point $\mathbf{a}$ evaluated at $\mathbf{a} + \mathbf{h}$ is:

Where $ o( \Vert \mathbf{h} \Vert)\to 0$ as $ \alpha \to 0$.

At $\mathbf{a}$, the graph of $f$ (a surface in $\mathbb{R}^{p+1}$) has a tangent hyperplane. The Taylor expansion says: the function value at a nearby point equals the tangent hyperplane's value at that point, plus an error that becomes negligible as you zoom in.

3.2. Formal Development.

3.2.1. Hypothesis Sets.

As we said before, we don't search for a hypothesis $h$ in the abstract; we first form a family of candidates $\mathcal{H}$. This is parameterized by a vector of real numbers $\theta \in \mathbb{R}^p$:

Where,

• $\theta = (\theta_1, \ldots, \theta_p) \in \mathbb{R}^p$.

  Is the parameter vector. Each specific $\theta$ picks out one specific hypothesis $h_{\theta} \in \mathcal{H}$​.

• $p \in \mathbb{N}$, is the number of parameters (in modern deep learning, $p$ can be in the billions).

• $\Theta \subseteq \mathbb{R}^p$ is the parameter space over all $\mathbb{R}^p$

Linear Model

As a brief example, let's take a look at the linear model, which, as we said before, is the assumption that $h$ is an affine transformation. Remember that an affine transformation is a linear transformation $y=mx$ plus a translation $b$, taking the form of the line: $y_{m,b} := b + mx$, then in our parameterized function:

Where $\theta_0$ is the bias and $\theta_i \ \forall i \in [p-1]$ is the parameter of the $i$-th feature $x_i$. Observe that each $\theta$ gives us a different linear function.

3.2.2. Measuring the error: Cost function.

It is reasonable to ask, for a certain $\theta$, how far the prediction is from the real, labeled, output. Formally; for a pair $(x,y) \in \mathcal{X} \times \mathcal{Y}$, how wrong is the prediction $h_\theta(x)$, let's first formalize the notion of 'wrong' in mathematical terms.

Loss Function

Let's define:

$L(y,h_\theta(x))$ takes the true value $y$ and the predicted value $h_\theta(x)$ and returns a non-negative real number: the penalty for that single prediction. The larger $L(y,h_\theta(x))$, the worse the prediction.

The loss function is a design choice; different losses encode different priorities about what kinds of errors matter based on the space in which the output set is defined. For example, applied to the linear regression model, the output space is $\mathbb{R}$ which is a metric space, thus, the loss function adopts the form of the squared loss; penalizes errors proportionally to the square of the deviation; large errors are punished disproportionately.

But note that $L$ only measures error at a single point. A model doesn't predict one point; it predicts across the entire input space. We need to aggregate.

True Risk

The natural next question: how wrong is $h_{\boldsymbol{\theta}}$​ across all the input space.

Ideally, we would measure the expected loss over the entire data-generating process. The data comes from some unknown probability distribution $P$ over $\mathcal{X} \times \mathcal{Y}$; this unknown probability pattern associates inputs to outputs in what we call the phenomenon data. Think for example of the height and the age of a person; it is the formal way of saying "nature produces input-output pairs according to some pattern".

The population risk, also called true risk, of a hypothesis $h$ is:

This is a sum (or integral, in the continuous case) over all possible pairs $(x, y) \in \mathcal{X} \times \mathcal{Y}$, each weighted by its probability under $P$. It captures the average error of $h$ across the entire distribution, including inputs you'll never see in your training set.

This the object we truly want to minimize. It captures how well $h$ performs on all data, including data we haven't seen.

However, the fundamental problem is we cannot compute $R(h)$ since it is based on $P$ and the distribution $P$ is unknown (if we knew it, we would know $f$, and there would be no learning problem). $R(h)$ is a theoretical ideal, not a computable quantity.

Empirical Risk

A reasonable approximation to the true risk (based on $P$) is the empirical risk based on a dataset we have access to: $\mathcal{D}$. The empirical risk of $h$ on $\mathcal{D}$ is:

Observe that, as a discrete, finite sample, the expected value $\mathbb{E}_{(x,y) \sim P}$ converges to a sum. Each training example gets weight as $\frac{1}{m}$, we treat them all equally because we don't know $P$ so we assume that they are all probably equal.

This is the average loss computed on the $m$ training examples a quantity we can compute, because we know every $(x^{(i)}, y^{(i)}) \in \mathcal{X} \times \mathcal{Y}$ and we can evaluate $h(x^{(i)})$.

The empirical risk is a finite-sample estimate of the true risk. Under reasonable conditions, the law of large numbers guarantees that as $m \to \infty \implies \hat{R}_{\mathcal{D}}(h) \to R(h)$.

At this point, we would think that an acceptable strategy to choose $h_\theta$ is to choose an $h$ that minimizes $\hat{R}_{\mathcal{D}}$​, which is what we call the Empirical Risk Minimization:

Let's observe that $\min_{a\in A}​f(a)$ is the minimum value $f$ returns in a subset of his evaluation domain, then $\arg\min_{a\in A}​f(a)$ is the input (the argument) that produces that minimum value.

This is conceptually clean, but still abstract because we're minimizing over a set of functions $\mathcal{H}$. To make this into an actual algorithm we can run on a computer, we need one more step.

The Cost Function

Since every $h \in \mathcal{H}$ is indexed by a parameter vector $\theta \in \mathbb{R}^p$, the empirical risk becomes a function of $\theta$:

This function measures the global error in the training set between the prediction and the labeled data; it is the cost function (also called objective function). Observe carefully that having a provided training set $\mathcal{D}$, and an appropriate loss function $L$, the last free variable is $\theta$.

The learning problem has become to find:

Find the point in $\mathbb{R}^p$ where $J$ attains its minimum. This is a standard optimization problem where all the math presented before applies.

Each step is forced by the limitations of the previous one: $L$ only measures one point, so we average to get $R$; $R$ is not computable, so we approximate with $\hat{R}_{\mathcal{D}}$​;$\hat{R}_{\mathcal{D}}$​ is still abstract over $\mathcal{H}$, so we parameterize to get $J(\theta)$, which is a concrete function in $\mathbb{R}^p$ that we can differentiate and minimize with gradient descent.

3.3. Gradient Descent Algorithm.

We arrived at a concrete problem: we modeled a function $J$ that measures the error of the prediction function $h_\theta$ over the training set $\mathcal{D} \subset \mathcal{X} \times \mathcal{Y}$ that depends on the parameter $\theta$. The objective is to find the $\theta$ that minimizes the error, this is, that minimizes the $J$ output.

3.3.1. Optimization. Classical Approach.

An optimization problem is, in its most general form, the task of finding the input to a function that produces the smallest (or largest) output value. For our purposes, we want to find the value for $\theta \in \mathbb{R}^p$ that minimizes the $J$ output. Again:

That's it, we have a real-valued function of several variables and we want the point where it attains its minimum. This is an extremely old and central problem in mathematics. The ML learning problem becomes an optimization problem the moment we parameterize $\mathcal{H}$ in terms of $\theta$ and define $J(\theta)$; finding the best hypothesis reduces to finding the lowest point of a surface in $\mathbb{R}^p$.

Classically, for example, in a one-variable function context, we would leverage the geometric interpretation of the derivative of a continuous function at a point (with which we are already familiar from the prerequisite section) as the slope of the tangent line to the function and attempt to calculate the point in which that slope is horizontal, this is, zero: $\theta_0 \in \mathbb{R} : J'(\theta_0) = 0$

This is a first approach and necessary condition (although not sufficient since other points like local maxima satisfy it). In multiple variables, as far as we know, the most similar tool we have is the gradient which we already saw points towards the steepest ascent direction. This property is furnished from each partial derivative which have a similar geometric interpretation as the derivative in the example above. In a local minimum, the slope of the tangent line to the curve formed by the surface of the function and each plane $x_i=x_0$ is $0$, thus, $\nabla J(\theta_0) = 0$ and points to no direction.

Successful Example. Linear equations.

Let's take the linear regression model presented before, the hypothesis $h_\theta(\mathbf{x}) = \theta_0 + \boldsymbol{\theta}^\top \mathbf{x}$, and let's suppose that our loss function is the squared loss $L(y,h_\theta(\mathbf{x})) = (y - h_\theta(\mathbf{x}))^2$. Then, the cost function would be:

And the gradient is:

Observe that $\nabla J = \mathbf{0}$ becomes a system of $p$ linear equations in $p$ unknowns. The solution tells us exactly the point because linear algebra gives us a well-known algebraic method to solve linear systems of equations.

Unsuccessful example. Non-linear equations.

Let's consider now a simple neural network. The relations are multiple and $h_\theta$ is not linear in $\theta$, as an example:

where $\sigma$ is a nonlinear function.

When you compute $\nabla J$ and set it to $0$, you get a system of $p$ equations in $p$ unknowns where the equations involve more than additions and scaling of the unknowns (products of parameters, exponentials of parameters, compositions of nonlinear functions of parameters, etc).

This is a nonlinear system of equations, and there is no general algebraic method for solving systems of nonlinear equations. Also, even if a solution is found $J$ typically has multiple critical points, many local minima, local maxima, and saddle points (modern models handle millions or billions of parameters). The equation $\nabla J = 0$ synthesizes the constraints imposed over all the critical points; solving it gives you all those points and distinguishing among them is itself a hard problem.

Thus, as a summary, the problems of the direct approach are Algebraic Intractability, Multiple critical points and Scale.

3.3.2. Optimization. Iterative Approach.

Since we can't find $\theta^\ast$ in one shot, we settle for an iterative strategy: start at some $\theta^{(0)}$, and apply a rule that produces a finite $k$ steps sequence $\theta^{(0)}, \theta^{(1)}, \theta^{(2)}, \dots, \theta^{(k)}:J(\theta^{(t+1)})<J(\theta^{(t)}) $, this is that $J$ decreases at each step.

We may never reach $\theta^* \in \Theta : \nabla J(\theta^*)=0$ exactly, but we can get arbitrarily close which in practice, is perfectly adequate. The core insight is we trade an unsolvable global equation for a sequence of cheap local computations.

3.3.3. Explaining the algorithm.

Until now we have presented the problem and developed a motivation about the need for an iterative method.

Let's develop the brief explanation presented before about the Gradient Descent Algorithm. We said that we need a finite sequence $\theta^{(0)}, \theta^{(1)}, \ldots$ such that the image of $\theta$ through $J$ gets smaller in each step. Thus, given a position $\theta^{(t)}$ we need a procedure to calculate some $\theta^{(t+1)}:J(\theta^{(t+1)}) < J(\theta^{(t)})$ here is when we retrieve the Gradient concept presented on section $1.3.1.4$

We already explained that, being $f :S \subset \mathbb{R}^n \to \mathbb{R}$, the gradient $\nabla f(\mathbf{a}) \in \mathbb{R}^n$ is a vector that points towards the steepest ascent, meaning that it provides information about the behaviour of $f$ in an infinitesimal environment of $\mathbf{a}$ aiming towards the direction that maximizes $f$'s growth (it doesn't aim at the global maximum nor the local maximum).

We remember that if $\nabla f$ points to the steepest ascent, then $-\nabla f$ points to the steepest descent. Thus, recovering our cost function $J$, for a generic $\theta^{(t)}$ the next $\theta^{(t+1)}$ can be calculated according to the gradient:

Where $\alpha > 0$ is called the learning rate and determines how wide is the step between $\theta^{(t)}$ and $\theta^{(t+1)}$.

If $\theta \in \mathbb{R}^p$, then, a component-wise reformulation would be:

Each parameter $\theta_j$ is adjusted independently by the partial derivative of $J$ with respect to that parameter, scaled by $\alpha$.

3.3.4. Descent Mechanism.

The algorithm, as we just saw, is pretty simple in theory. We only need to adjust the learning rate parameter and, iteratively, we can get as close as we want to a local minimum.

Let's explore now why this descent is mathematically guaranteed. As a brief explanation, Gradient Descent works since Taylor expansions let you "see beyond" each current point when the next point is close enough. Taylor allows reconstructing $J$'s behavior in a neighborhood around $\theta^{(t)}$, making it possible to see clearly that the next point $J(\theta^{(t+1)})$ is smaller than the current one $J(\theta^{(t)})$.

Consider again the cost function we built above such $J : \mathbb{R}^p \to \mathbb{R}$

For our purposes, we will assume that $J$ is differentiable; this assumption holds for the models we've seen so far, although there are activation functions used in practice that are not differentiable everywhere.

Let's take, if $\theta^{(t+1)}:= \theta^{(t)} - \alpha \nabla J(\theta^{(t)}): \alpha \in \mathbb{R}_{>0} \implies \mathbf{h} = - \alpha \nabla J(\theta^{(t)})$, then the first-order expansion:

Observe that we can simplify the expression:

• The linear term:

  \[\nabla J(\boldsymbol{\theta}^{(t)}) \cdot \big(-\alpha\,\nabla J(\boldsymbol{\theta}^{(t)})\big) = -\alpha\,\nabla J(\boldsymbol{\theta}^{(t)}) \cdot \nabla J(\boldsymbol{\theta}^{(t)}) = -\alpha\,\left \Vert \nabla J(\boldsymbol{\theta}^{(t)})\right \Vert ^2\]

  We used the property that $\mathbf{v} \cdot \mathbf{v} = \Vert \mathbf{v} \Vert ^2 \ \ \ \forall \mathbf{v} \in \mathbb{R}^p$

  
  

• The error term:

  \[o\!\left(\left \Vert -\alpha\,\nabla J(\boldsymbol{\theta}^{(t)})\right \Vert \right) = o\!\left(\alpha\,\left \Vert \nabla J(\boldsymbol{\theta}^{(t)})\right \Vert \right)\]

  since $ \Vert -\alpha \mathbf{v} \Vert = \vert \alpha \vert \Vert \mathbf{v} \Vert = \alpha \Vert \mathbf{v} \Vert$ (as $\alpha > 0$).

  
  

Then, assembling: \(J(\boldsymbol{\theta}^{(t)}) = J(\boldsymbol{\theta}^{(t+1)}) + \alpha\,\left \Vert \nabla J(\boldsymbol{\theta}^{(t)})\right \Vert ^2 - o\!\left(\alpha\,\left \Vert \nabla J(\boldsymbol{\theta}^{(t)})\right \Vert \right)\)

Observe that $\alpha\, \Vert \nabla J(\boldsymbol{\theta}^{(t)}) \Vert ^2 - o(\alpha\, \Vert \nabla J(\boldsymbol{\theta}^{(t)}) \Vert ) > 0$ since $o(\alpha \Vert \nabla J \Vert )$ goes to zero faster as $\alpha \to 0$ while the first term has the fixed positive coefficient $ \Vert \nabla J \Vert ^2$; thus a small enough learning rate $\alpha > 0$ allows us to assert:

Observe that, in a pragmatic way, the role of the learning rate is crucial; all the theoretical justification spins around the size of $\alpha$. Too large: the steps overshoot the minimum; too small: convergence is extremely slow. The algorithm takes tiny steps and may require an impractical number of iterations.

There is no closed-form formula for the optimal learning rate in general; it is tuned empirically or adapted by more sophisticated algorithms.

4. Summary.

Thus, essentially, Machine Learning is applied maths.

1.1. Introducing machine learning.

Machine learning is about designing algorithms that automatically extract valuable information from data. The emphasis here is on “automatic”, machine learning is concerned about general-purpose methodologies that can be applied to many datasets, while producing something that is meaningful.

There are three concepts that are at the core of machine learning: data, a model, and learning.

• Since machine learning is inherently data driven, data is at the core of machine learning.

  The goal of machine learning is to design general-purpose methodologies to extract valuable patterns from data, ideally without much domain-specific expertise. For example, given a large corpus of documents (e.g., books in many libraries), machine learning methods can be used to automatically find relevant topics that are shared across documents.

  
  

• To achieve this goal, we design models that are typically related to the process that generates data, in order to model the dataset we are given.

  For example, in a regression setting, the model would describe a function that maps inputs to real-valued outputs. To paraphrase Mitchell (1997): A model is said to learn from data if its performance on a given task improves after the data is taken into account. The goal is to find good models that generalize well to yet unseen data, which we may care about in the future.

  
  

• Learning can be understood as a way to automatically find patterns and structure in data by optimizing the parameters of the model.

  
  

While machine learning has seen many success stories, and software is readily available to design and train rich and flexible machine learning systems, we believe that the mathematical foundations of machine learning are important in order to understand fundamental principles upon which more complicated machine learning systems are built. Understanding these principles can facilitate creating new machine learning solutions, understanding and debugging existing approaches, and learning about the inherent assumptions and limitations of the methodologies we are working with.

1.2. Finding words for intuitions.

A challenge we face regularly in machine learning is that concepts and words are slippery, and a particular component of the machine learning system can be abstracted to different mathematical concepts.

For example, the word “algorithm” is used in at least two different senses in the context of machine learning:

• In the first sense, we use the phrase “machine learning algorithm” to mean a system that makes predictions based on input data. We refer to these algorithms as predictors.

• In the second sense, we use the exact same phrase “machine learning algorithm” to mean a system that adapts some internal parameters of the predictor so that it performs well on future unseen input data. Here we refer to this adaptation as training a system.

We attempt to make the context sufficiently clear to reduce the level of ambiguity.

A model is typically used to describe a process for generating data, similar to the dataset at hand. Therefore, good models can also be thought of as simplified versions of the real (unknown) data-generating process, capturing aspects that are relevant for modeling the data and extracting hidden patterns from it. A good model can then be used to predict what would happen in the real world without performing real-world experiments.

We now come to the crux of the matter, the learning component of machine learning.

Assume we are given a dataset and a suitable model. Training the model means to use the data available to optimize some parameters of the model with respect to a utility function that evaluates how well the model predicts the training data. Most training methods can be thought of as an approach analogous to climbing a hill to reach its peak. In this analogy, the peak of the hill corresponds to a maximum of some desired performance measure. However, in practice, we are interested in the model to perform well on unseen data. Performing well on data that we have already seen (training data) may only mean that we found a good way to memorize the data. However, this may not generalize well to unseen data, and, in practical applications, we often need to expose our machine learning system to situations that it has not encountered before.

Let us summarize the main concepts of machine learning that we cover in this book:

• We represent data as vectors.
• We choose an appropriate model, either using the probabilistic or optimization view.
• We learn from available data by using numerical optimization methods with the aim that the model performs well on data not used for training.

When formalizing intuitive concepts, a common approach is to construct a set of objects (symbols) and a set of rules to manipulate these objects. This is known as an algebra; the study of symbolic manipulation and relation of structures. Essentially, algebra studies what is preserved under operations and mappings.

Linear Algebra is the study of those properties that are preserved under linear transformations (transformations that respect addition and scalar multiplication), these are: vectors spaces.

A vector is an element of a vector space. A vector space is not a bare set; it is a set $S$ equipped with two operations (addition and scalar multiplication over a field $F$) satisfying eight axioms. The definition is purely structural: vectors have no privileged concrete representation. Arrows, tuples, polynomials, and matrices can all be vectors when you define the right operations on them and verify the axioms.

In general, vectors are special objects that can be added together and multiplied by scalars to produce another object of the same kind. From an abstract mathematical viewpoint, any object that satisfies these two properties can be considered a vector. Some examples of such vector objects are geometric vectors, polynomials, tuples of $\mathbb{R}^n$ or matrices.

2. System of Linear Equations.

2.1. Formal definition.

A system of linear equations over a field $F$ is a finite collection of equations of the form:

Where the $a_{ij}, b_i \in F$ are given scalars and $x_1, \ldots, x_n$​ are unknowns. They are basically a set of constraints that points of $F^n$ must satisfy simultaneously.

The solution set is $S = {x \in F^n : Ax = b}$. Three things can happen:

• $S = \varnothing$ (inconsistent)

• $\vert S \vert = 1$ (unique solution)

• $\vert S \vert = \infty$

When $b = 0$ (the homogeneous case), $S$ is always nonempty (it contains $0 \in F^n$) and is in fact a subspace of $F^n$.

2.2. Geometric Interpretation of Systems of Linear Equations.

As an illustrative example of how these systems work as constraints over points, let's talk about the geometric interpretation of a system of linear equations in two variables.

In a system of linear equations with two variables $x1, x2$, each linear equation defines a line on the $x1x2$-plane.

Since a solution to a system of linear equations must satisfy all equations simultaneously, the solution set is the intersection of these lines.

This intersection set can be a line (if $\vert S \vert = \infty$ and the equations describe the same line), a point (if $\vert S \vert = 1$, unique solution), or empty ($S = \varnothing$ , when the lines are parallel).

Similarly, for three variables, each linear equation determines a plane in three-dimensional space. When we intersect these planes, i.e., satisfy all linear equations at the same time, we can obtain a solution set that is a plane, a line, a point or empty (when the planes have no common intersection).

3. Matrices.

For a systematic approach to solving systems of linear equations, we will introduce a useful compact notation. We collect the coefficients $a_{ij}$ into vectors and collect the vectors into matrices. In other words, we write the system from above in the following form:

In this context, we say that $A$ is the coefficient matrix, $X$ is the unknown matrix and $Y$ is the constant matrix.

Matrices play a central role in linear algebra. They can be used to compactly represent systems of linear equations, but they also represent linear functions (linear mappings) as we will see later. Before we discuss some of these interesting topics, let us first define what a matrix is and what kind of operations we can do with matrices.

3.1. Definition and Operations with Matrices.

3.1.1. Definition of a Matrix.

With $m,n \in \mathbb{N}$ a real-valued $(m, n)$ matrix $A$ is an $m·n$-tuple of elements $a_{ij} , i = 1, . . . , m, j = 1, . . . , n$, which is ordered according to a rectangular scheme consisting of $m$ rows and $n$ columns:

By convention $(1, n)$-matrices are called rows and $(m, 1)$-matrices are called columns. These special matrices are also called row/column vectors.

We say that $M_{m \times n}(F)$ is the set of all $(m,n)$-matrices and we write $A \in M_{m \times n} (F)$ or $(a_{ij}){i \in [m], j\in [n]} \in M{m \times n} (F)$

Let's observe that, by stacking its columns, a matrix $A \in M_{m \times n}(F)$ can be represented as a long vector $a \in F^{mn}$:

3.1.2. Addition and multiplication between matrices.

Let's now define operations between matrices.