What is a Vector Space in Machine Learning? (With Math and Intuition)
Source: Dev.to
Machine‑Learning and Vector Spaces
In machine learning, data is often represented in vector spaces so that mathematical operations such as addition (combination) and scalar multiplication (scaling) are possible.
1. What Is a Field?
A field is a set equipped with two operations, addition ((+)) and multiplication ((\cdot)), that satisfy the following:
| Property | Description |
|---|---|
| Closure | Performing the operation on elements of the set yields an element that is still in the set. |
| Axioms | Commutativity, associativity, distributivity, existence of additive and multiplicative identities, and existence of additive inverses (for all elements) and multiplicative inverses (for all non‑zero elements). |
| Division | Division is allowed for any non‑zero element (i.e., every non‑zero element has a multiplicative inverse). |
Intuition: A field is like a kitchen that contains all the basic tools you need to “cook” with numbers—except you can’t divide by zero.
Common examples of fields:
- Real numbers (\mathbb{R})
- Rational numbers (\mathbb{Q})
- Complex numbers (\mathbb{C})
The integers (\mathbb{Z}) are not a field because division does not stay within the set (e.g., (2 \div 3) is not an integer). (\mathbb{Z}) forms a ring, which guarantees addition, subtraction, and multiplication, but not division.
Formally, a field (F) satisfies
[ (F,+) \text{ is an abelian group},\qquad (F\setminus{0},\cdot) \text{ is an abelian group}, ] [ \forall a,b,c\in F,; a\cdot(b+c)=a\cdot b + a\cdot c. ]
2. What Is a Space?
A space is a set together with a specific structure—the collection of rules that dictate which operations are allowed on the elements and what properties those operations must satisfy. Different structures give rise to different kinds of spaces.
3. Vector Spaces
A vector space is a structure built over a field (usually (\mathbb{R}) or (\mathbb{C})). It consists of a set (V) together with two operations:
- Vector addition (+: V \times V \to V)
- Scalar multiplication (\cdot : F \times V \to V)
Both operations must be closed in (V) and satisfy the usual vector‑space axioms (associativity, commutativity of addition, distributivity, existence of a zero vector, etc.).
Formally, for all (u,v,w \in V) and (\alpha,\beta \in F):
[ \begin{aligned} &u+v \in V,\qquad \alpha v \in V,\ &u+0 = u,\qquad \alpha(u+v)=\alpha u+\alpha v,\ &(\alpha+\beta)v = \alpha v + \beta v,\qquad (\alpha\beta)v = \alpha(\beta v),\ &1_F v = v. \end{aligned} ]
Analogy: Think of a vector space as a spice rack in a kitchen. The rack (the space) holds the spices (vectors), and the rules (the structure) tell you how you may mix spices (vector addition) and scale recipes up or down (scalar multiplication).
What Is a Vector?
A vector is simply an element of a vector space. In many familiar cases (e.g., (\mathbb{R}^n)), a vector can be represented as an ordered list of field elements:
[ v = (v_1, v_2, \dots, v_n) \in \mathbb{R}^n. ]
Interpretations of vectors:
- Physics: direction and magnitude.
- Mathematics: coordinates.
- Machine learning: data points or feature representations.
Analogy: A vector is a particular recipe made from the ingredients (vectors) on the spice rack, obeying the kitchen’s mixing rules.
4. When Something Is Not a Vector Space
Not every collection of objects can be turned into a vector space. Below are common examples that fail the vector‑space axioms (especially closure under addition and scalar multiplication).
| Example | Why It Fails |
|---|---|
| Categorical values (e.g., “red”, “blue”, “green”) | Adding or scaling categories has no meaningful interpretation; they must be encoded (e.g., one‑hot) before use. |
| Probability distributions ([0.2, 0.5, 0.3]) | Adding two probability vectors may not sum to 1, and scaling by a negative scalar yields negative “probabilities”. |
| Sets (e.g., ({ \text{black}, \text{white}, \text{blue} })) | Set operations like union or intersection are not the same as vector addition or scalar multiplication. |
5. What We Have Covered So Far
- Fields provide the arithmetic foundation (addition, subtraction, multiplication, division).
- Spaces give a set plus a structure that tells us which operations are allowed.
- Vector spaces are a specific kind of space where we can add vectors and scale them by field elements, and we have a distinguished zero vector.
- Vectors are the individual elements (recipes) living inside a vector space.
At this point, we have only introduced addition and scalar multiplication. Concepts such as vector length, distance, or angle require additional structure (e.g., norms, inner products) and will be discussed later.
6. Summary (Kitchen Metaphor)
| Concept | Kitchen Analogy |
|---|---|
| Field | The kitchen’s basic tools (knives, spoons, etc.) that let you add, subtract, multiply, and divide (except by zero). |
| Space | The organization and rules of the kitchen (what you’re allowed to do with the tools). |
| Vector Space | A specific kitchen setup where you can mix ingredients (vector addition) and scale recipes (scalar multiplication) while staying inside the kitchen. |
| Vector | A concrete recipe made using the ingredients and rules of that kitchen. |
| Machine‑Learning Data | Raw ingredients that we must process (encode, normalize, etc.) so they can live in a well‑defined kitchen—a vector space—where learning algorithms make sense. |
The next step is to enrich vector spaces with additional structure (norms, inner products) so we can talk about lengths, distances, and angles—crucial concepts for many machine‑learning algorithms.
Will be introduced in a future post.
Thanks for reading. Vector spaces are foundational in machine learning, and intuition goes a long way in understanding them.