A concise, beginner-friendly introduction to the core ideas of linear algebra.
See the codeA concise, beginner-friendly introduction to the core ideas of Linear Algebra.
A clear, hands-on guide to the geometry and structure behind vectors and matrices.
.tex source
For old version (v1), here is the link
.tex sourceWe use Quarto to generate all outputs.
Preview locally:
quarto preview
Render outputs:
# All configured formats
quarto render
# Individual formats
quarto render --to html # site into docs/
quarto render --to pdf # docs/book.pdf
quarto render --to epub # docs/book.epub
quarto render --to latex # docs/book-latex/book.tex
Arrows in the air,
directions whisper softly—
the plane comes alive.
When we begin learning linear algebra, everything starts with the simplest building blocks: scalars and vectors. A scalar is just a single number, like 3, –7, or π. It carries only magnitude and no direction. Scalars are what we use for counting, measuring length, or scaling other objects up and down. A vector, by contrast, is an ordered collection of numbers. You can picture it as an arrow pointing somewhere in space, or simply as a list like (2, 5) in 2D or (1, –3, 4) in 3D. Where scalars measure "how much," vectors measure both "how much" and "which way."
To talk about vectors, we need a coordinate system. Imagine laying down two perpendicular axes on a sheet of paper: the x-axis (left to right) and the y-axis (up and down). Every point on the sheet can be described with two numbers: how far along the x-axis, and how far along the y-axis. This pair of numbers is a vector in 2D. Add a z-axis pointing up from the page, and you have 3D space. Each coordinate system gives us a way to describe vectors numerically, even though the underlying "space" is the same.
Understanding the difference between scalars and vectors is the foundation for everything else in linear algebra. Every concept-matrices, linear transformations, eigenvalues-eventually reduces to how we manipulate vectors and scale them with scalars. Without this distinction, the rest of the subject would have no anchor.
Nearly every field of science and engineering depends on this idea. Physics uses vectors for velocity, acceleration, and force. Computer graphics uses them to represent points, colors, and transformations. Data science treats entire datasets as high-dimensional vectors. By mastering scalars and vectors early, you unlock the language in which modern science and technology are written.
These simple experiments already give you intuition for the operations you'll perform again and again in linear algebra.
Linear algebra gives us powerful ways to describe and manipulate vectors, but before we can do anything with them, we need a precise notation system. Notation is not just cosmetic-it tells us how to read, write, and think about vectors clearly and unambiguously. In this section, we'll explore how vectors are written, how their components are represented, and how we can interpret them visually as arrows.
Vectors are usually denoted by lowercase letters in bold (like $\mathbf{v}, \mathbf{w}, \mathbf{x}$)
or with an arrow overhead (like $\vec{v}$).
For instance, the vector $\mathbf{v} = (2, 5)$ is the same as $\vec{v} = (2, 5)$.
The style depends on context: mathematicians often use bold, physicists often use arrows.
In handwritten notes, people sometimes underline vectors (e.g., $\underline{v}$) to avoid confusion with scalars.
The important thing is to distinguish vectors from scalars at a glance.
A vector in two dimensions has two components, written as $(x, y)$.
In three dimensions, it has three components: $(x, y, z)$.
More generally, an $n$-dimensional vector has $n$ components: $(v_1, v_2, \ldots, v_n)$.
Each component tells us how far the vector extends along one axis of the coordinate system.
For example:
We often refer to the $i$-th component of a vector $\mathbf{v}$ as $v_i$.
So, for $\mathbf{v} = (3, 4, 5)$, we have $v_1 = 3$, $v_2 = 4$, $v_3 = 5$.
Vectors can be written in two common ways:
As a row vector: $(v_1, v_2, v_3)$
As a column vector:
$$ \begin{bmatrix} v_1 \ v_2 \ v_3 \end{bmatrix} $$
Both represent the same abstract object.
Row vectors are convenient for quick writing, while column vectors are essential when we start multiplying by matrices, because the dimensions must align.
The most intuitive way to picture a vector is as an arrow:
For example, the vector (2, 3) in 2D is drawn as an arrow from (0, 0) to (2, 3). The arrow has both direction (where it points) and magnitude (its length). This geometric picture makes abstract algebraic manipulations much easier to grasp.
There are two common interpretations of vectors:
In linear algebra, we often treat vectors as free vectors, because their meaning does not depend on where they are drawn.
Suppose u = (–3, 2).
Clear notation is the backbone of linear algebra. Without it, equations quickly become unreadable, and intuition about direction and size is lost. The way we write vectors determines how easily we can connect the algebra (numbers and symbols) to the geometry (arrows and spaces). This dual perspective-symbolic and visual-is what makes linear algebra powerful and practical.
By practicing both the notation and the arrow picture, you'll develop fluency in switching between abstract symbols and concrete visualizations. This skill will make every later concept in linear algebra far more intuitive.
Once we know how to describe vectors with components and arrows, the next step is to learn how to combine them. Two fundamental operations form the backbone of linear algebra: adding vectors together and scaling vectors with numbers (scalars). These two moves, though simple, generate everything else we'll build later. With them, we can describe motion, forces, data transformations, and more.
Suppose we have two vectors in 2D:
$\mathbf{u} = (u_1, u_2), \quad \mathbf{v} = (v_1, v_2)$.
Their sum is defined as:
$$
\mathbf{u} + \mathbf{v} = (u_1 + v_1, ; u_2 + v_2).
$$
In words, you add corresponding components.
This works in higher dimensions too:
$$ (u_1, u_2, \ldots, u_n) + (v_1, v_2, \ldots, v_n) = (u_1 + v_1, ; u_2 + v_2, ; \ldots, ; u_n + v_n). $$
Example: $$ (2, 3) + (-1, 4) = (2 - 1, ; 3 + 4) = (1, 7). $$
The geometric picture is even more illuminating. If you draw vector u as an arrow, then place the tail of v at the head of u, the arrow from the start of u to the head of v is u + v. This is called the tip-to-tail rule. The parallelogram rule is another visualization: place u and v tail-to-tail, form a parallelogram, and the diagonal is their sum.
Example: u = (3, 1), v = (2, 2). Draw both from the origin. Their sum (5, 3) is exactly the diagonal of the parallelogram they span.
Scalars stretch or shrink vectors.
If $\mathbf{u} = (u_1, u_2, \ldots, u_n)$ and $c$ is a scalar, then:
$$ c \cdot \mathbf{u} = (c \cdot u_1, ; c \cdot u_2, ; \ldots, ; c \cdot u_n). $$
Example:
$$ 2 \cdot (3, 4) = (6, 8). $$
$$ (-1) \cdot (3, 4) = (-3, -4). $$
Multiplying by a positive scalar stretches or compresses the arrow while keeping the direction the same. Multiplying by a negative scalar flips the arrow to point the opposite way.
Imagine the vector (1, 2). Draw it on graph paper: it goes right 1, up 2. Now double it: (2, 4). The arrow points in the same direction but is twice as long. Halve it: (0.5, 1). It's the same direction but shorter. Negate it: (–1, –2). Now the arrow points backward.
This geometric picture explains why we call these numbers "scalars": they scale the vector.
Vector addition and scalar multiplication are not just separate tricks-they combine to form the heart of linear algebra: linear combinations.
A linear combination of vectors $u$ and $v$ is any vector of the form
$a \cdot u + b \cdot v$, where $a$ and $b$ are scalars.
Example:
If $u = (1, 0)$ and $v = (0, 1)$, then
$3 \cdot u + 2 \cdot v = (3, 2)$.
This shows how any point on the grid can be reached by scaling and adding these two basic vectors.
That’s the essence of constructing spaces.
Vector addition and scalar multiplication obey rules that mirror arithmetic with numbers:
These rules are not trivial bookkeeping - they guarantee that linear algebra behaves predictably,
which is why it works as the language of science.
With only these two operations-addition and scaling-you can already describe lines, planes, and entire spaces. Any system that grows by combining influences, like physics, economics, or machine learning, is built on these simple rules. Later, when we define matrix multiplication, dot products, and eigenvalues, they all reduce to repeated patterns of adding and scaling vectors.
This simple practice shows you how combining two basic vectors through addition and scaling generates a whole structured space, the first glimpse of linear algebra's real power.
After learning to add vectors and scale them, the natural next question is: what can we build from these two operations? The answer is the concept of linear combinations, which leads directly to one of the most fundamental ideas in linear algebra: the span of a set of vectors. These ideas tell us not only what individual vectors can do, but how groups of vectors can shape entire spaces.
A linear combination is any vector formed by multiplying vectors with scalars and then adding the results together.
Formally, given vectors $v_1, v_2, \ldots, v_k$ and scalars $a_1, a_2, \ldots, a_k$, a linear combination looks like:
$$ a_1 \cdot v_1 + a_2 \cdot v_2 + \cdots + a_k \cdot v_k. $$
This is nothing more than repeated addition and scaling, but the idea is powerful because it describes how vectors combine to generate new ones.
Example:
Let $u = (1, 0)$ and $v = (0, 1)$. Then any linear combination $a \cdot u + b \cdot v = (a, b)$.
This shows that every point in the 2D plane can be expressed as a linear combination of these two simple vectors.
Linear combinations are about mixing directions and magnitudes. Each vector acts like a "directional ingredient," and the scalars control how much of each ingredient you use.
This progression shows that the power of linear combinations depends not just on the vectors themselves but on how they relate to each other.
The span of a set of vectors is the collection of all possible linear combinations of them.
It answers the question: "What space do these vectors generate?"
Notation:
$$ \text{Span}{v_1, v_2, \ldots, v_k} = {a_1 v_1 + a_2 v_2 + \cdots + a_k v_k ;|; a_i \in \mathbb{R}}. $$
Examples:
So the span depends heavily on whether the vectors add new directions or just duplicate what’s already there.
If vectors point in the same or opposite directions (one is a scalar multiple of another), then their span is just a line. They don't add any new coverage of space. But if they point in different directions, they open up new dimensions. This leads to the critical idea of linear independence, which we'll explore later: vectors are independent if none of them is a linear combination of the others.
In 2D:
In 3D:
In higher dimensions: The same pattern continues. A set of k independent vectors spans a k-dimensional subspace inside the larger space.
Linear combinations and span are the foundation for almost everything else in linear algebra:
In short, the span tells us the "reach" of a set of vectors, and linear combinations are the mechanism to explore that reach.
By experimenting with simple examples, you'll see clearly how the idea of span captures the richness or limitations of combining vectors.
So far, vectors have been arrows with direction and components. To compare them more meaningfully, we need ways to talk about how long they are and how far apart they are. These notions are formalized through the norm of a vector (its length) and the distance between vectors. These concepts tie together the algebra of components and the geometry of space.
The norm of a vector measures its magnitude, or how long the arrow is.
For a vector $v = (v_1, v_2, \ldots, v_n)$ in $n$-dimensional space, its norm is defined as:
$$ |v| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}. $$
This formula comes directly from the Pythagorean theorem: the length of the hypotenuse equals the square root of the sum of squares of the legs.
In 2D, this is the familiar distance formula between the origin and a point.
Examples:
For $v = (3, 4)$:
$$
|v| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = 5.
$$
For $w = (1, -2, 2)$:
$$
|w| = \sqrt{1^2 + (-2)^2 + 2^2} = \sqrt{1 + 4 + 4} = \sqrt{9} = 3.
$$
A unit vector is a vector whose length is exactly 1.
These are important because they capture direction without scaling.
To create a unit vector from any nonzero vector, divide by its norm:
$$ u = \frac{v}{|v|}. $$
Example:
For $v = (3, 4)$, the unit vector is
$$ u = \left(\tfrac{3}{5}, \tfrac{4}{5}\right). $$
This points in the same direction as $(3, 4)$ but has length 1.
Unit vectors are like pure directions.
They're especially useful for projections, defining coordinate systems, and normalizing data.
The distance between two vectors $u$ and $v$ is defined as the length of their difference:
$$ \text{dist}(u, v) = |u - v|. $$
Example:
Let $u = (2, 1)$ and $v = (5, 5)$. Then
$$ u - v = (-3, -4). $$
Its norm is
$$ \sqrt{(-3)^2 + (-4)^2} = \sqrt{9 + 16} = 5. $$
So the distance is 5. This matches our intuition: the straight-line distance between points $(2, 1)$ and $(5, 5)$.
Both are computed with the same formula-the square root of sums of squares-but applied in slightly different contexts.
The formula above defines the Euclidean norm (or $\ell_2$ norm), the most common one.
But in linear algebra, other norms are also useful:
$\ell_1$ norm:
$$
|v|_1 = |v_1| + |v_2| + \cdots + |v_n|
$$
(sum of absolute values).
$\ell_\infty$ norm:
$$
|v|_\infty = \max(|v_1|, |v_2|, \ldots, |v_n|)
$$
(largest component).
These norms change the geometry of "length" and "distance." For example, in the ℓ₁ norm, the unit circle is shaped like a diamond; in the ℓ∞ norm, it looks like a square.
Norms and distances satisfy critical properties that make them consistent measures:
These properties are why norms and distances are robust tools across mathematics.
Understanding length and distance is the first step toward geometry in higher dimensions. These notions:
By working through these examples, you'll see how norms and distances make abstract vectors feel as real as points and arrows you can measure in everyday life.
The dot product is one of the most fundamental operations in linear algebra. It looks like a simple formula, but it unlocks the ability to measure angles, detect orthogonality, project one vector onto another, and compute energy or work in physics. Understanding it requires seeing both the algebraic view (a formula on components) and the geometric view (a way to compare directions).
For two vectors of the same dimension, $u = (u_1, u_2, \ldots, u_n)$ and $v = (v_1, v_2, \ldots, v_n)$, the dot product is defined as:
$$ u \cdot v = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n. $$
This is simply multiplying corresponding components and summing the results.
Examples:
Notice that the dot product is always a scalar, not a vector.
The dot product can also be defined in terms of vector length and angle:
$$ u \cdot v = |u| , |v| \cos(\theta), $$
where $\theta$ is the angle between $u$ and $v$ ($0^\circ \leq \theta \leq 180^\circ$).
This formula tells us:
Thus, the sign of the dot product encodes directional alignment.
At first glance, the algebraic sum of products and the geometric length–angle formula seem unrelated. But they are equivalent. To see why, consider the law of cosines applied to a triangle formed by u, v, and u – v. Expanding both sides leads directly to the equivalence between the two formulas. This dual interpretation is what makes the dot product so powerful: it is both a computation rule and a geometric measurement.
Two vectors are orthogonal (perpendicular) if and only if their dot product is zero:
$$ u \cdot v = 0 ;;\Longleftrightarrow;; \theta = 90^\circ. $$
This gives us an algebraic way to check for perpendicularity without drawing diagrams.
Example:
$(2, 1) \cdot (-1, 2) = (2 \times -1) + (1 \times 2) = -2 + 2 = 0$,
so the vectors are orthogonal.
The dot product also provides a way to project one vector onto another.
The scalar projection of $u$ onto $v$ is:
$$ \text{proj}_{\text{scalar}}(u \text{ onto } v) = \frac{u \cdot v}{|v|}. $$
The vector projection is then:
$$ \text{proj}_{\text{vector}}(u \text{ onto } v) = \frac{u \cdot v}{|v|^2} , v. $$
This allows us to decompose vectors into "parallel" and "perpendicular" components, which is central in geometry, physics, and data analysis.
Compute $u = (3, 4)$ and $v = (4, 3)$.
Compute $u = (1, 2, -1)$ and $v = (2, -1, 1)$.
In physics, the dot product computes work:
$$ \text{Work} = \text{Force} \cdot \text{Displacement} = |\text{Force}| , |\text{Displacement}| \cos(\theta). $$
Only the component of the force in the direction of motion contributes. If you push straight down on a box while trying to move it horizontally, the dot product is zero: no work is done in the direction of motion.
These guarantee that the dot product behaves consistently and meshes with the structure of vector spaces.
The dot product is the first bridge between algebra and geometry. It:
Without the dot product, linear algebra would lack a way to connect numbers with geometry and meaning.
These exercises reveal the dual power of the dot product: as a formula to compute and as a geometric tool to interpret.
Having defined the dot product, we are now ready to measure angles between vectors. In everyday life, angles tell us how two lines or directions relate-whether they point the same way, are perpendicular, or are opposed. In linear algebra, the dot product and cosine function give us a precise, generalizable way to define angles in any dimension, not just in 2D or 3D. This section explores how we compute, interpret, and apply vector angles.
For two nonzero vectors $u$ and $v$, the angle $\theta$ between them is defined by:
$$ \cos(\theta) = \frac{u \cdot v}{|u| , |v|}. $$
This formula comes directly from the geometric definition of the dot product.
Rearranging gives:
$$ \theta = \arccos!\left(\frac{u \cdot v}{|u| , |v|}\right). $$
Key points:
The cosine tells us about the directional relationship:
Thus, the cosine compresses geometric alignment into a single number.
$u = (1, 0), ; v = (0, 1)$
$u = (2, 3), ; v = (4, 6)$
$u = (1, 1), ; v = (-1, 1)$
The beauty of the formula is that it works in any dimension.
Even in $\mathbb{R}^{100}$ or higher, we can define the angle between two vectors using only their dot product and norms.
While we cannot visualize the geometry directly in high dimensions, the cosine formula still captures how aligned two directions are:
$$ \cos(\theta) = \frac{u \cdot v}{|u| , |v|}. $$
This is critical in machine learning, where data often lives in very high-dimensional spaces.
The cosine of the angle between two vectors is often called cosine similarity. It is widely used in data analysis and machine learning to measure how similar two data vectors are, independent of their magnitude.
This demonstrates how a geometric concept extends far beyond pure math.
The angle formula reinforces the special role of orthogonality.
If $\cos(\theta) = 0$, then $u \cdot v = 0$.
This means the dot product not only computes length but also serves as a direct test for perpendicularity.
This algebraic shortcut is far easier than manually checking geometric right angles.
Angles are closely tied to projections.
The length of the projection of $u$ onto $v$ is $|u|\cos(\theta)$.
If the angle is small, the projection is large — most of $u$ lies in the direction of $v$.
If the angle is close to $90^\circ$, the projection shrinks toward zero.
Thus, the cosine acts as a scaling factor between directions.
Teamwork analogy: If two people push a heavy object, the effectiveness of their combined effort depends on the angle.
If they push in nearly the same direction (small $\theta$), they cooperate efficiently.
If they push at right angles ($\theta = 90^\circ$), they waste effort.
If they push opposite each other ($\theta \approx 180^\circ$), they cancel out.
Conversation analogy: If two people's opinions align ($\cos \theta \approx 1$), they agree strongly.
If they are orthogonal ($\cos \theta = 0$), they are unrelated.
If they oppose each other ($\cos \theta \approx -1$), they fundamentally disagree.
Navigation analogy: If two roads meet at a small angle, their directions are similar.
If they cross perpendicularly, their directions are independent.
Angles between vectors provide:
Without the ability to measure angles, we could not connect algebraic manipulations with geometric intuition or practical applications.
By experimenting with these problems, you will see how angles provide the missing link between algebraic formulas and geometric meaning in linear algebra.
In earlier sections, we saw how the dot product measures alignment and how the cosine formula gives us angles between vectors. The next natural step is to use these tools to project one vector onto another. Projection is a way to "shadow" one vector onto the direction of another, splitting vectors into meaningful parts: one along a given direction and one perpendicular to it. This is the essence of decomposition, and it is everywhere in linear algebra, geometry, physics, and data science.
The scalar projection of a vector $u$ onto a vector $v$ measures how much of $u$ lies in the direction of $v$. It is given by:
$$ \text{proj}_{\text{scalar}}(u \text{ onto } v) = \frac{u \cdot v}{|v|}. $$
Example:
$u = (3, 4)$, $v = (1, 0)$.
Dot product: $(3 \times 1 + 4 \times 0) = 3$.
$|v| = 1$.
So the scalar projection is $3$. This tells us $u$ has a "shadow" of length $3$ on the $x$-axis.
The vector projection gives the actual arrow in the direction of $v$ that corresponds to this scalar amount:
$$ \text{proj}_{\text{vector}}(u \text{ onto } v) = \frac{u \cdot v}{|v|^2} , v. $$
This formula normalizes $v$ into a unit vector, then scales it by the scalar projection.
The result is a new vector lying along $v$, capturing exactly the "parallel" part of $u$.
Example:
$u = (3, 4)$, $v = (1, 2)$
So the part of $(3, 4)$ in the direction of $(1, 2)$ is $(2.2, 4.4)$.
Once we have the projection, we can find the perpendicular component (often called the rejection) simply by subtracting:
$$ u_{\perp} = u - \text{proj}_{\text{vector}}(u \text{ onto } v). $$
This gives the part of $u$ that is entirely orthogonal to $v$.
Example continued:
$u_{\perp} = (3, 4) - (2.2, 4.4) = (0.8, -0.4)$
Check:
$(0.8, -0.4) \cdot (1, 2) = 0.8 \times 1 + (-0.4) \times 2 = 0.8 - 0.8 = 0$.
Indeed, orthogonal.
Projection is like dropping a perpendicular from one vector onto another. Imagine shining a light perpendicular to v: the shadow of u on the line spanned by v is the projection. This visualization explains why projections split vectors naturally into two pieces:
Together, these two parts reconstruct the original vector exactly.
Every vector $u$ can be decomposed relative to another vector $v$ into two parts:
$$ u = \text{proj}{\text{vector}}(u \text{ onto } v) + \big(u - \text{proj}{\text{vector}}(u \text{ onto } v)\big). $$
This decomposition is unique and geometrically meaningful.
It generalizes to subspaces: we can project onto entire planes or higher-dimensional spans, splitting a vector into a "within-subspace" part and a "perpendicular-to-subspace" part.
Physics (Work and Forces): Work is the projection of force onto displacement. Only the part of the force in the direction of motion contributes. Example: Pushing on a sled partly sideways wastes effort-the sideways component projects to zero.
Geometry and Engineering: Projections are used in CAD (computer-aided design) to flatten 3D objects onto 2D surfaces, like blueprints or shadows.
Computer Graphics: Rendering 3D scenes onto a 2D screen is fundamentally a projection process.
Data Science: Projecting high-dimensional data onto a lower-dimensional subspace (like the first two principal components in PCA) makes patterns visible while preserving as much information as possible.
Signal Processing: Decomposition into projections onto sine and cosine waves forms the basis of Fourier analysis, which powers audio, image, and video compression.
Projection is not just a geometric trick; it is the core of many advanced topics:
Without projection, we could not cleanly separate influence along directions or reduce complexity in structured ways.
Through these exercises, you will see that projection is more than an operation-it is a lens through which we decompose, interpret, and simplify vectors and spaces.
Linear algebra is not only about operations with vectors-it also involves understanding the fundamental relationships between them. Two of the most important results in this regard are the Cauchy–Schwarz inequality and the triangle inequality. These are cornerstones of vector spaces because they establish precise boundaries for lengths, angles, and inner products. Without them, the geometry of linear algebra would fall apart.
For any two vectors $u$ and $v$ in $\mathbb{R}^n$, the Cauchy–Schwarz inequality states:
$$ |u \cdot v| \leq |u| , |v|. $$
This means that the absolute value of the dot product of two vectors is always less than or equal to the product of their lengths.
Equality holds if and only if u and v are linearly dependent (i.e., one is a scalar multiple of the other).
Recall the geometric formula for the dot product:
$$ u \cdot v = |u| , |v| \cos(\theta). $$
Since $-1 \leq \cos(\theta) \leq 1$, the magnitude of the dot product cannot exceed $|u| , |v|$.
This is exactly the inequality.
Let $u = (3, 4)$ and $v = (-4, 3)$.
Equality does not hold since they are not multiples - they are perpendicular.
The inequality tells us that two vectors can never "overlap" more strongly than the product of their magnitudes. If they align perfectly, the overlap is maximum (equality). If they're perpendicular, the overlap is zero.
Think of it as: "the shadow of one vector on another can never be longer than the vector itself."
For any vectors $u$ and $v$, the triangle inequality states:
$$ |u + v| \leq |u| + |v|. $$
This mirrors the geometric fact that in a triangle, any side is at most as long as the sum of the other two sides.
Let $u = (1, 2)$ and $v = (3, 4)$.
Indeed, $7.21 \leq 7.24$, very close in this case.
The triangle inequality becomes equality when the vectors point in exactly the same direction (or are scalar multiples with nonnegative coefficients). For example, (1, 1) and (2, 2) produce equality because adding them gives a vector whose length equals the sum of their lengths.
Without these inequalities, we could not trust that the geometry of vector spaces behaves consistently.
Working through these problems will show you why these inequalities are not abstract curiosities but the structural glue of linear algebra's geometry.
Up to now, we've discussed vectors, their lengths, angles, and how to project one onto another. A natural culmination of these ideas is the concept of orthonormal sets. These are collections of vectors that are not only orthogonal (mutually perpendicular) but also normalized (each of length 1). Orthonormal sets form the cleanest, most efficient coordinate systems in linear algebra. They are the mathematical equivalent of having rulers at right angles, perfectly calibrated to unit length.
Let's break the term "orthonormal" into two parts:
Orthogonal: Two vectors $u$ and $v$ are orthogonal if $u \cdot v = 0$.
In $\mathbb{R}^2$, this means the vectors meet at a right angle.
In $\mathbb{R}^3$, it means they form perpendicular directions.
Normalized: A vector $v$ is normalized if its length is $1$, i.e., $|v| = 1$.
Such vectors are called unit vectors.
When we combine both conditions, we get orthonormal vectors: vectors that are both perpendicular to each other and have unit length.
In two dimensions, an orthonormal set typically consists of two vectors.
A classic example is:
$e_1 = (1, 0), \quad e_2 = (0, 1)$
Thus, ${e_1, e_2}$ is an orthonormal set.
In fact, this is the standard basis for $\mathbb{R}^2$.
Any vector $(x, y)$ can be written as $x e_1 + y e_2$.
This is the simplest coordinate system.
In three dimensions, an orthonormal set usually has three vectors.
The standard basis is:
$e_1 = (1, 0, 0), \quad e_2 = (0, 1, 0), \quad e_3 = (0, 0, 1)$
Geometrically, they correspond to the $x$-, $y$-, and $z$-axes in 3D space.
Any vector $(x, y, z)$ can be written as a linear combination $x e_1 + y e_2 + z e_3$.
The standard basis is not the only orthonormal set. For example:
$u = \left(\tfrac{1}{\sqrt{2}}, \tfrac{1}{\sqrt{2}}\right), \quad v = \left(-\tfrac{1}{\sqrt{2}}, \tfrac{1}{\sqrt{2}}\right)$
So ${u, v}$ is also orthonormal in $\mathbb{R}^2$.
These vectors are rotated $45^\circ$ relative to the standard axes.
Similarly, in $\mathbb{R}^3$, you can construct rotated orthonormal sets (such as unit vectors along diagonals), as long as the conditions of perpendicularity and unit length hold.
Simplified coordinates: If ${v_1, \ldots, v_k}$ is an orthonormal set, then for any vector $u$ in their span, the coefficients are easy to compute:
$$
c_i = u \cdot v_i
$$
This is much simpler than solving systems of equations.
Pythagorean theorem generalized: If vectors are orthonormal, the squared length of their sum is the sum of the squares of their coefficients.
For example, if $u = a v_1 + b v_2$, then
$$
|u|^2 = a^2 + b^2
$$
Projection is easy: Projecting onto an orthonormal set is straightforward — just take dot products.
Matrices become nice: When vectors form the columns of a matrix, orthonormality makes that matrix an orthogonal matrix, which has special properties: its transpose equals its inverse, and it preserves lengths and angles.
Even though $\mathbb{R}^2$ and $\mathbb{R}^3$ are relatively simple, the same ideas extend naturally to higher dimensions, where visualization is impossible but the algebra is identical.
Orthonormality is the gold standard for building bases in linear algebra:
Whenever possible, mathematicians and engineers prefer orthonormal bases over arbitrary ones.
Through these exercises, you will see how orthonormal sets make every aspect of linear algebra-from projections to decompositions-simpler, cleaner, and more powerful.
Lengths, angles revealed,
projections trace hidden lines,
clarity takes shape.
Rows and columns meet,
woven grids of silent rules,
machines of order.
The next stage in our journey is to move from vectors to matrices. A matrix may look like just a rectangular array of numbers, but in linear algebra it plays two distinct and equally important roles:
Both views are valid, and learning to switch between them is crucial to building intuition.
At the most basic level, a matrix is a grid of numbers arranged into rows and columns.
A $2 \times 2$ matrix has 2 rows and 2 columns:
$$ A = \begin{bmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{bmatrix} $$
A $3 \times 2$ matrix has 3 rows and 2 columns:
$$ B = \begin{bmatrix} b_{11} & b_{12} \ b_{21} & b_{22} \ b_{31} & b_{32} \end{bmatrix} $$
Each entry $a_{ij}$ or $b_{ij}$ tells us the number in the i-th row and j-th column. The rows of a matrix can represent constraints, equations, or observations; the columns can represent features, variables, or directions.
In this sense, matrices are data containers, organizing information efficiently. That's why matrices show up in spreadsheets, statistics, computer graphics, and scientific computing.
The deeper view of a matrix is as a function from vectors to vectors. If x is a column vector, then multiplying A·x produces a new vector.
For example:
$$ A = \begin{bmatrix} 2 & 0 \ 1 & 3 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 4 \ 5 \end{bmatrix}. $$
Multiplying:
$$ A\mathbf{x} = \begin{bmatrix} 2×4 + 0×5 \ 1×4 + 3×5 \end{bmatrix} = \begin{bmatrix} 8 \ 19 \end{bmatrix}. $$
Here, the matrix is acting as a machine that takes input (4, 5) and outputs (8, 19). The "machine rules" are encoded in the rows of A.
Another way to see it: multiplying A·x is the same as taking a linear combination of A's columns.
If
$$ A = \begin{bmatrix} a_1 & a_2 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \end{bmatrix}, $$
then:
$$ A\mathbf{x} = x_1 a_1 + x_2 a_2. $$
So the vector x tells the machine "how much" of each column to mix together. This column view is critical-it connects matrices to span, dimension, and basis ideas we saw earlier.
This duality is not just conceptual-it's the key to understanding why linear algebra is so powerful. A dataset, once stored as a table, can be interpreted as a transformation. Likewise, a transformation, once understood, can be encoded as a table.
Every 2×2 or 3×3 matrix corresponds to some linear transformation in the plane or space. Examples:
These are not just tables of numbers-they are precise, reusable machines.
This section sets the stage for all matrix theory:
By mastering both perspectives, you'll see matrices not just as numbers but as dynamic objects that encode and execute transformations.
Matrices come in many shapes and sizes, and the way we label their entries matters. This section is about learning how to read and write matrices carefully, how to work with rows and columns, and how to use block structure to simplify problems. These seemingly simple ideas are what allow us to manipulate large systems with precision and efficiency.
The shape of a matrix is given by its number of rows and columns:
Examples:
A 2×3 matrix:
$$ \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix} $$
A 3×2 matrix:
$$ \begin{bmatrix} 7 & 8 \ 9 & 10 \ 11 & 12 \end{bmatrix} $$
Shape matters because it determines whether certain operations (like multiplication) are possible.
Each entry in a matrix has two indices: one for its row, one for its column.
For example, in
$$ A = \begin{bmatrix} 1 & 4 & 7 \ 2 & 5 & 8 \ 3 & 6 & 9 \end{bmatrix}, $$
we have:
Indexing is the grammar of matrix language. Without it, we can't specify positions or write formulas clearly.
Every row and every column of a matrix is itself a vector.
Example: From the matrix above,
This duality is powerful: rows often represent constraints or equations, while columns represent directions or features. Later, when we interpret matrix–vector products, we'll see that multiplying A·x means combining columns, while multiplying yᵀ·A means combining rows.
Sometimes we want just part of a matrix. A submatrix is formed by selecting certain rows and columns.
Example: From
$$ B = \begin{bmatrix} 2 & 4 & 6 \ 1 & 3 & 5 \ 7 & 8 & 9 \end{bmatrix}, $$
the submatrix of the first two rows and last two columns is:
$$ \begin{bmatrix} 4 & 6 \ 3 & 5 \end{bmatrix}. $$
Submatrices allow us to zoom in and isolate parts of a problem.
Large matrices can often be broken into blocks, which are smaller submatrices arranged inside. This is like dividing a spreadsheet into quadrants.
For example:
$$ C = \begin{bmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{bmatrix}, $$
where each $A_{ij}$ is itself a smaller matrix.
This structure is useful in:
Example: Splitting a 4×4 matrix into four 2×2 blocks helps us treat it as a "matrix of matrices."
Some shapes of matrices are so common they deserve names:
Recognizing these shapes saves time and clarifies reasoning.
Careful attention to matrix shapes, indexing, and block views ensures:
These basic tools prepare us for multiplication, transformations, and factorization.
Write a 3×4 matrix and label the entry in row 2, column 3.
Extract a 2×2 submatrix from the corners of a 4×4 matrix of your choice.
Break a 6×6 matrix into four 3×3 blocks. How would you represent it compactly?
Challenge: Given
$$ D = \begin{bmatrix} 1 & 2 & 3 & 4 \ 5 & 6 & 7 & 8 \ 9 & 10 & 11 & 12 \end{bmatrix}, $$
write it as a block matrix with a 2×2 block in the top-left, a 2×2 block in the top-right, and a 1×4 block in the bottom row.
By practicing with shapes, indexing, and blocks, you'll develop the ability to navigate matrices not just as raw grids of numbers but as structured objects ready for deeper algebraic and geometric insights.
Before exploring matrix–vector and matrix–matrix multiplication, it is essential to understand the simplest operations we can perform with matrices: addition and scalar multiplication. These operations extend the rules we learned for vectors, but now applied to entire grids of numbers. Although straightforward, they are the foundation for more complex algebraic manipulations and help establish the idea of matrices as elements of a vector space.
If two matrices $A$ and $B$ have the same shape (same number of rows and columns), we can add them by adding corresponding entries.
Formally: If
$$ A = [a_{ij}], \quad B = [b_{ij}], $$
then
$$ A + B = [a_{ij} + b_{ij}]. $$
Example:
\begin{bmatrix} 8 & 10 & 12 \ 14 & 16 & 18 \end{bmatrix}. $$
Key point: Addition is only defined if the matrices are the same shape. A 2×3 matrix cannot be added to a 3×2 matrix.
A scalar multiplies every entry of a matrix.
Formally: For scalar $c$ and matrix $A = [a_{ij}]$,
$$ cA = [c \cdot a_{ij}]. $$
Example:
\begin{bmatrix} 6 & -3 \ 0 & 12 \end{bmatrix}. $$
This mirrors vector scaling: stretching or shrinking the whole matrix by a constant factor.
These two operations satisfy familiar algebraic properties that make the set of all m×n matrices into a vector space:
These guarantee that working with matrices feels like working with numbers and vectors, only in a higher-level setting.
From the table view, addition and scalar multiplication are just simple bookkeeping: line up two tables of the same shape and add entry by entry; multiply the whole table by a constant.
Example: Imagine two spreadsheets of monthly expenses. Adding them gives combined totals. Multiplying by 12 converts a monthly table into a yearly estimate.
From the machine view, these operations adjust the behavior of linear transformations:
Example: Let $A$ rotate vectors slightly, and $B$ stretch vectors. The matrix $A + B$ represents a transformation that applies both influences together. Scaling by 2 doubles the effect of the transformation.
These act as neutral or scaling elements in matrix arithmetic.
This shows that even before multiplication, addition and scaling already have geometric meaning.
Though simple, these operations:
Without addition and scalar multiplication, we could not treat matrices systematically as algebraic objects.
$$ \begin{bmatrix} 2 & 0 \ 1 & 3 \end{bmatrix} \quad \text{and} \quad \begin{bmatrix} -2 & 5 \ 4 & -3 \end{bmatrix}. $$
$$ \begin{bmatrix} 1 & -1 & 2 \ 0 & 3 & 4 \end{bmatrix} $$
by –2.
Show that (A + B) + C = A + (B + C) with explicit 2×2 matrices.
Challenge: Construct two 3×3 matrices A and B such that A + B = 0. What does that tell you about B?
By practicing these fundamentals, you will see that even the most basic operations on matrices already build the algebraic backbone for deeper results like matrix multiplication, transformations, and factorization.
We now arrive at one of the most important operations in all of linear algebra: the matrix–vector product. This operation takes a matrix $A$ and a vector x, and produces a new vector. While the computation is straightforward, its interpretations are deep: it can be seen as combining rows, as combining columns, or as applying a linear transformation. This is the operation that connects matrices to the geometry of vector spaces.
Suppose $A$ is an $m \times n$ matrix, and x is a vector in $\mathbb{R}^n$. The product $A\mathbf{x}$ is a vector in $\mathbb{R}^m$, defined as:
$$ A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \ a_{21} & a_{22} & \cdots & a_{2n} \ \vdots & \vdots & \ddots & \vdots \ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \ \vdots \ x_n \end{bmatrix}. $$
Then:
$$ A\mathbf{x} = \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n \ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n \ \vdots \ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n \end{bmatrix}. $$
Each entry of the output is a dot product between one row of $A$ and the vector x.
From the row perspective, $A\mathbf{x}$ is computed row by row:
Example:
$$ A = \begin{bmatrix} 2 & 1 \ 3 & 4 \ -1 & 2 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 5 \ -1 \end{bmatrix}. $$
So:
$$ A\mathbf{x} = \begin{bmatrix} 9 \ 11 \ -7 \end{bmatrix}. $$
From the column perspective, $A\mathbf{x}$ is a linear combination of the columns of A.
If
$$ A = \begin{bmatrix} | & | & & | \ a_1 & a_2 & \cdots & a_n \ | & | & & | \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \ \vdots \ x_n \end{bmatrix}, $$
then:
$$ A\mathbf{x} = x_1 a_1 + x_2 a_2 + \cdots + x_n a_n. $$
That is: multiply each column of $A$ by the corresponding entry in x, then add them up.
This interpretation connects directly to the idea of span: the set of all vectors $A\mathbf{x}$ as x varies is exactly the span of the columns of $A$.
The machine view ties everything together: multiplying a vector by a matrix means applying the linear transformation represented by the matrix.
Thus, the algebraic definition encodes geometric and functional meaning.
$$ A = \begin{bmatrix} 2 & 0 \ 0 & 2 \end{bmatrix}. $$
Then $A\mathbf{x}$ doubles the length of any vector x.
$$ A = \begin{bmatrix} 1 & 0 \ 0 & -1 \end{bmatrix}. $$
This flips vectors across the x-axis.
$$ A = \begin{bmatrix} \cosθ & -\sinθ \ \sinθ & \cosθ \end{bmatrix}. $$
This rotates vectors counterclockwise by θ in the plane.
The matrix–vector product is the building block of everything in linear algebra:
$$ \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix} \begin{bmatrix} 2 \ 0 \ 1 \end{bmatrix}. $$
Express the result of the above product as a linear combination of the columns of the matrix.
Construct a 2×2 matrix that reflects vectors across the line $y = x$. Test it on (1, 0) and (0, 1).
Challenge: For a 3×3 matrix, show that the set of all possible $A\mathbf{x}$ (as x varies) is exactly the column space of $A$.
By mastering both the computational rules and the interpretations of the matrix–vector product, you will gain the most important insight in linear algebra: matrices are not just tables-they are engines that transform space.
Having understood how a matrix acts on a vector, the next natural step is to understand how one matrix can act on another. This leads us to the matrix–matrix product, a rule for combining two matrices into a single new matrix. Though the arithmetic looks complicated at first, the underlying idea is elegant: multiplying two matrices represents composing two linear transformations.
Suppose $A$ is an $m \times n$ matrix and $B$ is an $n \times p$ matrix. Their product $C = AB$ is an $m \times p$ matrix defined by:
$$ c_{ij} = \sum_{k=1}^n a_{ik} b_{kj}. $$
That is: each entry of $C$ is the dot product of the i-th row of $A$ with the j-th column of $B$.
$$ A = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix}, \quad B = \begin{bmatrix} 7 & 8 \ 9 & 10 \ 11 & 12 \end{bmatrix}. $$
Product: $C = AB$ will be 2×2.
So:
$$ C = \begin{bmatrix} 58 & 64 \ 139 & 154 \end{bmatrix}. $$
From the column perspective, $AB$ is computed by applying $A$ to each column of $B$.
If $B = [b_1 ; b_2 ; \cdots ; b_p]$, then:
$$ AB = [A b_1 ; A b_2 ; \cdots ; A b_p]. $$
That is: multiply $A$ by each column of $B$. This is often the simplest way to think of the product.
From the row perspective, each row of $AB$ is formed by combining rows of $B$ using coefficients from a row of $A$. This dual view is less common but equally useful, especially in proofs and algorithms.
The most important interpretation is the machine view: multiplying matrices corresponds to composing transformations.
Example:
$$ A = \begin{bmatrix} 2 & 0 \ 0 & 2 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & -1 \ 1 & 0 \end{bmatrix}. $$
Then $AB$ scales vectors by 2 after rotating them 90°.
These properties highlight that while multiplication is structured, it is not symmetric. The order encodes the order of operations in transformations.
Matrix multiplication is the core of linear algebra because:
Almost every application of linear algebra-solving equations, computing eigenvalues, training neural networks-relies on efficient matrix multiplication.
$$ \begin{bmatrix} 1 & 0 \ 2 & 3 \end{bmatrix} \begin{bmatrix} 4 & 5 \ 6 & 7 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 0 & 1 \ 0 & 0 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & 0 \ 1 & 0 \end{bmatrix}. $$
Construct two 2×2 matrices where $AB = BA$. Why does commutativity happen here?
Challenge: If $A$ is a projection and $B$ is a rotation, compute $AB$ and $BA$. Do they represent the same geometric operation?
Through these perspectives, the matrix–matrix product shifts from being a mechanical formula to being a language for combining linear steps-each product telling the story of "do this, then that."
With addition, scalar multiplication, and matrix multiplication in place, we now introduce three special operations and objects that form the backbone of matrix algebra: the identity matrix, the inverse of a matrix, and the transpose of a matrix. Each captures a fundamental principle-neutrality, reversibility, and symmetry-and together they provide the algebraic structure that makes linear algebra so powerful.
The identity matrix is the matrix equivalent of the number 1 in multiplication.
Example (3×3):
$$ I_3 = \begin{bmatrix} 1 & 0 & 0 \ 0 & 1 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
Property: For any $n \times n$ matrix $A$,
$$ AI_n = I_nA = A. $$
Machine view: $I$ does nothing-it maps every vector to itself.
The inverse is the matrix equivalent of the reciprocal of a number.
Definition: For a square matrix $A$, its inverse $A^{-1}$ is the matrix such that
$$ AA^{-1} = A^{-1}A = I. $$
Not all matrices have inverses. A matrix is invertible if and only if it is square and its determinant is nonzero.
Example:
$$ A = \begin{bmatrix} 2 & 1 \ 1 & 1 \end{bmatrix}, \quad A^{-1} = \begin{bmatrix} 1 & -1 \ -1 & 2 \end{bmatrix}. $$
Check:
\begin{bmatrix} 1 & 0 \ 0 & 1 \end{bmatrix} = I. $$
Some matrices cannot be inverted. These are called singular.
Example:
$$ B = \begin{bmatrix} 2 & 4 \ 1 & 2 \end{bmatrix}. $$
Here, the second column is a multiple of the first. The transformation squashes vectors into a line, losing information-so it cannot be reversed.
This ties invertibility to geometry: a transformation that collapses dimensions cannot be undone.
The transpose reflects a matrix across its diagonal.
Definition: For $A = [a_{ij}]$,
$$ A^T = [a_{ji}]. $$
In words: rows become columns, columns become rows.
Example:
$$ A = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix}, \quad A^T = \begin{bmatrix} 1 & 4 \ 2 & 5 \ 3 & 6 \end{bmatrix}. $$
Properties:
Two important classes emerge from the transpose:
Symmetric matrices: $A = A^T$. Example:
$$ \begin{bmatrix} 2 & 3 \ 3 & 5 \end{bmatrix}. $$
These have beautiful properties: real eigenvalues and orthogonal eigenvectors.
Orthogonal matrices: $Q^TQ = I$. Their columns form an orthonormal set, and they represent pure rotations/reflections.
Without these tools, matrix algebra would lack structure and reversibility.
$$ \begin{bmatrix} 1 & 0 & 2 \ -3 & 4 & 5 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 1 & 2 \ 0 & 3 \end{bmatrix}, \quad B = \begin{bmatrix} 4 & 0 \ 5 & 6 \end{bmatrix}. $$
$$ \begin{bmatrix} 3 & 2 \ 1 & 1 \end{bmatrix}. $$
Through these exercises, you'll see how identity, inverse, and transpose anchor the structure of linear algebra, providing neutrality, reversibility, and symmetry in every calculation.
Not all matrices are created equal-some have special shapes or patterns that give them unique properties. These structured matrices are the workhorses of linear algebra: they simplify computation, reveal geometry, and form the building blocks for algorithms. In this section, we study four especially important classes: symmetric, diagonal, triangular, and permutation matrices.
A matrix is symmetric if it equals its transpose:
$$ A = A^T. $$
Example:
$$ \begin{bmatrix} 2 & 3 & 4 \ 3 & 5 & 6 \ 4 & 6 & 9 \end{bmatrix}. $$
A matrix is diagonal if all non-diagonal entries are zero.
$$ D = \begin{bmatrix} d_1 & 0 & 0 \ 0 & d_2 & 0 \ 0 & 0 & d_3 \end{bmatrix}. $$
Multiplying by $D$ scales each coordinate separately.
Computations with diagonals are lightning fast:
Example:
\begin{bmatrix} 2x \ 3y \end{bmatrix}. $$
This is why diagonalization is so valuable: turning a general matrix into a diagonal one simplifies everything.
A matrix is upper triangular if all entries below the main diagonal are zero, and lower triangular if all entries above the diagonal are zero.
Upper triangular example:
$$ \begin{bmatrix} 1 & 2 & 3 \ 0 & 4 & 5 \ 0 & 0 & 6 \end{bmatrix}. $$
Lower triangular example:
$$ \begin{bmatrix} 7 & 0 & 0 \ 8 & 9 & 0 \ 10 & 11 & 12 \end{bmatrix}. $$
Why they matter:
A permutation matrix is obtained by permuting the rows (or columns) of an identity matrix.
Example:
$$ P = \begin{bmatrix} 0 & 1 & 0 \ 1 & 0 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
Multiplying by $P$:
Permutation matrices are used in pivoting strategies in elimination, ensuring numerical stability in solving systems. They are also orthogonal: $P^{-1} = P^T$.
Together, these classes show that structure leads to simplicity-many computational algorithms exploit these patterns for speed and stability.
Almost every advanced method in numerical linear algebra relies on reducing general matrices into one of these structured forms.
$$ \begin{bmatrix} 1 & 2 \ 2 & 5 \end{bmatrix} $$
is symmetric. Find its transpose.
$$ \begin{bmatrix} 3 & 0 & 0 \ 0 & 4 & 0 \ 0 & 0 & 5 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 3 & 1 \ 0 & 5 & 2 \ 0 & 0 & 4 \end{bmatrix} \mathbf{x} = \begin{bmatrix} 1 \ 2 \ 3 \end{bmatrix} $$
using back substitution.
By exploring these four structured families, you'll start to see that not all matrices are messy-many have order hidden in their arrangement, and exploiting that order is the key to both theoretical understanding and efficient computation.
So far we have studied shapes, multiplication rules, and special classes of matrices. In this section we introduce a simple but surprisingly powerful quantity: the trace of a matrix. Along with it, we review a set of basic matrix properties that provide shortcuts, invariants, and insights into how matrices behave.
For a square matrix $A = [a_{ij}]$ of size $n \times n$, the trace is the sum of the diagonal entries:
$$ \text{tr}(A) = a_{11} + a_{22} + \cdots + a_{nn}. $$
Example:
$$ A = \begin{bmatrix} 2 & 5 & 7 \ 0 & 3 & 1 \ 4 & 6 & 8 \end{bmatrix}, \quad \text{tr}(A) = 2 + 3 + 8 = 13. $$
The trace extracts a single number summarizing the "diagonal content" of a matrix.
The trace is linear and interacts nicely with multiplication and transposition:
Linearity:
Cyclic Property:
Transpose Invariance:
Similarity Invariance:
One of the most important connections is between the trace and eigenvalues:
$$ \text{tr}(A) = \lambda_1 + \lambda_2 + \cdots + \lambda_n, $$
where $\lambda_i$ are the eigenvalues of $A$ (counting multiplicity).
This links the simple diagonal sum to the deep spectral properties of the matrix.
Example:
$$ A = \begin{bmatrix} 1 & 0 \ 0 & 3 \end{bmatrix}, \quad \text{tr}(A) = 4, \quad \lambda_1 = 1, ; \lambda_2 = 3, \quad \lambda_1 + \lambda_2 = 4. $$
Alongside the trace, here are some important algebraic facts that every student of linear algebra must know:
Determinant vs. Trace:
Norms and Inner Products:
Orthogonal Invariance:
The trace is deceptively simple but incredibly powerful:
$$ \begin{bmatrix} 4 & 2 & 0 \ -1 & 3 & 5 \ 7 & 6 & 1 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 1 & 2 \ 0 & 3 \end{bmatrix}, \quad B = \begin{bmatrix} 4 & 0 \ 5 & 6 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 1 \ 1 & 2 \end{bmatrix}, $$
compute its eigenvalues and check that their sum equals the trace.
Mastering the trace and its properties will prepare you for the next leap: understanding how matrices interact with volume, orientation, and determinants.
Up to now, matrices have been used to describe linear transformations: scaling, rotating, reflecting, projecting. But real-world geometry often involves more than just linear effects-it includes translations (shifts) as well. A pure linear map cannot move the origin, so to handle translations (and combinations of them with rotations, scalings, and shears), we extend our toolkit to affine transformations. The secret weapon that makes this work is the idea of homogeneous coordinates.
An affine transformation is any map of the form:
$$ f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}, $$
where $A$ is a matrix (linear part) and $\mathbf{b}$ is a vector (translation part).
Examples in 2D:
Affine maps preserve parallel lines and ratios of distances along lines, but not necessarily angles or lengths.
If we only use a 2×2 matrix in 2D or 3×3 in 3D, the origin always stays fixed. That's a limitation: real-world movements (like moving a shape from one place to another) require shifting the origin too. To capture both linear and translational effects uniformly, we need a clever trick.
The trick is to add one extra coordinate.
This new representation is called homogeneous coordinates. It allows us to fold translations into matrix multiplication.
In 2D:
\begin{bmatrix} ax + by + t_x \ cx + dy + t_y \ 1 \end{bmatrix}. $$
Here,
So with one unified matrix, we can handle both linear transformations and shifts.
$$ \begin{bmatrix} 1 & 0 & 2 \ 0 & 1 & 3 \ 0 & 0 & 1 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 0 & -1 \ 0 & 3 & 4 \ 0 & 0 & 1 \end{bmatrix}. $$
$$ \begin{bmatrix} 0 & -1 & 5 \ 1 & 0 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
In 3D, affine transformations use 4×4 matrices. The upper-left 3×3 block handles rotation, scaling, or shear; the last column encodes translation.
Example: translation by (2, –1, 4):
$$ \begin{bmatrix} 1 & 0 & 0 & 2 \ 0 & 1 & 0 & -1 \ 0 & 0 & 1 & 4 \ 0 & 0 & 0 & 1 \end{bmatrix}. $$
This formulation is universal in computer graphics and robotics.
Mastering affine transformations and homogeneous coordinates bridges the gap between pure linear algebra and real-world geometry, giving you the mathematical foundation behind computer graphics, robotics, and spatial modeling.
Thus far, we have studied what matrices are and what they represent. But in practice, working with matrices also means thinking about computation-how much work operations take, how algorithms can be sped up, and why structure matters. This section introduces the basic ideas of computational cost in matrix operations, simple strategies for efficiency, and why these considerations are crucial in modern applications.
The simplest way to measure the cost of a matrix operation is to count the basic arithmetic operations (additions and multiplications).
Matrix–vector product: For an $m \times n$ matrix and an $n \times 1$ vector:
Matrix–matrix product: For an $m \times n$ matrix times an $n \times p$ matrix:
Gaussian elimination (solving $Ax=b$): For an $n \times n$ system:
These counts show how quickly costs grow with dimension. Doubling $n$ makes the work 8 times larger for elimination.
Cost is not only arithmetic: storage also matters.
Modern computing leverages hardware for speed:
This is why linear algebra libraries (BLAS, LAPACK, cuBLAS) are indispensable: they squeeze performance from hardware.
By learning to count costs and exploit structure, you prepare yourself not only to understand matrices abstractly but also to use them effectively in real-world, large-scale problems. This balance between theory and computation is at the heart of modern linear algebra.
Patterns intertwine,
transformations gently fold,
structure in the square.
Linear algebra often begins with systems of equations-collections of unknowns linked by linear relationships. While these systems can be solved directly using substitution or elimination, they quickly become messy when there are many variables. The key insight of linear algebra is that all systems of linear equations can be captured compactly by matrices and vectors. This section explains how we move from equations written out in words and symbols to the matrix form that powers computation.
Consider this system of two equations in two unknowns:
$$
\begin{cases}
2x + y = 5 \
3x - y = 4
\end{cases}
$$
At first glance, this is just algebra: two equations, two unknowns. But notice the structure: each equation is a sum of multiples of the variables, set equal to a constant. This pattern-linear combinations of unknowns equal to a result-is exactly what matrices capture.
Extract the coefficients of each variable from the system:
Arrange these coefficients in a rectangular array:
$$
A = \begin{bmatrix}
2 & 1 \
3 & -1
\end{bmatrix}.
$$
This matrix $A$ is called the coefficient matrix.
Next, write the unknowns as a vector:
$$ \mathbf{x} = \begin{bmatrix} x \ y \end{bmatrix}. $$
Finally, write the right-hand side constants as another vector:
$$ \mathbf{b} = \begin{bmatrix} 5 \ 4 \end{bmatrix}. $$
Now the entire system can be written in a single line:
$$ A\mathbf{x} = \mathbf{b}. $$
This compact form hides no information; it is equivalent to the original equations. But it gives us enormous advantages:
System of three equations in three unknowns:
$$
\begin{cases}
x + 2y - z = 2 \
2x - y + 3z = 1 \
3x + y + 2z = 4
\end{cases}
$$
Coefficient matrix:
$$
A = \begin{bmatrix}
1 & 2 & -1 \
2 & -1 & 3 \
3 & 1 & 2
\end{bmatrix}.
$$
Unknown vector:
$$ \mathbf{x} = \begin{bmatrix} x \ y \ z \end{bmatrix}. $$
Constant vector:
$$ \mathbf{b} = \begin{bmatrix} 2 \ 1 \ 4 \end{bmatrix}. $$
Matrix form:
$$ A\mathbf{x} = \mathbf{b}. $$
This single equation captures three equations and three unknowns in one object.
For the 2×2 case earlier:
$$
A\mathbf{x} = \begin{bmatrix} 2 & 1 \ 3 & -1 \end{bmatrix}
\begin{bmatrix} x \ y \end{bmatrix}
= x \begin{bmatrix} 2 \ 3 \end{bmatrix} + y \begin{bmatrix} 1 \ -1 \end{bmatrix}.
$$
So solving the system means finding scalars $x, y$ that combine the columns of $A$ to reach $\mathbf{b}$.
Sometimes we want to save space further. We can put the coefficients and constants side by side in an augmented matrix:
$$
[A | \mathbf{b}] =
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}.
$$
This form is especially useful for elimination methods, where we manipulate rows without writing variables at each step.
This step-rewriting equations as matrix form-is the gateway into linear algebra. Once you can do it, you no longer think of systems of equations as isolated lines on paper, but as a unified object that can be studied with general tools. It opens the door to:
Every major idea flows from this compact representation.
Write the system
$$
\begin{cases}
4x - y = 7 \
-2x + 3y = 5
\end{cases}
$$
in matrix form.
For the system
$$
\begin{cases}
x + y + z = 6 \
2x - y + z = 3 \
x - y - z = -2
\end{cases}
$$
build the coefficient matrix, unknown vector, and constant vector.
Express the augmented matrix for the above system.
Challenge: Interpret the system in column view. What does it mean geometrically to express $(6, 3, -2)$ as a linear combination of the columns of the coefficient matrix?
By practicing these rewrites, you will see that linear algebra is not about juggling many equations-it is about seeing structure in one compact equation. This step transforms scattered equations into the language of matrices, where the real power begins.
Once a system of linear equations has been expressed as a matrix, the next step is to simplify that matrix into a form where the solutions become clear. The main tool for this simplification is the set of elementary row operations. These operations allow us to manipulate the rows of a matrix in systematic ways that preserve the solution set of the corresponding system of equations.
There are exactly three types of legal row operations, each with a clear algebraic meaning:
Row Swapping ($R_i \leftrightarrow R_j$): Exchange two rows. This corresponds to reordering equations in a system. Since the order of equations doesn't change the solutions, this operation is always valid.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \longrightarrow \quad
\begin{bmatrix}
3 & -1 & | & 4 \
2 & 1 & | & 5
\end{bmatrix}.
$$
Row Scaling ($R_i \to cR_i, ; c \neq 0$): Multiply all entries in a row by a nonzero constant. This is like multiplying both sides of an equation by the same number, which doesn't change its truth.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \longrightarrow \quad
\begin{bmatrix}
1 & \tfrac{1}{2} & | & \tfrac{5}{2} \
3 & -1 & | & 4
\end{bmatrix}.
$$
Row Replacement ($R_i \to R_i + cR_j$): Add a multiple of one row to another. This corresponds to replacing one equation with a linear combination of itself and another, a fundamental elimination step.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \overset{R_2 \to R_2 - \tfrac{3}{2}R_1}{\longrightarrow} \quad
\begin{bmatrix}
2 & 1 & | & 5 \
0 & -\tfrac{5}{2} & | & -\tfrac{7}{2}
\end{bmatrix}.
$$
These three operations are the backbone of elimination because they do not alter the solution set of the system. Each is equivalent to applying an invertible transformation:
Thus, each operation is invertible, and the transformed system is always equivalent to the original.
Each elementary row operation can itself be represented by multiplying on the left with a special matrix called an elementary matrix.
For example:
Swapping rows 1 and 2 in a 2×2 system is done by
$$ E = \begin{bmatrix} 0 & 1 \ 1 & 0 \end{bmatrix}. $$
Scaling row 1 by 3 in a 2×2 system is done by
$$ E = \begin{bmatrix} 3 & 0 \ 0 & 1 \end{bmatrix}. $$
This perspective is crucial later for factorization methods like LU decomposition, where elimination is expressed as a product of elementary matrices.
System:
$$
\begin{cases}
x + 2y = 4 \
3x + 4y = 10
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & | & 4 \
3 & 4 & | & 10
\end{bmatrix}.
$$
Eliminate the $3x$ under the first pivot: $R_2 \to R_2 - 3R_1$.
$$
\begin{bmatrix}
1 & 2 & | & 4 \
0 & -2 & | & -2
\end{bmatrix}.
$$
Scale the second row: $R_2 \to -\tfrac{1}{2}R_2$.
$$
\begin{bmatrix}
1 & 2 & | & 4 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Eliminate above the pivot: $R_1 \to R_1 - 2R_2$.
$$
\begin{bmatrix}
1 & 0 & | & 2 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Solution: $x = 2, ; y = 1$.
Row operations do not alter the solution space:
Thus, row operations act like "reshaping the system" while leaving the intersection intact.
Row operations are the essential moves in solving linear systems by hand or computer. They:
Apply row operations to reduce
$$
\begin{bmatrix}
2 & 1 & | & 7 \
1 & -1 & | & 1
\end{bmatrix}
$$
to a form where the solution is obvious.
Show explicitly why swapping two equations in a system doesn't change its solutions.
Construct the elementary matrix for "add –2 times row 1 to row 3" in a 3×3 system.
Challenge: Prove that any elementary row operation corresponds to multiplication by an invertible matrix.
Mastering these operations equips you with the mechanical and conceptual foundation for the next stage: systematically reducing matrices to row-echelon form.
After introducing row operations, the natural question is: what are we trying to achieve by performing them? The answer is to transform a matrix into a standardized, simplified form where the solutions to the corresponding system of equations can be read off directly. Two such standardized forms are central in linear algebra: row-echelon form (REF) and reduced row-echelon form (RREF).
A matrix is in row-echelon form if:
Example of REF:
$$
\begin{bmatrix}
1 & 2 & 3 & | & 4 \
0 & 1 & -1 & | & 2 \
0 & 0 & 5 & | & -3 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Here, the pivots are the first 1 in row 1, the 1 in row 2, and the 5 in row 3. Each pivot is to the right of the one above it, and all entries below pivots are zero.
A matrix is in reduced row-echelon form if, in addition to the rules of REF:
Example of RREF:
$$
\begin{bmatrix}
1 & 0 & 0 & | & 3 \
0 & 1 & 0 & | & -2 \
0 & 0 & 1 & | & 1
\end{bmatrix}.
$$
This form is so simplified that solutions can be read directly: here, $x=3$, $y=-2$, $z=1$.
System:
$$
\begin{cases}
x + 2y + z = 4 \
2x + 5y + z = 7 \
3x + 6y + 2z = 10
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
2 & 5 & 1 & | & 7 \
3 & 6 & 2 & | & 10
\end{bmatrix}.
$$
Eliminate below first pivot (the 1 in row 1, col 1):
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & -1 \
0 & 0 & -1 & | & -2
\end{bmatrix}.
$$
This is now in REF.
Scale pivots and eliminate above them:
Final:
$$
\begin{bmatrix}
1 & 0 & 0 & | & 2 \
0 & 1 & 0 & | & 1 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
Solution: $x=2, y=1, z=2$.
Reduce
$$
\begin{bmatrix}
2 & 4 & | & 6 \
1 & 3 & | & 5
\end{bmatrix}
$$
to REF, then RREF.
Find the RREF of
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
2 & 3 & 4 & | & 8 \
1 & 2 & 3 & | & 5
\end{bmatrix}.
$$
Explain why two different elimination sequences can lead to different REF but the same RREF.
Challenge: Prove that every matrix has a unique RREF by considering the effect of row operations systematically.
Reaching row-echelon and reduced row-echelon forms transforms messy systems into structured ones, turning algebraic clutter into an organized path to solutions.
When reducing a matrix to row-echelon or reduced row-echelon form, certain positions in the matrix take on a special importance. These are the pivots-the leading nonzero entries in each row. Around them, the entire solution structure of a linear system is organized. Understanding pivots, the variables they anchor, and the freedom that arises from non-pivot columns is essential to solving linear equations systematically.
In row-echelon form, a pivot is the first nonzero entry in a row, moving from left to right. After scaling in reduced row-echelon form, each pivot is set to exactly 1.
Example:
$$
\begin{bmatrix}
1 & 2 & 0 & | & 5 \
0 & 1 & 3 & | & -2 \
0 & 0 & 0 & | & 0
\end{bmatrix}
$$
Columns with pivots are pivot columns. Columns without pivots correspond to free variables.
Example:
$$
\begin{bmatrix}
1 & 0 & 2 & | & 3 \
0 & 1 & -1 & | & 4
\end{bmatrix}.
$$
This corresponds to:
$$ x_1 + 2x_3 = 3, \quad x_2 - x_3 = 4. $$
Here:
Thus, $x_1$ and $x_2$ depend on $x_3$:
$$ x_1 = 3 - 2x_3, \quad x_2 = 4 + x_3, \quad x_3 \text{ free}. $$
The solution set is infinite, described by the freedom in $x_3$.
In 2D:
Thus, the number of free variables determines the dimension of the solution set.
The number of pivot columns equals the rank of the matrix.
If the coefficient matrix $A$ is $m \times n$:
This is the rank–nullity connection in action:
$$ \text{number of variables} = \text{rank} + \text{nullity}. $$
System:
$$
\begin{cases}
x + 2y + z = 4 \
2x + 5y + z = 7
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
2 & 5 & 1 & | & 7
\end{bmatrix}.
$$
Reduce:
$R_2 \to R_2 - 2R_1$ →
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & -1
\end{bmatrix}.
$$
Now:
Solution:
$$ x = 4 - 2y - z, \quad y = -1 + z, \quad z \text{ free}. $$
Substitute:
$$ (x, y, z) = (6 - 3z, ; -1 + z, ; z). $$
Solutions form a line in 3D parameterized by $z$.
In RREF, each pivot is scaled to 1, making it easy to isolate pivot variables. Without leading ones, equations may still be correct but harder to interpret.
For example:
$$
\begin{bmatrix}
2 & 0 & | & 6 \
0 & -3 & | & 9
\end{bmatrix}
$$
becomes
$$
\begin{bmatrix}
1 & 0 & | & 3 \
0 & 1 & | & -3
\end{bmatrix}.
$$
The solutions are immediately visible: $x=3, y=-3$.
Reduce
$$
\begin{bmatrix}
1 & 3 & 1 & | & 5 \
2 & 6 & 2 & | & 10
\end{bmatrix}
$$
and identify pivot and free variables.
For the system
$$ x + y + z = 2, \quad 2x + 3y + 5z = 7, $$
write the RREF and express the solution with free variables.
Compute the rank and number of free variables of a 3×5 matrix with two pivot columns.
Challenge: Show that if the number of pivots equals the number of variables, the system has either no solution or a unique solution, but never infinitely many.
Understanding pivots and free variables provides the key to classifying solution sets: unique, infinite, or none. This classification lies at the heart of solving linear systems.
A system of linear equations is called consistent if it has at least one solution. Consistency is the first property to check when working with a system, because before worrying about uniqueness or parametrization, we must know whether a solution exists at all. This section explains how to recognize consistent systems, how to solve them using row-reduction, and how to describe their solutions in terms of pivots and free variables.
Given a system $A\mathbf{x} = \mathbf{b}$:
Consistency depends on the relationship between the vector $\mathbf{b}$ and the column space of $A$:
$$ \mathbf{b} \in \text{Col}(A) \quad \iff \quad \text{system is consistent}. $$
If $\mathbf{b}$ cannot be written as a linear combination of the columns of $A$, the system has no solution.
To test consistency, reduce the augmented matrix $[A | \mathbf{b}]$ to row-echelon form.
If you find a row of the form:
$$ [0 ;; 0 ;; \dots ;; 0 ;|; c], \quad c \neq 0, $$
then the system is inconsistent (contradiction: 0 = c).
If no such contradiction appears, the system is consistent.
System:
$$
\begin{cases}
x + y = 2 \
x - y = 0
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
1 & -1 & | & 0
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - R_1$:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & -2 & | & -2
\end{bmatrix}.
$$
$R_2 \to -\tfrac{1}{2}R_2$:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & 1 & | & 1
\end{bmatrix}.
$$
$R_1 \to R_1 - R_2$:
$$
\begin{bmatrix}
1 & 0 & | & 1 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Solution: $x = 1, ; y = 1$. Unique solution.
System:
$$
\begin{cases}
x + y + z = 3 \
2x + 2y + 2z = 6
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
2 & 2 & 2 & | & 6
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - 2R_1$:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
No contradiction, so consistent. Solution:
$$ x = 3 - y - z, \quad y \text{ free}, \quad z \text{ free}. $$
The solution set is a plane in $\mathbb{R}^3$.
System:
$$
\begin{cases}
x + y = 1 \
x + y = 2
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
1 & 1 & | & 2
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - R_1$:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
0 & 0 & | & 1
\end{bmatrix}.
$$
Contradiction: $0 = 1$. Inconsistent, no solution.
In 2D:
In 3D:
Reduce the augmented matrix
$$
\begin{bmatrix}
1 & 2 & 1 & | & 5 \
2 & 4 & 2 & | & 10 \
3 & 6 & 3 & | & 15
\end{bmatrix}
$$
and determine if the system is consistent.
Classify the system as having unique, infinite, or no solutions:
$$
\begin{cases}
x + y + z = 2 \
x - y + z = 0 \
2x + 0y + 2z = 3
\end{cases}
$$
Explain geometrically what it means when the augmented matrix has a contradictory row.
Challenge: Show algebraically that a system is consistent if and only if $\mathbf{b}$ lies in the span of the columns of $A$.
Consistent systems mark the balance point between algebraic rules and geometric reality: they are where equations and space meet in harmony.
Not every system of linear equations has a solution. Some are inconsistent, meaning the equations contradict one another and no vector $\mathbf{x}$ can satisfy them all at once. Detecting such inconsistency early is crucial: it saves wasted effort trying to solve an impossible system and reveals important geometric and algebraic properties.
Consider the system:
$$
\begin{cases}
x + y = 1 \
x + y = 3
\end{cases}
$$
Clearly, the two equations cannot both be true. In augmented matrix form:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
1 & 1 & | & 3
\end{bmatrix}.
$$
Row reduction gives:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
0 & 0 & | & 2
\end{bmatrix}.
$$
The bottom row says $0 = 2$, a contradiction. This is the hallmark of inconsistency: a row of zeros in the coefficient part, with a nonzero constant in the augmented part.
A system $A\mathbf{x} = \mathbf{b}$ is inconsistent if, after row reduction, the augmented matrix contains a row of the form:
$$ [0 ;; 0 ;; \dots ;; 0 ;|; c], \quad c \neq 0. $$
This indicates that all variables vanish from the equation, leaving an impossible statement like $0 = c$.
$$
\begin{cases}
x + y = 2 \
2x + 2y = 5
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
2 & 2 & | & 5
\end{bmatrix}.
$$
Row reduce:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & 0 & | & 1
\end{bmatrix}.
$$
Contradiction: no solution. Geometrically, the two equations are parallel lines that never intersect.
$$
\begin{cases}
x + y + z = 1 \
2x + 2y + 2z = 2 \
x + y + z = 3
\end{cases}
$$
The first and third equations already conflict: the same plane equation is forced to equal two different constants.
Augmented matrix reduces to:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 1 \
0 & 0 & 0 & | & 0 \
0 & 0 & 0 & | & 2
\end{bmatrix}.
$$
Contradiction: no solution. The "planes" fail to intersect in common.
Another way to detect inconsistency is using ranks.
Rule:
This rank condition is fundamental and works in any dimension.
$$
\begin{bmatrix}
1 & -1 & | & 2 \
2 & -2 & | & 5
\end{bmatrix}
$$
and decide if the system is consistent.
$$ x + y = 0, \quad x + y = 1 $$
is inconsistent.
$$
\begin{cases}
x + y + z = 2 \
2x + 2y + 2z = 4 \
3x + 3y + 3z = 5
\end{cases}
$$
Detecting inconsistency is not just about spotting contradictions-it connects algebra, geometry, and linear transformations, showing exactly when a system cannot possibly fit together.
Gaussian elimination is the systematic procedure for solving systems of linear equations by using row operations to simplify the augmented matrix. The goal is to transform the system into row-echelon form (REF) and then use back substitution to find the solutions. This method is the backbone of linear algebra computations and is the foundation of most computer algorithms for solving linear systems.
Suppose we have $n$ equations with $n$ unknowns.
System:
$$
\begin{cases}
x + 2y = 5 \
3x + 4y = 11
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & | & 5 \
3 & 4 & | & 11
\end{bmatrix}.
$$
Pivot at (1,1) = 1.
Eliminate below: $R_2 \to R_2 - 3R_1$.
$$
\begin{bmatrix}
1 & 2 & | & 5 \
0 & -2 & | & -4
\end{bmatrix}.
$$
Back substitution: From row 2: $-2y = -4 \implies y = 2$. Substitute into row 1: $x + 2(2) = 5 \implies x = 1$.
Solution: $(x, y) = (1, 2)$.
System:
$$
\begin{cases}
x + y + z = 6 \
2x + 3y + z = 14 \
x - y + 2z = 2
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
2 & 3 & 1 & | & 14 \
1 & -1 & 2 & | & 2
\end{bmatrix}.
$$
Step 1: Pivot at (1,1). Eliminate below:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & -2 & 1 & | & -4
\end{bmatrix}.
$$
Step 2: Pivot at (2,2). Eliminate below: $R_3 \to R_3 + 2R_2$.
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & 0 & -1 & | & 0
\end{bmatrix}.
$$
Step 3: Pivot at (3,3). Scale row: $R_3 \to -R_3$.
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & 0 & 1 & | & 0
\end{bmatrix}.
$$
Back substitution:
Solution: $(x, y, z) = (4, 2, 0)$.
Elimination corresponds to progressively restricting the solution set:
Solve the system
$$
\begin{cases}
2x + y = 7 \
4x + 3y = 15
\end{cases}
$$
using Gaussian elimination.
Reduce
$$
\begin{bmatrix}
1 & 2 & -1 & | & 3 \
3 & 8 & 1 & | & 12 \
2 & 6 & 3 & | & 11
\end{bmatrix}
$$
to REF and solve.
Practice with a system that has infinitely many solutions:
$$ x + y + z = 4, \quad 2x + 2y + 2z = 8. $$
Challenge: Explain why Gaussian elimination always terminates in at most $n$ pivot steps for an $n \times n$ system.
Gaussian elimination transforms the complexity of many equations into an orderly process, making the hidden structure of solutions visible step by step.
Once Gaussian elimination reduces a system to row-echelon form (REF), the next step is to actually solve for the unknowns. This process is called back substitution: we begin with the bottom equation (which involves the fewest variables) and work our way upward, solving step by step. Back substitution is what converts the structured triangular system into explicit solutions.
A system in REF looks like this:
$$ \begin{bmatrix}
& * & * & * & | & * \
0 & * & * & * & | & * \
0 & 0 & * & * & | & * \
0 & 0 & 0 & * & | & *
\end{bmatrix}
$$
Each row corresponds to an equation with fewer variables than the row above.
The bottom equation has only one or two variables.
This triangular form makes it possible to solve "from the bottom up."
System after elimination:
$$
\begin{bmatrix}
1 & 2 & -1 & | & 3 \
0 & 1 & 2 & | & 4 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
This corresponds to:
$$
\begin{cases}
x + 2y - z = 3 \
y + 2z = 4 \
z = 2
\end{cases}
$$
Solution: $(x, y, z) = (5, 0, 2)$.
Not all systems reduce to unique solutions. If there are free variables (non-pivot columns), back substitution expresses pivot variables in terms of free ones.
Example:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & 1 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Equations:
$$
\begin{cases}
x + 2y + z = 4 \
y - z = 1
\end{cases}
$$
Solution set:
$$ (x, y, z) = (2 - 3t, ; 1 + t, ; t), \quad t \in \mathbb{R}. $$
Here $z = t$ is the free variable. The solutions form a line in 3D.
For a consistent system:
For the infinite-solution example above:
$$ (x, y, z) = (2, 1, 0) + t(-3, 1, 1). $$
This expresses the solution set as a base point plus a direction vector, making the geometry clear.
$$
\begin{bmatrix}
1 & -1 & 2 & | & 3 \
0 & 1 & 3 & | & 5 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
$$ x + y + z = 2, \quad 2x + 2y + 2z = 4. $$
Express the solution set of the above system in parametric vector form.
Challenge: For a 4×4 system with two free variables, explain why the solution set forms a plane in $\mathbb{R}^4$.
Back substitution completes the elimination process, translating triangular structure into explicit solutions, and shows how algebra and geometry meet in the classification of solution sets.
The concept of rank lies at the heart of linear algebra. It connects the algebra of solving systems, the geometry of subspaces, and the structure of matrices into one unifying idea. Rank measures the amount of independent information in a matrix: how many rows or columns carry unique directions instead of being repetitions or combinations of others.
The rank of a matrix $A$ is the number of pivots in its row-echelon form. Equivalently, it is:
All these definitions agree.
When solving a system with Gaussian elimination:
Example:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & 2 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Here, there are 2 pivots. So:
A set of vectors is linearly independent if none can be expressed as a combination of the others.
Example:
$$
\begin{bmatrix}
1 & 2 & 3 \
2 & 4 & 6 \
3 & 6 & 9
\end{bmatrix}.
$$
Here, each row is a multiple of the first. Rank = 1, since only one independent row/column direction exists.
Consider $A\mathbf{x} = \mathbf{b}$.
Thus, rank classifies solution sets.
Rank tells us the dimension of the subspace spanned by rows or columns.
Example:
It is a remarkable fact that the number of independent rows = number of independent columns. This is not obvious at first glance, but it is always true. So we can define rank either by rows or by columns-it makes no difference.
Find the rank of
$$
\begin{bmatrix}
1 & 2 & 3 \
2 & 4 & 5 \
3 & 6 & 8
\end{bmatrix}.
$$
Solve the system
$$ x + y + z = 2, \quad 2x + 2y + 2z = 4, $$
and identify the rank of the coefficient matrix.
In $\mathbb{R}^3$, what is the geometric meaning of a 3×3 matrix of rank 2?
Challenge: Prove that the row rank always equals the column rank by considering the echelon form of the matrix.
Rank is the first truly unifying concept in linear algebra: it tells us how much independent structure a matrix contains and sets the stage for understanding spaces, dimensions, and transformations.
Gaussian elimination not only solves systems but also reveals a deeper structure: many matrices can be factored into simpler pieces. One of the most useful is the LU factorization, where a matrix $A$ is written as the product of a lower-triangular matrix $L$ and an upper-triangular matrix $U$. This factorization captures all the elimination steps in a compact form and allows systems to be solved efficiently.
If $A$ is an $n \times n$ matrix, then
$$ A = LU, $$
where:
This means:
Take
$$
A = \begin{bmatrix}
2 & 3 \
4 & 7
\end{bmatrix}.
$$
Elimination: $R_2 \to R_2 - 2R_1$.
Multiplier = 2 (used to eliminate entry 4).
Resulting $U$:
$$
U = \begin{bmatrix}
2 & 3 \
0 & 1
\end{bmatrix}.
$$
$L$:
$$
L = \begin{bmatrix}
1 & 0 \
2 & 1
\end{bmatrix}.
$$
Check:
$$
LU = \begin{bmatrix}
1 & 0 \
2 & 1
\end{bmatrix}
\begin{bmatrix}
2 & 3 \
0 & 1
\end{bmatrix}
= \begin{bmatrix}
2 & 3 \
4 & 7
\end{bmatrix} = A.
$$
$$
A = \begin{bmatrix}
2 & 1 & 1 \
4 & -6 & 0 \
-2 & 7 & 2
\end{bmatrix}.
$$
Step 1: Eliminate below pivot (row 1).
Step 2: Eliminate below pivot in column 2.
Result:
$$
L = \begin{bmatrix}
1 & 0 & 0 \
2 & 1 & 0 \
-1 & -1 & 1
\end{bmatrix}, \quad
U = \begin{bmatrix}
2 & 1 & 1 \
0 & -8 & -2 \
0 & 0 & 1
\end{bmatrix}.
$$
Thus $A = LU$.
Suppose $Ax = b$. If $A = LU$:
This two-step process is much faster than elimination from scratch each time, especially if solving multiple systems with the same $A$ but different $b$.
Sometimes elimination requires row swaps (to avoid division by zero or instability). Then factorization is written as:
$$ PA = LU, $$
where $P$ is a permutation matrix recording the row swaps. This is the practical form used in numerical computing.
LU decomposition separates the elimination process into:
Together, they represent the same linear transformation as $A$, but decomposed into simpler building blocks.
Factor
$$
A = \begin{bmatrix}
1 & 2 \
3 & 8
\end{bmatrix}
$$
into $LU$.
Solve
$$
\begin{bmatrix}
2 & 1 \
6 & 3
\end{bmatrix}
\begin{bmatrix}
x \ y
\end{bmatrix} =
\begin{bmatrix}
5 \ 15
\end{bmatrix}
$$
using LU decomposition.
Compute $\det(A)$ for
$$
A = \begin{bmatrix}
2 & 1 & 1 \
4 & -6 & 0 \
-2 & 7 & 2
\end{bmatrix}
$$
by using its LU factorization.
Challenge: Prove that if $A$ is invertible, then it has an LU factorization (possibly after row swaps).
LU factorization organizes elimination into a powerful tool: compact, efficient, and deeply tied to both the theory and practice of linear algebra.
Paths diverge or merge,
pivots mark the way forward,
truth distilled in rows.
Endless skies expand,
spaces within spaces grow,
freedom takes its shape.
Up to now, we have worked with vectors in $\mathbb{R}^2$, $\mathbb{R}^3$, and higher-dimensional Euclidean spaces. But the true power of linear algebra comes from abstracting away from coordinates. A vector space is not tied to arrows in physical space-it is any collection of objects that behave like vectors, provided they satisfy certain rules. These rules are called the axioms of vector spaces.
A vector space is a set $V$ equipped with two operations:
The magic is that as long as certain rules (axioms) hold, the objects in $V$ can be treated as vectors. They need not be arrows or coordinate lists-they could be polynomials, functions, matrices, or sequences.
Let $u, v, w \in V$ (vectors) and $a, b \in \mathbb{R}$ (scalars). The axioms are:
(These are sometimes listed as eight, with some grouped together, but the essence is the same.)
All of these satisfy the vector space axioms.
These examples show why the axioms are essential: without them, the structure breaks.
Every vector space must contain a zero vector. This is not optional. It is the "do nothing" element for addition. In $\mathbb{R}^n$, this is $(0,0,\dots,0)$. In polynomials, it is the zero polynomial. In function spaces, it is the function $f(x) = 0$.
For every vector $v$, we require $-v$. This ensures that equations like $u+v=w$ can always be rearranged to $u=w-v$. Without additive inverses, solving linear equations would not work.
Scalars come from a field: usually the real numbers $\mathbb{R}$ or the complex numbers $\mathbb{C}$. The choice of scalars matters:
Together, these rules make vector spaces stable and reliable mathematical objects.
The axioms of vector spaces provide the foundation on which the rest of linear algebra is built. Everything that follows-subspaces, independence, basis, dimension-grows naturally from this formal framework.
Once the idea of a vector space is in place, the next step is to recognize smaller vector spaces that live inside bigger ones. These are called subspaces. Subspaces are central in linear algebra because they reveal the internal structure of matrices and linear systems. Two special subspaces-the column space and the null space-play particularly important roles.
A subspace $W$ of a vector space $V$ is a subset of $V$ that is itself a vector space under the same operations. To qualify as a subspace, $W$ must satisfy:
That's it-no further checking of all ten vector space axioms is needed, because those are inherited from $V$.
In $\mathbb{R}^3$:
In the space of polynomials:
In function spaces:
Given a matrix $A$, the column space is the set of all linear combinations of its columns. Formally,
$$ C(A) = { A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n }. $$
Example:
$$
A = \begin{bmatrix}
1 & 2 \
2 & 4 \
3 & 6
\end{bmatrix}.
$$
The second column is just twice the first. So the column space is all multiples of $\begin{bmatrix}1 \ 2 \ 3\end{bmatrix}$, which is a line in $\mathbb{R}^3$. Rank = 1.
The null space (or kernel) of a matrix $A$ is the set of all vectors $\mathbf{x}$ such that
$$ A\mathbf{x} = 0. $$
Example:
$$
A = \begin{bmatrix}
1 & 2 & 3 \
4 & 5 & 6
\end{bmatrix}.
$$
Solve $A\mathbf{x} = 0$. This yields a null space spanned by one vector, meaning it is a line through the origin in $\mathbb{R}^3$.
Together, they capture the full behavior of a matrix.
Show that the set ${(x,y,0) : x,y \in \mathbb{R}}$ is a subspace of $\mathbb{R}^3$.
For
$$
A = \begin{bmatrix}
1 & 2 & 3 \
0 & 0 & 0 \
1 & 2 & 3
\end{bmatrix},
$$
find the column space and its dimension.
For the same $A$, compute the null space and its dimension.
Challenge: Prove that the null space of $A$ is always a subspace of $\mathbb{R}^n$.
Subspaces-especially the column space and null space-are the first glimpse of the hidden geometry inside every matrix, showing us which directions survive and which vanish.
The idea of a span captures the simplest and most powerful way to build new vectors from old ones: by taking linear combinations. A span is not just a set of scattered points but a structured, complete collection of all combinations of a given set of vectors. Understanding span leads directly to the concepts of bases, dimension, and the structure of subspaces.
Given vectors $v_1, v_2, \dots, v_k \in V$, the span of these vectors is
$$ \text{span}{v_1, v_2, \dots, v_k} = {a_1 v_1 + a_2 v_2 + \dots + a_k v_k : a_i \in \mathbb{R}}. $$
In $\mathbb{R}^2$:
In $\mathbb{R}^3$:
A set of vectors is a generating set (or spanning set) for a subspace if their span equals that subspace.
Different generating sets can span the same space. Some may be redundant, others minimal. Later, the concept of a basis refines this idea.
Consider the system $Ax=b$.
The concept of span transforms our perspective: instead of focusing on single vectors, we see the entire landscape of possibilities they generate.
Having introduced span and generating sets, the natural question arises: when are the vectors in a spanning set truly necessary, and when are some redundant? This leads to the idea of linear independence. It is the precise way to distinguish between essential vectors (those that add new directions) and dependent vectors (those that can be expressed in terms of others).
A set of vectors ${v_1, v_2, \dots, v_k}$ is linearly independent if the only solution to
$$ a_1 v_1 + a_2 v_2 + \dots + a_k v_k = 0 $$
is
$$ a_1 = a_2 = \dots = a_k = 0. $$
If there exists a nontrivial solution (some $a_i \neq 0$), then the vectors are linearly dependent.
$(1,0)$ and $(0,1)$ are independent.
$(1,0)$ and $(2,0)$ are dependent.
Any set of 3 vectors in $\mathbb{R}^2$ is dependent, since the dimension of the space is 2.
Put the vectors as columns in a matrix. Perform row reduction:
Example:
$$
\begin{bmatrix}
1 & 2 & 3 \
0 & 1 & 4 \
0 & 0 & 0
\end{bmatrix}.
$$
Here the third column has no pivot → the 3rd vector is dependent on the first two.
Linear independence is the tool that separates essential directions from redundant ones. It is the key to defining bases, counting dimensions, and understanding the structure of all vector spaces.
The concepts of span and linear independence come together in the powerful idea of a basis. A basis gives us the minimal set of building blocks needed to generate an entire vector space, with no redundancy. Once a basis is chosen, every vector in the space can be described uniquely by a list of numbers called its coordinates.
A basis of a vector space $V$ is a set of vectors ${v_1, v_2, \dots, v_k}$ that satisfies two properties:
In short: a basis is a spanning set with no redundancy.
These are called standard because they are the default way of describing coordinates.
One of the most important facts about bases is that they provide unique representations of vectors.
Given a basis ${v_1, \dots, v_k}$, any vector $x \in V$ can be written uniquely as:
$$ x = a_1 v_1 + a_2 v_2 + \dots + a_k v_k. $$
The coefficients $(a_1, a_2, \dots, a_k)$ are the coordinates of $x$ relative to that basis.
This uniqueness distinguishes bases from arbitrary spanning sets, where redundancy allows multiple representations.
Let basis = ${(1,0), (0,1)}$.
If we switch to a different basis, the coordinates change even though the vector itself does not.
Basis = ${(1,1), (1,-1)}$ in $\mathbb{R}^2$. Find coordinates of $x = (2,0)$.
Solve $a(1,1) + b(1,-1) = (2,0)$. This gives system:
$$ a + b = 2, \quad a - b = 0. $$
So $a=1, b=1$. Coordinates relative to this basis: $(1,1)$.
Notice: coordinates depend on basis choice.
For polynomials of degree ≤ 2: basis = ${1, x, x^2}$.
For continuous functions on $[0,1]$, one possible basis is the infinite set ${1, x, x^2, \dots}$.
This shows bases are not restricted to geometric vectors.
The number of vectors in a basis is the dimension of the vector space.
Dimension tells us how many independent directions exist in the space.
A basis is the minimal, elegant foundation of a vector space, turning the infinite into the manageable by providing a finite set of independent building blocks.
Dimension is one of the most profound and unifying ideas in linear algebra. It gives a single number that captures the "size" or "capacity" of a vector space: how many independent directions it has. Unlike length, width, or height in everyday geometry, dimension in linear algebra applies to spaces of any kind-geometric, algebraic, or even function spaces.
The dimension of a vector space $V$ is the number of vectors in any basis of $V$.
Since all bases of a vector space have the same number of elements, dimension is well-defined.
If $\dim V = n$, then:
$\dim(\mathbb{R}^2) = 2$.
$\dim(\mathbb{R}^3) = 3$.
The set of all polynomials of degree ≤ 2 has dimension 3.
The space of all $m \times n$ matrices has dimension $mn$.
Finite-dimensional spaces: $\mathbb{R}^n$, polynomials of degree ≤ $k$.
Infinite-dimensional spaces:
When solving $A\mathbf{x} = \mathbf{b}$:
The dimension of the column space = rank = number of independent directions in the outputs.
The dimension of the null space = number of free variables.
By the rank–nullity theorem:
$$ \dim(\text{column space}) + \dim(\text{null space}) = \text{number of variables}. $$
Thus, dimension = length of coordinate list.
Dimension is the measuring stick of linear algebra: it tells us how many independent pieces of information are needed to describe the whole space.
The rank–nullity theorem is one of the central results of linear algebra. It gives a precise balance between two fundamental aspects of a matrix: the dimension of its column space (rank) and the dimension of its null space (nullity). It shows that no matter how complicated a matrix looks, the distribution of information between its "visible" outputs and its "hidden" null directions always obeys a strict law.
Let $A$ be an $m \times n$ matrix (mapping $\mathbb{R}^n \to \mathbb{R}^m$):
$$ \text{rank}(A) + \text{nullity}(A) = n $$
where:
Think of a matrix as a machine that transforms input vectors into outputs:
This ensures nothing disappears mysteriously-every input direction is accounted for.
$$
A = \begin{bmatrix}
1 & 0 \
0 & 1 \
\end{bmatrix}.
$$
$$
A = \begin{bmatrix}
1 & 2 \
2 & 4 \
3 & 6 \
\end{bmatrix}.
$$
$$
A = \begin{bmatrix}
1 & 0 & 1 \
0 & 1 & 1
\end{bmatrix}.
$$
Row reduce $A$ to echelon form.
Pivots correspond to independent columns → count = rank.
Free variables correspond to null space directions → count = nullity.
Each column is either a pivot column or corresponds to a free variable, so:
$$ \text{rank} + \text{nullity} = \text{number of columns}. $$
The theorem guarantees the geometry of "surviving" and "vanishing" directions always adds up consistently.
Solving systems $Ax = b$:
Data compression: Rank identifies independent features; nullity shows redundancy.
Computer graphics: Rank–nullity explains how 3D coordinates collapse into 2D images: one dimension of depth is lost.
Machine learning: Rank signals how much real information a dataset contains; nullity indicates degrees of freedom that add nothing new.
Verify rank–nullity for
$$
A = \begin{bmatrix}
1 & 2 & 3 \
4 & 5 & 6
\end{bmatrix}.
$$
For a $4 \times 5$ matrix of rank 3, what is its nullity?
In $\mathbb{R}^3$, suppose a matrix maps all of space onto a line. What are its rank and nullity?
Challenge: Prove rigorously that the row space and null space are orthogonal complements, and use this to derive rank–nullity again.
The rank–nullity theorem is the law of balance in linear algebra: every input dimension is accounted for, either as a surviving direction (rank) or as one that vanishes (nullity).
Once a basis for a vector space is chosen, every vector in that space can be described uniquely in terms of the basis. These descriptions are called coordinates. Coordinates transform abstract vectors into concrete lists of numbers, making computation possible. Changing the basis changes the coordinates, but the underlying vector remains the same.
Given a vector space $V$ and a basis $B = {v_1, v_2, \dots, v_n}$, every vector $x \in V$ can be written uniquely as:
$$ x = a_1 v_1 + a_2 v_2 + \dots + a_n v_n. $$
The coefficients $(a_1, a_2, \dots, a_n)$ are the coordinates of $x$ with respect to the basis $B$.
This representation is unique because basis vectors are independent.
Standard basis: $B = {(1,0), (0,1)}$.
Non-standard basis: $B = {(1,1), (1,-1)}$.
Write $x = (3,5)$ as $a(1,1) + b(1,-1)$.
Solve:
$$ a+b = 3, \quad a-b = 5. $$
Adding: $2a = 8 \implies a = 4$. Subtracting: $2b = -2 \implies b = -1$.
Coordinates relative to this basis: $(4, -1)$.
The same vector looks different depending on the chosen basis.
Let $B = {(1,0,0), (1,1,0), (1,1,1)}$. Find coordinates of $x = (2,3,4)$.
Solve $a(1,0,0) + b(1,1,0) + c(1,1,1) = (2,3,4)$. This gives system:
$$ a+b+c = 2, \quad b+c = 3, \quad c = 4. $$
From $c=4$, we get $b+c=3 \implies b=-1$. Then $a+b+c=2 \implies a-1+4=2 \implies a=-1$. Coordinates: $(-1, -1, 4)$.
If $B = {v_1, \dots, v_n}$, form the basis matrix
$$ P = [v_1 \ v_2 \ \dots \ v_n]. $$
Then for a vector $x$, its coordinate vector $[x]_B$ satisfies
$$ P [x]_B = x. $$
Thus,
$$ [x]_B = P^{-1}x. $$
This shows coordinate transformation is simply matrix multiplication.
Suppose a vector has coordinates $[x]_B$ relative to basis $B$. If we switch to another basis $C$, we use a change-of-basis matrix to convert coordinates:
$$ [x]_C = (P_C^{-1} P_B) [x]_B. $$
This process is fundamental in computer graphics, robotics, and data transformations.
Coordinates relative to a basis are the bridge between geometry and algebra: they turn abstract spaces into numerical systems where computation, reasoning, and transformation become systematic and precise.
Every vector space allows multiple choices of basis, and each basis provides a different way of describing the same vectors. The process of moving from one basis to another is called a change of basis. To perform this change systematically, we use a change-of-basis matrix. This matrix acts as a translator between coordinate systems: it converts the coordinates of a vector relative to one basis into coordinates relative to another.
Let $V$ be a vector space with two bases:
Suppose a vector $x \in V$ has coordinates $[x]_B$ relative to $B$, and $[x]_C$ relative to $C$.
We want a matrix $P_{B \to C}$ such that:
$$ [x]C = P{B \to C} [x]_B. $$
This matrix $P_{B \to C}$ is the change-of-basis matrix from $B$ to $C$.
In matrix form:
$$ P_{B \to C} = \big[ [b_1]_C \ [b_2]_C \ \dots \ [b_n]_C \big]. $$
Let
To build $P_{B \to C}$:
Solve:
$$ (1,0) = a(1,1) + b(1,-1). $$
This gives system:
$$ a+b=1, \quad a-b=0. $$
Solution: $a=\tfrac{1}{2}, b=\tfrac{1}{2}$. So $(1,0) = \tfrac{1}{2}(1,1) + \tfrac{1}{2}(1,-1)$.
Next:
$$ (0,1) = a(1,1) + b(1,-1). $$
System:
$$ a+b=0, \quad a-b=1. $$
Solution: $a=\tfrac{1}{2}, b=-\tfrac{1}{2}$.
Thus:
$$
P_{B \to C} = \begin{bmatrix}
\tfrac{1}{2} & \tfrac{1}{2} \
\tfrac{1}{2} & -\tfrac{1}{2}
\end{bmatrix}.
$$
So for any vector $x$,
$$ [x]C = P{B \to C}[x]_B. $$
If $P_{B \to C}$ is the change-of-basis matrix from $B$ to $C$, then its inverse is the change-of-basis matrix in the opposite direction:
$$ P_{C \to B} = (P_{B \to C})^{-1}. $$
This makes sense: translating back and forth between languages should undo itself.
Let
$$ P_B = [b_1 \ b_2 \ \dots \ b_n], \quad P_C = [c_1 \ c_2 \ \dots \ c_n], $$
the matrices whose columns are basis vectors written in standard coordinates.
Then the change-of-basis matrix from $B$ to $C$ is:
$$ P_{B \to C} = P_C^{-1} P_B. $$
This formula is extremely useful because it reduces the problem to matrix multiplication.
Change-of-basis matrices give us the precise mechanism for switching perspectives. They ensure that although bases change, vectors remain invariant, and computations remain consistent.
So far, vector spaces and subspaces have always passed through the origin. But in many real-world situations, we deal with shifted versions of these spaces: planes not passing through the origin, lines offset from the zero vector, or solution sets to linear equations with nonzero constants. These structures are called affine subspaces. They extend the idea of subspaces by allowing "translation away from the origin."
An affine subspace of a vector space $V$ is a set of the form
$$ x_0 + W = {x_0 + w : w \in W}, $$
where:
Thus, an affine subspace is simply a subspace shifted by a vector.
Affine subspaces naturally arise as solution sets of linear equations.
Homogeneous system: $Ax = 0$.
Non-homogeneous system: $Ax = b$ with $b \neq 0$.
Solution set is affine.
If $x_p$ is one particular solution, then the general solution is:
$$ x = x_p + N(A), $$
where $N(A)$ is the null space.
Thus, the geometry of solving equations leads naturally to affine subspaces.
The dimension of an affine subspace is defined as the dimension of its direction subspace $W$.
Think of affine subspaces as "flat sheets" floating in space:
Show that the set of solutions to
$$ x+y+z=1 $$
is an affine subspace of $\mathbb{R}^3$. Identify its dimension.
Find the general solution to
$$ x+2y=3 $$
and describe it as an affine subspace.
Prove that the intersection of two affine subspaces is either empty or another affine subspace.
Challenge: Show that every affine subspace can be written uniquely as $x_0 + W$ with $W$ a subspace.
Affine subspaces are the natural setting for most real-world linear problems: they combine the strict structure of subspaces with the freedom of translation, capturing both direction and position.
Each basis a song,
dimension counts melodies,
the space breathes its form.
Maps preserve the line,
reflections ripple outward,
motion kept in frame.
A linear transformation is the heart of linear algebra. It is the rule that connects two vector spaces in a way that respects their linear structure: addition and scalar multiplication. Instead of thinking of vectors as static objects, linear transformations let us study how vectors move, stretch, rotate, project, or reflect. They give linear algebra its dynamic power and are the bridge between abstract theory and concrete applications.
A function $T: V \to W$ between vector spaces is called a linear transformation if for all $u, v \in V$ and scalars $a, b \in \mathbb{R}$ (or another field),
$$ T(au + bv) = aT(u) + bT(v). $$
This single condition encodes two rules:
If both are satisfied, the transformation is linear.
Scaling: $T(x) = 3x$ in $\mathbb{R}$. Every number is stretched threefold.
Rotation in the plane: $T(x,y) = (x\cos\theta - y\sin\theta, , x\sin\theta + y\cos\theta)$.
Projection: Projecting $(x,y,z)$ onto the $xy$-plane: $T(x,y,z) = (x,y,0)$.
Differentiation: On the space of polynomials, $T(p(x)) = p'(x)$.
Integration: On continuous functions, $T(f)(x) = \int_0^x f(t) , dt$.
All these are linear because they preserve addition and scaling.
Nonlinear rules break the structure of vector spaces.
Every linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ can be represented by a matrix.
If $T: \mathbb{R}^n \to \mathbb{R}^m$, then there exists an $m \times n$ matrix $A$ such that:
$$ T(x) = Ax. $$
The columns of $A$ are simply $T(e_1), T(e_2), \dots, T(e_n)$, where $e_i$ are the standard basis vectors.
Example: Let $T(x,y) = (2x+y, x-y)$.
$$ A = \begin{bmatrix} 2 & 1 \ 1 & -1 \end{bmatrix}. $$
Then $T(x,y) = A \begin{bmatrix} x \ y \end{bmatrix}$.
For $T: V \to W$:
$$ \dim(\ker T) + \dim(\text{im}(T)) = \dim(V). $$
Linear transformations reshape space:
The key feature: straight lines remain straight, and the origin stays fixed.
Truncated — view the full README on GitHub.
(top 24 of 34)
12,947 followers · starred Sep 2025
2,437 followers · starred Sep 2025
135 followers · starred Sep 2025
85 followers · starred Sep 2025
A concise, beginner-friendly introduction to the core ideas of linear algebra.
See the codeA concise, beginner-friendly introduction to the core ideas of Linear Algebra.
A clear, hands-on guide to the geometry and structure behind vectors and matrices.
.tex source
For old version (v1), here is the link
.tex sourceWe use Quarto to generate all outputs.
Preview locally:
quarto preview
Render outputs:
# All configured formats
quarto render
# Individual formats
quarto render --to html # site into docs/
quarto render --to pdf # docs/book.pdf
quarto render --to epub # docs/book.epub
quarto render --to latex # docs/book-latex/book.tex
Arrows in the air,
directions whisper softly—
the plane comes alive.
When we begin learning linear algebra, everything starts with the simplest building blocks: scalars and vectors. A scalar is just a single number, like 3, –7, or π. It carries only magnitude and no direction. Scalars are what we use for counting, measuring length, or scaling other objects up and down. A vector, by contrast, is an ordered collection of numbers. You can picture it as an arrow pointing somewhere in space, or simply as a list like (2, 5) in 2D or (1, –3, 4) in 3D. Where scalars measure "how much," vectors measure both "how much" and "which way."
To talk about vectors, we need a coordinate system. Imagine laying down two perpendicular axes on a sheet of paper: the x-axis (left to right) and the y-axis (up and down). Every point on the sheet can be described with two numbers: how far along the x-axis, and how far along the y-axis. This pair of numbers is a vector in 2D. Add a z-axis pointing up from the page, and you have 3D space. Each coordinate system gives us a way to describe vectors numerically, even though the underlying "space" is the same.
Understanding the difference between scalars and vectors is the foundation for everything else in linear algebra. Every concept-matrices, linear transformations, eigenvalues-eventually reduces to how we manipulate vectors and scale them with scalars. Without this distinction, the rest of the subject would have no anchor.
Nearly every field of science and engineering depends on this idea. Physics uses vectors for velocity, acceleration, and force. Computer graphics uses them to represent points, colors, and transformations. Data science treats entire datasets as high-dimensional vectors. By mastering scalars and vectors early, you unlock the language in which modern science and technology are written.
These simple experiments already give you intuition for the operations you'll perform again and again in linear algebra.
Linear algebra gives us powerful ways to describe and manipulate vectors, but before we can do anything with them, we need a precise notation system. Notation is not just cosmetic-it tells us how to read, write, and think about vectors clearly and unambiguously. In this section, we'll explore how vectors are written, how their components are represented, and how we can interpret them visually as arrows.
Vectors are usually denoted by lowercase letters in bold (like $\mathbf{v}, \mathbf{w}, \mathbf{x}$)
or with an arrow overhead (like $\vec{v}$).
For instance, the vector $\mathbf{v} = (2, 5)$ is the same as $\vec{v} = (2, 5)$.
The style depends on context: mathematicians often use bold, physicists often use arrows.
In handwritten notes, people sometimes underline vectors (e.g., $\underline{v}$) to avoid confusion with scalars.
The important thing is to distinguish vectors from scalars at a glance.
A vector in two dimensions has two components, written as $(x, y)$.
In three dimensions, it has three components: $(x, y, z)$.
More generally, an $n$-dimensional vector has $n$ components: $(v_1, v_2, \ldots, v_n)$.
Each component tells us how far the vector extends along one axis of the coordinate system.
For example:
We often refer to the $i$-th component of a vector $\mathbf{v}$ as $v_i$.
So, for $\mathbf{v} = (3, 4, 5)$, we have $v_1 = 3$, $v_2 = 4$, $v_3 = 5$.
Vectors can be written in two common ways:
As a row vector: $(v_1, v_2, v_3)$
As a column vector:
$$ \begin{bmatrix} v_1 \ v_2 \ v_3 \end{bmatrix} $$
Both represent the same abstract object.
Row vectors are convenient for quick writing, while column vectors are essential when we start multiplying by matrices, because the dimensions must align.
The most intuitive way to picture a vector is as an arrow:
For example, the vector (2, 3) in 2D is drawn as an arrow from (0, 0) to (2, 3). The arrow has both direction (where it points) and magnitude (its length). This geometric picture makes abstract algebraic manipulations much easier to grasp.
There are two common interpretations of vectors:
In linear algebra, we often treat vectors as free vectors, because their meaning does not depend on where they are drawn.
Suppose u = (–3, 2).
Clear notation is the backbone of linear algebra. Without it, equations quickly become unreadable, and intuition about direction and size is lost. The way we write vectors determines how easily we can connect the algebra (numbers and symbols) to the geometry (arrows and spaces). This dual perspective-symbolic and visual-is what makes linear algebra powerful and practical.
By practicing both the notation and the arrow picture, you'll develop fluency in switching between abstract symbols and concrete visualizations. This skill will make every later concept in linear algebra far more intuitive.
Once we know how to describe vectors with components and arrows, the next step is to learn how to combine them. Two fundamental operations form the backbone of linear algebra: adding vectors together and scaling vectors with numbers (scalars). These two moves, though simple, generate everything else we'll build later. With them, we can describe motion, forces, data transformations, and more.
Suppose we have two vectors in 2D:
$\mathbf{u} = (u_1, u_2), \quad \mathbf{v} = (v_1, v_2)$.
Their sum is defined as:
$$
\mathbf{u} + \mathbf{v} = (u_1 + v_1, ; u_2 + v_2).
$$
In words, you add corresponding components.
This works in higher dimensions too:
$$ (u_1, u_2, \ldots, u_n) + (v_1, v_2, \ldots, v_n) = (u_1 + v_1, ; u_2 + v_2, ; \ldots, ; u_n + v_n). $$
Example: $$ (2, 3) + (-1, 4) = (2 - 1, ; 3 + 4) = (1, 7). $$
The geometric picture is even more illuminating. If you draw vector u as an arrow, then place the tail of v at the head of u, the arrow from the start of u to the head of v is u + v. This is called the tip-to-tail rule. The parallelogram rule is another visualization: place u and v tail-to-tail, form a parallelogram, and the diagonal is their sum.
Example: u = (3, 1), v = (2, 2). Draw both from the origin. Their sum (5, 3) is exactly the diagonal of the parallelogram they span.
Scalars stretch or shrink vectors.
If $\mathbf{u} = (u_1, u_2, \ldots, u_n)$ and $c$ is a scalar, then:
$$ c \cdot \mathbf{u} = (c \cdot u_1, ; c \cdot u_2, ; \ldots, ; c \cdot u_n). $$
Example:
$$ 2 \cdot (3, 4) = (6, 8). $$
$$ (-1) \cdot (3, 4) = (-3, -4). $$
Multiplying by a positive scalar stretches or compresses the arrow while keeping the direction the same. Multiplying by a negative scalar flips the arrow to point the opposite way.
Imagine the vector (1, 2). Draw it on graph paper: it goes right 1, up 2. Now double it: (2, 4). The arrow points in the same direction but is twice as long. Halve it: (0.5, 1). It's the same direction but shorter. Negate it: (–1, –2). Now the arrow points backward.
This geometric picture explains why we call these numbers "scalars": they scale the vector.
Vector addition and scalar multiplication are not just separate tricks-they combine to form the heart of linear algebra: linear combinations.
A linear combination of vectors $u$ and $v$ is any vector of the form
$a \cdot u + b \cdot v$, where $a$ and $b$ are scalars.
Example:
If $u = (1, 0)$ and $v = (0, 1)$, then
$3 \cdot u + 2 \cdot v = (3, 2)$.
This shows how any point on the grid can be reached by scaling and adding these two basic vectors.
That’s the essence of constructing spaces.
Vector addition and scalar multiplication obey rules that mirror arithmetic with numbers:
These rules are not trivial bookkeeping - they guarantee that linear algebra behaves predictably,
which is why it works as the language of science.
With only these two operations-addition and scaling-you can already describe lines, planes, and entire spaces. Any system that grows by combining influences, like physics, economics, or machine learning, is built on these simple rules. Later, when we define matrix multiplication, dot products, and eigenvalues, they all reduce to repeated patterns of adding and scaling vectors.
This simple practice shows you how combining two basic vectors through addition and scaling generates a whole structured space, the first glimpse of linear algebra's real power.
After learning to add vectors and scale them, the natural next question is: what can we build from these two operations? The answer is the concept of linear combinations, which leads directly to one of the most fundamental ideas in linear algebra: the span of a set of vectors. These ideas tell us not only what individual vectors can do, but how groups of vectors can shape entire spaces.
A linear combination is any vector formed by multiplying vectors with scalars and then adding the results together.
Formally, given vectors $v_1, v_2, \ldots, v_k$ and scalars $a_1, a_2, \ldots, a_k$, a linear combination looks like:
$$ a_1 \cdot v_1 + a_2 \cdot v_2 + \cdots + a_k \cdot v_k. $$
This is nothing more than repeated addition and scaling, but the idea is powerful because it describes how vectors combine to generate new ones.
Example:
Let $u = (1, 0)$ and $v = (0, 1)$. Then any linear combination $a \cdot u + b \cdot v = (a, b)$.
This shows that every point in the 2D plane can be expressed as a linear combination of these two simple vectors.
Linear combinations are about mixing directions and magnitudes. Each vector acts like a "directional ingredient," and the scalars control how much of each ingredient you use.
This progression shows that the power of linear combinations depends not just on the vectors themselves but on how they relate to each other.
The span of a set of vectors is the collection of all possible linear combinations of them.
It answers the question: "What space do these vectors generate?"
Notation:
$$ \text{Span}{v_1, v_2, \ldots, v_k} = {a_1 v_1 + a_2 v_2 + \cdots + a_k v_k ;|; a_i \in \mathbb{R}}. $$
Examples:
So the span depends heavily on whether the vectors add new directions or just duplicate what’s already there.
If vectors point in the same or opposite directions (one is a scalar multiple of another), then their span is just a line. They don't add any new coverage of space. But if they point in different directions, they open up new dimensions. This leads to the critical idea of linear independence, which we'll explore later: vectors are independent if none of them is a linear combination of the others.
In 2D:
In 3D:
In higher dimensions: The same pattern continues. A set of k independent vectors spans a k-dimensional subspace inside the larger space.
Linear combinations and span are the foundation for almost everything else in linear algebra:
In short, the span tells us the "reach" of a set of vectors, and linear combinations are the mechanism to explore that reach.
By experimenting with simple examples, you'll see clearly how the idea of span captures the richness or limitations of combining vectors.
So far, vectors have been arrows with direction and components. To compare them more meaningfully, we need ways to talk about how long they are and how far apart they are. These notions are formalized through the norm of a vector (its length) and the distance between vectors. These concepts tie together the algebra of components and the geometry of space.
The norm of a vector measures its magnitude, or how long the arrow is.
For a vector $v = (v_1, v_2, \ldots, v_n)$ in $n$-dimensional space, its norm is defined as:
$$ |v| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}. $$
This formula comes directly from the Pythagorean theorem: the length of the hypotenuse equals the square root of the sum of squares of the legs.
In 2D, this is the familiar distance formula between the origin and a point.
Examples:
For $v = (3, 4)$:
$$
|v| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = 5.
$$
For $w = (1, -2, 2)$:
$$
|w| = \sqrt{1^2 + (-2)^2 + 2^2} = \sqrt{1 + 4 + 4} = \sqrt{9} = 3.
$$
A unit vector is a vector whose length is exactly 1.
These are important because they capture direction without scaling.
To create a unit vector from any nonzero vector, divide by its norm:
$$ u = \frac{v}{|v|}. $$
Example:
For $v = (3, 4)$, the unit vector is
$$ u = \left(\tfrac{3}{5}, \tfrac{4}{5}\right). $$
This points in the same direction as $(3, 4)$ but has length 1.
Unit vectors are like pure directions.
They're especially useful for projections, defining coordinate systems, and normalizing data.
The distance between two vectors $u$ and $v$ is defined as the length of their difference:
$$ \text{dist}(u, v) = |u - v|. $$
Example:
Let $u = (2, 1)$ and $v = (5, 5)$. Then
$$ u - v = (-3, -4). $$
Its norm is
$$ \sqrt{(-3)^2 + (-4)^2} = \sqrt{9 + 16} = 5. $$
So the distance is 5. This matches our intuition: the straight-line distance between points $(2, 1)$ and $(5, 5)$.
Both are computed with the same formula-the square root of sums of squares-but applied in slightly different contexts.
The formula above defines the Euclidean norm (or $\ell_2$ norm), the most common one.
But in linear algebra, other norms are also useful:
$\ell_1$ norm:
$$
|v|_1 = |v_1| + |v_2| + \cdots + |v_n|
$$
(sum of absolute values).
$\ell_\infty$ norm:
$$
|v|_\infty = \max(|v_1|, |v_2|, \ldots, |v_n|)
$$
(largest component).
These norms change the geometry of "length" and "distance." For example, in the ℓ₁ norm, the unit circle is shaped like a diamond; in the ℓ∞ norm, it looks like a square.
Norms and distances satisfy critical properties that make them consistent measures:
These properties are why norms and distances are robust tools across mathematics.
Understanding length and distance is the first step toward geometry in higher dimensions. These notions:
By working through these examples, you'll see how norms and distances make abstract vectors feel as real as points and arrows you can measure in everyday life.
The dot product is one of the most fundamental operations in linear algebra. It looks like a simple formula, but it unlocks the ability to measure angles, detect orthogonality, project one vector onto another, and compute energy or work in physics. Understanding it requires seeing both the algebraic view (a formula on components) and the geometric view (a way to compare directions).
For two vectors of the same dimension, $u = (u_1, u_2, \ldots, u_n)$ and $v = (v_1, v_2, \ldots, v_n)$, the dot product is defined as:
$$ u \cdot v = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n. $$
This is simply multiplying corresponding components and summing the results.
Examples:
Notice that the dot product is always a scalar, not a vector.
The dot product can also be defined in terms of vector length and angle:
$$ u \cdot v = |u| , |v| \cos(\theta), $$
where $\theta$ is the angle between $u$ and $v$ ($0^\circ \leq \theta \leq 180^\circ$).
This formula tells us:
Thus, the sign of the dot product encodes directional alignment.
At first glance, the algebraic sum of products and the geometric length–angle formula seem unrelated. But they are equivalent. To see why, consider the law of cosines applied to a triangle formed by u, v, and u – v. Expanding both sides leads directly to the equivalence between the two formulas. This dual interpretation is what makes the dot product so powerful: it is both a computation rule and a geometric measurement.
Two vectors are orthogonal (perpendicular) if and only if their dot product is zero:
$$ u \cdot v = 0 ;;\Longleftrightarrow;; \theta = 90^\circ. $$
This gives us an algebraic way to check for perpendicularity without drawing diagrams.
Example:
$(2, 1) \cdot (-1, 2) = (2 \times -1) + (1 \times 2) = -2 + 2 = 0$,
so the vectors are orthogonal.
The dot product also provides a way to project one vector onto another.
The scalar projection of $u$ onto $v$ is:
$$ \text{proj}_{\text{scalar}}(u \text{ onto } v) = \frac{u \cdot v}{|v|}. $$
The vector projection is then:
$$ \text{proj}_{\text{vector}}(u \text{ onto } v) = \frac{u \cdot v}{|v|^2} , v. $$
This allows us to decompose vectors into "parallel" and "perpendicular" components, which is central in geometry, physics, and data analysis.
Compute $u = (3, 4)$ and $v = (4, 3)$.
Compute $u = (1, 2, -1)$ and $v = (2, -1, 1)$.
In physics, the dot product computes work:
$$ \text{Work} = \text{Force} \cdot \text{Displacement} = |\text{Force}| , |\text{Displacement}| \cos(\theta). $$
Only the component of the force in the direction of motion contributes. If you push straight down on a box while trying to move it horizontally, the dot product is zero: no work is done in the direction of motion.
These guarantee that the dot product behaves consistently and meshes with the structure of vector spaces.
The dot product is the first bridge between algebra and geometry. It:
Without the dot product, linear algebra would lack a way to connect numbers with geometry and meaning.
These exercises reveal the dual power of the dot product: as a formula to compute and as a geometric tool to interpret.
Having defined the dot product, we are now ready to measure angles between vectors. In everyday life, angles tell us how two lines or directions relate-whether they point the same way, are perpendicular, or are opposed. In linear algebra, the dot product and cosine function give us a precise, generalizable way to define angles in any dimension, not just in 2D or 3D. This section explores how we compute, interpret, and apply vector angles.
For two nonzero vectors $u$ and $v$, the angle $\theta$ between them is defined by:
$$ \cos(\theta) = \frac{u \cdot v}{|u| , |v|}. $$
This formula comes directly from the geometric definition of the dot product.
Rearranging gives:
$$ \theta = \arccos!\left(\frac{u \cdot v}{|u| , |v|}\right). $$
Key points:
The cosine tells us about the directional relationship:
Thus, the cosine compresses geometric alignment into a single number.
$u = (1, 0), ; v = (0, 1)$
$u = (2, 3), ; v = (4, 6)$
$u = (1, 1), ; v = (-1, 1)$
The beauty of the formula is that it works in any dimension.
Even in $\mathbb{R}^{100}$ or higher, we can define the angle between two vectors using only their dot product and norms.
While we cannot visualize the geometry directly in high dimensions, the cosine formula still captures how aligned two directions are:
$$ \cos(\theta) = \frac{u \cdot v}{|u| , |v|}. $$
This is critical in machine learning, where data often lives in very high-dimensional spaces.
The cosine of the angle between two vectors is often called cosine similarity. It is widely used in data analysis and machine learning to measure how similar two data vectors are, independent of their magnitude.
This demonstrates how a geometric concept extends far beyond pure math.
The angle formula reinforces the special role of orthogonality.
If $\cos(\theta) = 0$, then $u \cdot v = 0$.
This means the dot product not only computes length but also serves as a direct test for perpendicularity.
This algebraic shortcut is far easier than manually checking geometric right angles.
Angles are closely tied to projections.
The length of the projection of $u$ onto $v$ is $|u|\cos(\theta)$.
If the angle is small, the projection is large — most of $u$ lies in the direction of $v$.
If the angle is close to $90^\circ$, the projection shrinks toward zero.
Thus, the cosine acts as a scaling factor between directions.
Teamwork analogy: If two people push a heavy object, the effectiveness of their combined effort depends on the angle.
If they push in nearly the same direction (small $\theta$), they cooperate efficiently.
If they push at right angles ($\theta = 90^\circ$), they waste effort.
If they push opposite each other ($\theta \approx 180^\circ$), they cancel out.
Conversation analogy: If two people's opinions align ($\cos \theta \approx 1$), they agree strongly.
If they are orthogonal ($\cos \theta = 0$), they are unrelated.
If they oppose each other ($\cos \theta \approx -1$), they fundamentally disagree.
Navigation analogy: If two roads meet at a small angle, their directions are similar.
If they cross perpendicularly, their directions are independent.
Angles between vectors provide:
Without the ability to measure angles, we could not connect algebraic manipulations with geometric intuition or practical applications.
By experimenting with these problems, you will see how angles provide the missing link between algebraic formulas and geometric meaning in linear algebra.
In earlier sections, we saw how the dot product measures alignment and how the cosine formula gives us angles between vectors. The next natural step is to use these tools to project one vector onto another. Projection is a way to "shadow" one vector onto the direction of another, splitting vectors into meaningful parts: one along a given direction and one perpendicular to it. This is the essence of decomposition, and it is everywhere in linear algebra, geometry, physics, and data science.
The scalar projection of a vector $u$ onto a vector $v$ measures how much of $u$ lies in the direction of $v$. It is given by:
$$ \text{proj}_{\text{scalar}}(u \text{ onto } v) = \frac{u \cdot v}{|v|}. $$
Example:
$u = (3, 4)$, $v = (1, 0)$.
Dot product: $(3 \times 1 + 4 \times 0) = 3$.
$|v| = 1$.
So the scalar projection is $3$. This tells us $u$ has a "shadow" of length $3$ on the $x$-axis.
The vector projection gives the actual arrow in the direction of $v$ that corresponds to this scalar amount:
$$ \text{proj}_{\text{vector}}(u \text{ onto } v) = \frac{u \cdot v}{|v|^2} , v. $$
This formula normalizes $v$ into a unit vector, then scales it by the scalar projection.
The result is a new vector lying along $v$, capturing exactly the "parallel" part of $u$.
Example:
$u = (3, 4)$, $v = (1, 2)$
So the part of $(3, 4)$ in the direction of $(1, 2)$ is $(2.2, 4.4)$.
Once we have the projection, we can find the perpendicular component (often called the rejection) simply by subtracting:
$$ u_{\perp} = u - \text{proj}_{\text{vector}}(u \text{ onto } v). $$
This gives the part of $u$ that is entirely orthogonal to $v$.
Example continued:
$u_{\perp} = (3, 4) - (2.2, 4.4) = (0.8, -0.4)$
Check:
$(0.8, -0.4) \cdot (1, 2) = 0.8 \times 1 + (-0.4) \times 2 = 0.8 - 0.8 = 0$.
Indeed, orthogonal.
Projection is like dropping a perpendicular from one vector onto another. Imagine shining a light perpendicular to v: the shadow of u on the line spanned by v is the projection. This visualization explains why projections split vectors naturally into two pieces:
Together, these two parts reconstruct the original vector exactly.
Every vector $u$ can be decomposed relative to another vector $v$ into two parts:
$$ u = \text{proj}{\text{vector}}(u \text{ onto } v) + \big(u - \text{proj}{\text{vector}}(u \text{ onto } v)\big). $$
This decomposition is unique and geometrically meaningful.
It generalizes to subspaces: we can project onto entire planes or higher-dimensional spans, splitting a vector into a "within-subspace" part and a "perpendicular-to-subspace" part.
Physics (Work and Forces): Work is the projection of force onto displacement. Only the part of the force in the direction of motion contributes. Example: Pushing on a sled partly sideways wastes effort-the sideways component projects to zero.
Geometry and Engineering: Projections are used in CAD (computer-aided design) to flatten 3D objects onto 2D surfaces, like blueprints or shadows.
Computer Graphics: Rendering 3D scenes onto a 2D screen is fundamentally a projection process.
Data Science: Projecting high-dimensional data onto a lower-dimensional subspace (like the first two principal components in PCA) makes patterns visible while preserving as much information as possible.
Signal Processing: Decomposition into projections onto sine and cosine waves forms the basis of Fourier analysis, which powers audio, image, and video compression.
Projection is not just a geometric trick; it is the core of many advanced topics:
Without projection, we could not cleanly separate influence along directions or reduce complexity in structured ways.
Through these exercises, you will see that projection is more than an operation-it is a lens through which we decompose, interpret, and simplify vectors and spaces.
Linear algebra is not only about operations with vectors-it also involves understanding the fundamental relationships between them. Two of the most important results in this regard are the Cauchy–Schwarz inequality and the triangle inequality. These are cornerstones of vector spaces because they establish precise boundaries for lengths, angles, and inner products. Without them, the geometry of linear algebra would fall apart.
For any two vectors $u$ and $v$ in $\mathbb{R}^n$, the Cauchy–Schwarz inequality states:
$$ |u \cdot v| \leq |u| , |v|. $$
This means that the absolute value of the dot product of two vectors is always less than or equal to the product of their lengths.
Equality holds if and only if u and v are linearly dependent (i.e., one is a scalar multiple of the other).
Recall the geometric formula for the dot product:
$$ u \cdot v = |u| , |v| \cos(\theta). $$
Since $-1 \leq \cos(\theta) \leq 1$, the magnitude of the dot product cannot exceed $|u| , |v|$.
This is exactly the inequality.
Let $u = (3, 4)$ and $v = (-4, 3)$.
Equality does not hold since they are not multiples - they are perpendicular.
The inequality tells us that two vectors can never "overlap" more strongly than the product of their magnitudes. If they align perfectly, the overlap is maximum (equality). If they're perpendicular, the overlap is zero.
Think of it as: "the shadow of one vector on another can never be longer than the vector itself."
For any vectors $u$ and $v$, the triangle inequality states:
$$ |u + v| \leq |u| + |v|. $$
This mirrors the geometric fact that in a triangle, any side is at most as long as the sum of the other two sides.
Let $u = (1, 2)$ and $v = (3, 4)$.
Indeed, $7.21 \leq 7.24$, very close in this case.
The triangle inequality becomes equality when the vectors point in exactly the same direction (or are scalar multiples with nonnegative coefficients). For example, (1, 1) and (2, 2) produce equality because adding them gives a vector whose length equals the sum of their lengths.
Without these inequalities, we could not trust that the geometry of vector spaces behaves consistently.
Working through these problems will show you why these inequalities are not abstract curiosities but the structural glue of linear algebra's geometry.
Up to now, we've discussed vectors, their lengths, angles, and how to project one onto another. A natural culmination of these ideas is the concept of orthonormal sets. These are collections of vectors that are not only orthogonal (mutually perpendicular) but also normalized (each of length 1). Orthonormal sets form the cleanest, most efficient coordinate systems in linear algebra. They are the mathematical equivalent of having rulers at right angles, perfectly calibrated to unit length.
Let's break the term "orthonormal" into two parts:
Orthogonal: Two vectors $u$ and $v$ are orthogonal if $u \cdot v = 0$.
In $\mathbb{R}^2$, this means the vectors meet at a right angle.
In $\mathbb{R}^3$, it means they form perpendicular directions.
Normalized: A vector $v$ is normalized if its length is $1$, i.e., $|v| = 1$.
Such vectors are called unit vectors.
When we combine both conditions, we get orthonormal vectors: vectors that are both perpendicular to each other and have unit length.
In two dimensions, an orthonormal set typically consists of two vectors.
A classic example is:
$e_1 = (1, 0), \quad e_2 = (0, 1)$
Thus, ${e_1, e_2}$ is an orthonormal set.
In fact, this is the standard basis for $\mathbb{R}^2$.
Any vector $(x, y)$ can be written as $x e_1 + y e_2$.
This is the simplest coordinate system.
In three dimensions, an orthonormal set usually has three vectors.
The standard basis is:
$e_1 = (1, 0, 0), \quad e_2 = (0, 1, 0), \quad e_3 = (0, 0, 1)$
Geometrically, they correspond to the $x$-, $y$-, and $z$-axes in 3D space.
Any vector $(x, y, z)$ can be written as a linear combination $x e_1 + y e_2 + z e_3$.
The standard basis is not the only orthonormal set. For example:
$u = \left(\tfrac{1}{\sqrt{2}}, \tfrac{1}{\sqrt{2}}\right), \quad v = \left(-\tfrac{1}{\sqrt{2}}, \tfrac{1}{\sqrt{2}}\right)$
So ${u, v}$ is also orthonormal in $\mathbb{R}^2$.
These vectors are rotated $45^\circ$ relative to the standard axes.
Similarly, in $\mathbb{R}^3$, you can construct rotated orthonormal sets (such as unit vectors along diagonals), as long as the conditions of perpendicularity and unit length hold.
Simplified coordinates: If ${v_1, \ldots, v_k}$ is an orthonormal set, then for any vector $u$ in their span, the coefficients are easy to compute:
$$
c_i = u \cdot v_i
$$
This is much simpler than solving systems of equations.
Pythagorean theorem generalized: If vectors are orthonormal, the squared length of their sum is the sum of the squares of their coefficients.
For example, if $u = a v_1 + b v_2$, then
$$
|u|^2 = a^2 + b^2
$$
Projection is easy: Projecting onto an orthonormal set is straightforward — just take dot products.
Matrices become nice: When vectors form the columns of a matrix, orthonormality makes that matrix an orthogonal matrix, which has special properties: its transpose equals its inverse, and it preserves lengths and angles.
Even though $\mathbb{R}^2$ and $\mathbb{R}^3$ are relatively simple, the same ideas extend naturally to higher dimensions, where visualization is impossible but the algebra is identical.
Orthonormality is the gold standard for building bases in linear algebra:
Whenever possible, mathematicians and engineers prefer orthonormal bases over arbitrary ones.
Through these exercises, you will see how orthonormal sets make every aspect of linear algebra-from projections to decompositions-simpler, cleaner, and more powerful.
Lengths, angles revealed,
projections trace hidden lines,
clarity takes shape.
Rows and columns meet,
woven grids of silent rules,
machines of order.
The next stage in our journey is to move from vectors to matrices. A matrix may look like just a rectangular array of numbers, but in linear algebra it plays two distinct and equally important roles:
Both views are valid, and learning to switch between them is crucial to building intuition.
At the most basic level, a matrix is a grid of numbers arranged into rows and columns.
A $2 \times 2$ matrix has 2 rows and 2 columns:
$$ A = \begin{bmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{bmatrix} $$
A $3 \times 2$ matrix has 3 rows and 2 columns:
$$ B = \begin{bmatrix} b_{11} & b_{12} \ b_{21} & b_{22} \ b_{31} & b_{32} \end{bmatrix} $$
Each entry $a_{ij}$ or $b_{ij}$ tells us the number in the i-th row and j-th column. The rows of a matrix can represent constraints, equations, or observations; the columns can represent features, variables, or directions.
In this sense, matrices are data containers, organizing information efficiently. That's why matrices show up in spreadsheets, statistics, computer graphics, and scientific computing.
The deeper view of a matrix is as a function from vectors to vectors. If x is a column vector, then multiplying A·x produces a new vector.
For example:
$$ A = \begin{bmatrix} 2 & 0 \ 1 & 3 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 4 \ 5 \end{bmatrix}. $$
Multiplying:
$$ A\mathbf{x} = \begin{bmatrix} 2×4 + 0×5 \ 1×4 + 3×5 \end{bmatrix} = \begin{bmatrix} 8 \ 19 \end{bmatrix}. $$
Here, the matrix is acting as a machine that takes input (4, 5) and outputs (8, 19). The "machine rules" are encoded in the rows of A.
Another way to see it: multiplying A·x is the same as taking a linear combination of A's columns.
If
$$ A = \begin{bmatrix} a_1 & a_2 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \end{bmatrix}, $$
then:
$$ A\mathbf{x} = x_1 a_1 + x_2 a_2. $$
So the vector x tells the machine "how much" of each column to mix together. This column view is critical-it connects matrices to span, dimension, and basis ideas we saw earlier.
This duality is not just conceptual-it's the key to understanding why linear algebra is so powerful. A dataset, once stored as a table, can be interpreted as a transformation. Likewise, a transformation, once understood, can be encoded as a table.
Every 2×2 or 3×3 matrix corresponds to some linear transformation in the plane or space. Examples:
These are not just tables of numbers-they are precise, reusable machines.
This section sets the stage for all matrix theory:
By mastering both perspectives, you'll see matrices not just as numbers but as dynamic objects that encode and execute transformations.
Matrices come in many shapes and sizes, and the way we label their entries matters. This section is about learning how to read and write matrices carefully, how to work with rows and columns, and how to use block structure to simplify problems. These seemingly simple ideas are what allow us to manipulate large systems with precision and efficiency.
The shape of a matrix is given by its number of rows and columns:
Examples:
A 2×3 matrix:
$$ \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix} $$
A 3×2 matrix:
$$ \begin{bmatrix} 7 & 8 \ 9 & 10 \ 11 & 12 \end{bmatrix} $$
Shape matters because it determines whether certain operations (like multiplication) are possible.
Each entry in a matrix has two indices: one for its row, one for its column.
For example, in
$$ A = \begin{bmatrix} 1 & 4 & 7 \ 2 & 5 & 8 \ 3 & 6 & 9 \end{bmatrix}, $$
we have:
Indexing is the grammar of matrix language. Without it, we can't specify positions or write formulas clearly.
Every row and every column of a matrix is itself a vector.
Example: From the matrix above,
This duality is powerful: rows often represent constraints or equations, while columns represent directions or features. Later, when we interpret matrix–vector products, we'll see that multiplying A·x means combining columns, while multiplying yᵀ·A means combining rows.
Sometimes we want just part of a matrix. A submatrix is formed by selecting certain rows and columns.
Example: From
$$ B = \begin{bmatrix} 2 & 4 & 6 \ 1 & 3 & 5 \ 7 & 8 & 9 \end{bmatrix}, $$
the submatrix of the first two rows and last two columns is:
$$ \begin{bmatrix} 4 & 6 \ 3 & 5 \end{bmatrix}. $$
Submatrices allow us to zoom in and isolate parts of a problem.
Large matrices can often be broken into blocks, which are smaller submatrices arranged inside. This is like dividing a spreadsheet into quadrants.
For example:
$$ C = \begin{bmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{bmatrix}, $$
where each $A_{ij}$ is itself a smaller matrix.
This structure is useful in:
Example: Splitting a 4×4 matrix into four 2×2 blocks helps us treat it as a "matrix of matrices."
Some shapes of matrices are so common they deserve names:
Recognizing these shapes saves time and clarifies reasoning.
Careful attention to matrix shapes, indexing, and block views ensures:
These basic tools prepare us for multiplication, transformations, and factorization.
Write a 3×4 matrix and label the entry in row 2, column 3.
Extract a 2×2 submatrix from the corners of a 4×4 matrix of your choice.
Break a 6×6 matrix into four 3×3 blocks. How would you represent it compactly?
Challenge: Given
$$ D = \begin{bmatrix} 1 & 2 & 3 & 4 \ 5 & 6 & 7 & 8 \ 9 & 10 & 11 & 12 \end{bmatrix}, $$
write it as a block matrix with a 2×2 block in the top-left, a 2×2 block in the top-right, and a 1×4 block in the bottom row.
By practicing with shapes, indexing, and blocks, you'll develop the ability to navigate matrices not just as raw grids of numbers but as structured objects ready for deeper algebraic and geometric insights.
Before exploring matrix–vector and matrix–matrix multiplication, it is essential to understand the simplest operations we can perform with matrices: addition and scalar multiplication. These operations extend the rules we learned for vectors, but now applied to entire grids of numbers. Although straightforward, they are the foundation for more complex algebraic manipulations and help establish the idea of matrices as elements of a vector space.
If two matrices $A$ and $B$ have the same shape (same number of rows and columns), we can add them by adding corresponding entries.
Formally: If
$$ A = [a_{ij}], \quad B = [b_{ij}], $$
then
$$ A + B = [a_{ij} + b_{ij}]. $$
Example:
\begin{bmatrix} 8 & 10 & 12 \ 14 & 16 & 18 \end{bmatrix}. $$
Key point: Addition is only defined if the matrices are the same shape. A 2×3 matrix cannot be added to a 3×2 matrix.
A scalar multiplies every entry of a matrix.
Formally: For scalar $c$ and matrix $A = [a_{ij}]$,
$$ cA = [c \cdot a_{ij}]. $$
Example:
\begin{bmatrix} 6 & -3 \ 0 & 12 \end{bmatrix}. $$
This mirrors vector scaling: stretching or shrinking the whole matrix by a constant factor.
These two operations satisfy familiar algebraic properties that make the set of all m×n matrices into a vector space:
These guarantee that working with matrices feels like working with numbers and vectors, only in a higher-level setting.
From the table view, addition and scalar multiplication are just simple bookkeeping: line up two tables of the same shape and add entry by entry; multiply the whole table by a constant.
Example: Imagine two spreadsheets of monthly expenses. Adding them gives combined totals. Multiplying by 12 converts a monthly table into a yearly estimate.
From the machine view, these operations adjust the behavior of linear transformations:
Example: Let $A$ rotate vectors slightly, and $B$ stretch vectors. The matrix $A + B$ represents a transformation that applies both influences together. Scaling by 2 doubles the effect of the transformation.
These act as neutral or scaling elements in matrix arithmetic.
This shows that even before multiplication, addition and scaling already have geometric meaning.
Though simple, these operations:
Without addition and scalar multiplication, we could not treat matrices systematically as algebraic objects.
$$ \begin{bmatrix} 2 & 0 \ 1 & 3 \end{bmatrix} \quad \text{and} \quad \begin{bmatrix} -2 & 5 \ 4 & -3 \end{bmatrix}. $$
$$ \begin{bmatrix} 1 & -1 & 2 \ 0 & 3 & 4 \end{bmatrix} $$
by –2.
Show that (A + B) + C = A + (B + C) with explicit 2×2 matrices.
Challenge: Construct two 3×3 matrices A and B such that A + B = 0. What does that tell you about B?
By practicing these fundamentals, you will see that even the most basic operations on matrices already build the algebraic backbone for deeper results like matrix multiplication, transformations, and factorization.
We now arrive at one of the most important operations in all of linear algebra: the matrix–vector product. This operation takes a matrix $A$ and a vector x, and produces a new vector. While the computation is straightforward, its interpretations are deep: it can be seen as combining rows, as combining columns, or as applying a linear transformation. This is the operation that connects matrices to the geometry of vector spaces.
Suppose $A$ is an $m \times n$ matrix, and x is a vector in $\mathbb{R}^n$. The product $A\mathbf{x}$ is a vector in $\mathbb{R}^m$, defined as:
$$ A = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \ a_{21} & a_{22} & \cdots & a_{2n} \ \vdots & \vdots & \ddots & \vdots \ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \ \vdots \ x_n \end{bmatrix}. $$
Then:
$$ A\mathbf{x} = \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n \ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n \ \vdots \ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n \end{bmatrix}. $$
Each entry of the output is a dot product between one row of $A$ and the vector x.
From the row perspective, $A\mathbf{x}$ is computed row by row:
Example:
$$ A = \begin{bmatrix} 2 & 1 \ 3 & 4 \ -1 & 2 \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} 5 \ -1 \end{bmatrix}. $$
So:
$$ A\mathbf{x} = \begin{bmatrix} 9 \ 11 \ -7 \end{bmatrix}. $$
From the column perspective, $A\mathbf{x}$ is a linear combination of the columns of A.
If
$$ A = \begin{bmatrix} | & | & & | \ a_1 & a_2 & \cdots & a_n \ | & | & & | \end{bmatrix}, \quad \mathbf{x} = \begin{bmatrix} x_1 \ x_2 \ \vdots \ x_n \end{bmatrix}, $$
then:
$$ A\mathbf{x} = x_1 a_1 + x_2 a_2 + \cdots + x_n a_n. $$
That is: multiply each column of $A$ by the corresponding entry in x, then add them up.
This interpretation connects directly to the idea of span: the set of all vectors $A\mathbf{x}$ as x varies is exactly the span of the columns of $A$.
The machine view ties everything together: multiplying a vector by a matrix means applying the linear transformation represented by the matrix.
Thus, the algebraic definition encodes geometric and functional meaning.
$$ A = \begin{bmatrix} 2 & 0 \ 0 & 2 \end{bmatrix}. $$
Then $A\mathbf{x}$ doubles the length of any vector x.
$$ A = \begin{bmatrix} 1 & 0 \ 0 & -1 \end{bmatrix}. $$
This flips vectors across the x-axis.
$$ A = \begin{bmatrix} \cosθ & -\sinθ \ \sinθ & \cosθ \end{bmatrix}. $$
This rotates vectors counterclockwise by θ in the plane.
The matrix–vector product is the building block of everything in linear algebra:
$$ \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix} \begin{bmatrix} 2 \ 0 \ 1 \end{bmatrix}. $$
Express the result of the above product as a linear combination of the columns of the matrix.
Construct a 2×2 matrix that reflects vectors across the line $y = x$. Test it on (1, 0) and (0, 1).
Challenge: For a 3×3 matrix, show that the set of all possible $A\mathbf{x}$ (as x varies) is exactly the column space of $A$.
By mastering both the computational rules and the interpretations of the matrix–vector product, you will gain the most important insight in linear algebra: matrices are not just tables-they are engines that transform space.
Having understood how a matrix acts on a vector, the next natural step is to understand how one matrix can act on another. This leads us to the matrix–matrix product, a rule for combining two matrices into a single new matrix. Though the arithmetic looks complicated at first, the underlying idea is elegant: multiplying two matrices represents composing two linear transformations.
Suppose $A$ is an $m \times n$ matrix and $B$ is an $n \times p$ matrix. Their product $C = AB$ is an $m \times p$ matrix defined by:
$$ c_{ij} = \sum_{k=1}^n a_{ik} b_{kj}. $$
That is: each entry of $C$ is the dot product of the i-th row of $A$ with the j-th column of $B$.
$$ A = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix}, \quad B = \begin{bmatrix} 7 & 8 \ 9 & 10 \ 11 & 12 \end{bmatrix}. $$
Product: $C = AB$ will be 2×2.
So:
$$ C = \begin{bmatrix} 58 & 64 \ 139 & 154 \end{bmatrix}. $$
From the column perspective, $AB$ is computed by applying $A$ to each column of $B$.
If $B = [b_1 ; b_2 ; \cdots ; b_p]$, then:
$$ AB = [A b_1 ; A b_2 ; \cdots ; A b_p]. $$
That is: multiply $A$ by each column of $B$. This is often the simplest way to think of the product.
From the row perspective, each row of $AB$ is formed by combining rows of $B$ using coefficients from a row of $A$. This dual view is less common but equally useful, especially in proofs and algorithms.
The most important interpretation is the machine view: multiplying matrices corresponds to composing transformations.
Example:
$$ A = \begin{bmatrix} 2 & 0 \ 0 & 2 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & -1 \ 1 & 0 \end{bmatrix}. $$
Then $AB$ scales vectors by 2 after rotating them 90°.
These properties highlight that while multiplication is structured, it is not symmetric. The order encodes the order of operations in transformations.
Matrix multiplication is the core of linear algebra because:
Almost every application of linear algebra-solving equations, computing eigenvalues, training neural networks-relies on efficient matrix multiplication.
$$ \begin{bmatrix} 1 & 0 \ 2 & 3 \end{bmatrix} \begin{bmatrix} 4 & 5 \ 6 & 7 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 0 & 1 \ 0 & 0 \end{bmatrix}, \quad B = \begin{bmatrix} 0 & 0 \ 1 & 0 \end{bmatrix}. $$
Construct two 2×2 matrices where $AB = BA$. Why does commutativity happen here?
Challenge: If $A$ is a projection and $B$ is a rotation, compute $AB$ and $BA$. Do they represent the same geometric operation?
Through these perspectives, the matrix–matrix product shifts from being a mechanical formula to being a language for combining linear steps-each product telling the story of "do this, then that."
With addition, scalar multiplication, and matrix multiplication in place, we now introduce three special operations and objects that form the backbone of matrix algebra: the identity matrix, the inverse of a matrix, and the transpose of a matrix. Each captures a fundamental principle-neutrality, reversibility, and symmetry-and together they provide the algebraic structure that makes linear algebra so powerful.
The identity matrix is the matrix equivalent of the number 1 in multiplication.
Example (3×3):
$$ I_3 = \begin{bmatrix} 1 & 0 & 0 \ 0 & 1 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
Property: For any $n \times n$ matrix $A$,
$$ AI_n = I_nA = A. $$
Machine view: $I$ does nothing-it maps every vector to itself.
The inverse is the matrix equivalent of the reciprocal of a number.
Definition: For a square matrix $A$, its inverse $A^{-1}$ is the matrix such that
$$ AA^{-1} = A^{-1}A = I. $$
Not all matrices have inverses. A matrix is invertible if and only if it is square and its determinant is nonzero.
Example:
$$ A = \begin{bmatrix} 2 & 1 \ 1 & 1 \end{bmatrix}, \quad A^{-1} = \begin{bmatrix} 1 & -1 \ -1 & 2 \end{bmatrix}. $$
Check:
\begin{bmatrix} 1 & 0 \ 0 & 1 \end{bmatrix} = I. $$
Some matrices cannot be inverted. These are called singular.
Example:
$$ B = \begin{bmatrix} 2 & 4 \ 1 & 2 \end{bmatrix}. $$
Here, the second column is a multiple of the first. The transformation squashes vectors into a line, losing information-so it cannot be reversed.
This ties invertibility to geometry: a transformation that collapses dimensions cannot be undone.
The transpose reflects a matrix across its diagonal.
Definition: For $A = [a_{ij}]$,
$$ A^T = [a_{ji}]. $$
In words: rows become columns, columns become rows.
Example:
$$ A = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix}, \quad A^T = \begin{bmatrix} 1 & 4 \ 2 & 5 \ 3 & 6 \end{bmatrix}. $$
Properties:
Two important classes emerge from the transpose:
Symmetric matrices: $A = A^T$. Example:
$$ \begin{bmatrix} 2 & 3 \ 3 & 5 \end{bmatrix}. $$
These have beautiful properties: real eigenvalues and orthogonal eigenvectors.
Orthogonal matrices: $Q^TQ = I$. Their columns form an orthonormal set, and they represent pure rotations/reflections.
Without these tools, matrix algebra would lack structure and reversibility.
$$ \begin{bmatrix} 1 & 0 & 2 \ -3 & 4 & 5 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 1 & 2 \ 0 & 3 \end{bmatrix}, \quad B = \begin{bmatrix} 4 & 0 \ 5 & 6 \end{bmatrix}. $$
$$ \begin{bmatrix} 3 & 2 \ 1 & 1 \end{bmatrix}. $$
Through these exercises, you'll see how identity, inverse, and transpose anchor the structure of linear algebra, providing neutrality, reversibility, and symmetry in every calculation.
Not all matrices are created equal-some have special shapes or patterns that give them unique properties. These structured matrices are the workhorses of linear algebra: they simplify computation, reveal geometry, and form the building blocks for algorithms. In this section, we study four especially important classes: symmetric, diagonal, triangular, and permutation matrices.
A matrix is symmetric if it equals its transpose:
$$ A = A^T. $$
Example:
$$ \begin{bmatrix} 2 & 3 & 4 \ 3 & 5 & 6 \ 4 & 6 & 9 \end{bmatrix}. $$
A matrix is diagonal if all non-diagonal entries are zero.
$$ D = \begin{bmatrix} d_1 & 0 & 0 \ 0 & d_2 & 0 \ 0 & 0 & d_3 \end{bmatrix}. $$
Multiplying by $D$ scales each coordinate separately.
Computations with diagonals are lightning fast:
Example:
\begin{bmatrix} 2x \ 3y \end{bmatrix}. $$
This is why diagonalization is so valuable: turning a general matrix into a diagonal one simplifies everything.
A matrix is upper triangular if all entries below the main diagonal are zero, and lower triangular if all entries above the diagonal are zero.
Upper triangular example:
$$ \begin{bmatrix} 1 & 2 & 3 \ 0 & 4 & 5 \ 0 & 0 & 6 \end{bmatrix}. $$
Lower triangular example:
$$ \begin{bmatrix} 7 & 0 & 0 \ 8 & 9 & 0 \ 10 & 11 & 12 \end{bmatrix}. $$
Why they matter:
A permutation matrix is obtained by permuting the rows (or columns) of an identity matrix.
Example:
$$ P = \begin{bmatrix} 0 & 1 & 0 \ 1 & 0 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
Multiplying by $P$:
Permutation matrices are used in pivoting strategies in elimination, ensuring numerical stability in solving systems. They are also orthogonal: $P^{-1} = P^T$.
Together, these classes show that structure leads to simplicity-many computational algorithms exploit these patterns for speed and stability.
Almost every advanced method in numerical linear algebra relies on reducing general matrices into one of these structured forms.
$$ \begin{bmatrix} 1 & 2 \ 2 & 5 \end{bmatrix} $$
is symmetric. Find its transpose.
$$ \begin{bmatrix} 3 & 0 & 0 \ 0 & 4 & 0 \ 0 & 0 & 5 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 3 & 1 \ 0 & 5 & 2 \ 0 & 0 & 4 \end{bmatrix} \mathbf{x} = \begin{bmatrix} 1 \ 2 \ 3 \end{bmatrix} $$
using back substitution.
By exploring these four structured families, you'll start to see that not all matrices are messy-many have order hidden in their arrangement, and exploiting that order is the key to both theoretical understanding and efficient computation.
So far we have studied shapes, multiplication rules, and special classes of matrices. In this section we introduce a simple but surprisingly powerful quantity: the trace of a matrix. Along with it, we review a set of basic matrix properties that provide shortcuts, invariants, and insights into how matrices behave.
For a square matrix $A = [a_{ij}]$ of size $n \times n$, the trace is the sum of the diagonal entries:
$$ \text{tr}(A) = a_{11} + a_{22} + \cdots + a_{nn}. $$
Example:
$$ A = \begin{bmatrix} 2 & 5 & 7 \ 0 & 3 & 1 \ 4 & 6 & 8 \end{bmatrix}, \quad \text{tr}(A) = 2 + 3 + 8 = 13. $$
The trace extracts a single number summarizing the "diagonal content" of a matrix.
The trace is linear and interacts nicely with multiplication and transposition:
Linearity:
Cyclic Property:
Transpose Invariance:
Similarity Invariance:
One of the most important connections is between the trace and eigenvalues:
$$ \text{tr}(A) = \lambda_1 + \lambda_2 + \cdots + \lambda_n, $$
where $\lambda_i$ are the eigenvalues of $A$ (counting multiplicity).
This links the simple diagonal sum to the deep spectral properties of the matrix.
Example:
$$ A = \begin{bmatrix} 1 & 0 \ 0 & 3 \end{bmatrix}, \quad \text{tr}(A) = 4, \quad \lambda_1 = 1, ; \lambda_2 = 3, \quad \lambda_1 + \lambda_2 = 4. $$
Alongside the trace, here are some important algebraic facts that every student of linear algebra must know:
Determinant vs. Trace:
Norms and Inner Products:
Orthogonal Invariance:
The trace is deceptively simple but incredibly powerful:
$$ \begin{bmatrix} 4 & 2 & 0 \ -1 & 3 & 5 \ 7 & 6 & 1 \end{bmatrix}. $$
$$ A = \begin{bmatrix} 1 & 2 \ 0 & 3 \end{bmatrix}, \quad B = \begin{bmatrix} 4 & 0 \ 5 & 6 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 1 \ 1 & 2 \end{bmatrix}, $$
compute its eigenvalues and check that their sum equals the trace.
Mastering the trace and its properties will prepare you for the next leap: understanding how matrices interact with volume, orientation, and determinants.
Up to now, matrices have been used to describe linear transformations: scaling, rotating, reflecting, projecting. But real-world geometry often involves more than just linear effects-it includes translations (shifts) as well. A pure linear map cannot move the origin, so to handle translations (and combinations of them with rotations, scalings, and shears), we extend our toolkit to affine transformations. The secret weapon that makes this work is the idea of homogeneous coordinates.
An affine transformation is any map of the form:
$$ f(\mathbf{x}) = A\mathbf{x} + \mathbf{b}, $$
where $A$ is a matrix (linear part) and $\mathbf{b}$ is a vector (translation part).
Examples in 2D:
Affine maps preserve parallel lines and ratios of distances along lines, but not necessarily angles or lengths.
If we only use a 2×2 matrix in 2D or 3×3 in 3D, the origin always stays fixed. That's a limitation: real-world movements (like moving a shape from one place to another) require shifting the origin too. To capture both linear and translational effects uniformly, we need a clever trick.
The trick is to add one extra coordinate.
This new representation is called homogeneous coordinates. It allows us to fold translations into matrix multiplication.
In 2D:
\begin{bmatrix} ax + by + t_x \ cx + dy + t_y \ 1 \end{bmatrix}. $$
Here,
So with one unified matrix, we can handle both linear transformations and shifts.
$$ \begin{bmatrix} 1 & 0 & 2 \ 0 & 1 & 3 \ 0 & 0 & 1 \end{bmatrix}. $$
$$ \begin{bmatrix} 2 & 0 & -1 \ 0 & 3 & 4 \ 0 & 0 & 1 \end{bmatrix}. $$
$$ \begin{bmatrix} 0 & -1 & 5 \ 1 & 0 & 0 \ 0 & 0 & 1 \end{bmatrix}. $$
In 3D, affine transformations use 4×4 matrices. The upper-left 3×3 block handles rotation, scaling, or shear; the last column encodes translation.
Example: translation by (2, –1, 4):
$$ \begin{bmatrix} 1 & 0 & 0 & 2 \ 0 & 1 & 0 & -1 \ 0 & 0 & 1 & 4 \ 0 & 0 & 0 & 1 \end{bmatrix}. $$
This formulation is universal in computer graphics and robotics.
Mastering affine transformations and homogeneous coordinates bridges the gap between pure linear algebra and real-world geometry, giving you the mathematical foundation behind computer graphics, robotics, and spatial modeling.
Thus far, we have studied what matrices are and what they represent. But in practice, working with matrices also means thinking about computation-how much work operations take, how algorithms can be sped up, and why structure matters. This section introduces the basic ideas of computational cost in matrix operations, simple strategies for efficiency, and why these considerations are crucial in modern applications.
The simplest way to measure the cost of a matrix operation is to count the basic arithmetic operations (additions and multiplications).
Matrix–vector product: For an $m \times n$ matrix and an $n \times 1$ vector:
Matrix–matrix product: For an $m \times n$ matrix times an $n \times p$ matrix:
Gaussian elimination (solving $Ax=b$): For an $n \times n$ system:
These counts show how quickly costs grow with dimension. Doubling $n$ makes the work 8 times larger for elimination.
Cost is not only arithmetic: storage also matters.
Modern computing leverages hardware for speed:
This is why linear algebra libraries (BLAS, LAPACK, cuBLAS) are indispensable: they squeeze performance from hardware.
By learning to count costs and exploit structure, you prepare yourself not only to understand matrices abstractly but also to use them effectively in real-world, large-scale problems. This balance between theory and computation is at the heart of modern linear algebra.
Patterns intertwine,
transformations gently fold,
structure in the square.
Linear algebra often begins with systems of equations-collections of unknowns linked by linear relationships. While these systems can be solved directly using substitution or elimination, they quickly become messy when there are many variables. The key insight of linear algebra is that all systems of linear equations can be captured compactly by matrices and vectors. This section explains how we move from equations written out in words and symbols to the matrix form that powers computation.
Consider this system of two equations in two unknowns:
$$
\begin{cases}
2x + y = 5 \
3x - y = 4
\end{cases}
$$
At first glance, this is just algebra: two equations, two unknowns. But notice the structure: each equation is a sum of multiples of the variables, set equal to a constant. This pattern-linear combinations of unknowns equal to a result-is exactly what matrices capture.
Extract the coefficients of each variable from the system:
Arrange these coefficients in a rectangular array:
$$
A = \begin{bmatrix}
2 & 1 \
3 & -1
\end{bmatrix}.
$$
This matrix $A$ is called the coefficient matrix.
Next, write the unknowns as a vector:
$$ \mathbf{x} = \begin{bmatrix} x \ y \end{bmatrix}. $$
Finally, write the right-hand side constants as another vector:
$$ \mathbf{b} = \begin{bmatrix} 5 \ 4 \end{bmatrix}. $$
Now the entire system can be written in a single line:
$$ A\mathbf{x} = \mathbf{b}. $$
This compact form hides no information; it is equivalent to the original equations. But it gives us enormous advantages:
System of three equations in three unknowns:
$$
\begin{cases}
x + 2y - z = 2 \
2x - y + 3z = 1 \
3x + y + 2z = 4
\end{cases}
$$
Coefficient matrix:
$$
A = \begin{bmatrix}
1 & 2 & -1 \
2 & -1 & 3 \
3 & 1 & 2
\end{bmatrix}.
$$
Unknown vector:
$$ \mathbf{x} = \begin{bmatrix} x \ y \ z \end{bmatrix}. $$
Constant vector:
$$ \mathbf{b} = \begin{bmatrix} 2 \ 1 \ 4 \end{bmatrix}. $$
Matrix form:
$$ A\mathbf{x} = \mathbf{b}. $$
This single equation captures three equations and three unknowns in one object.
For the 2×2 case earlier:
$$
A\mathbf{x} = \begin{bmatrix} 2 & 1 \ 3 & -1 \end{bmatrix}
\begin{bmatrix} x \ y \end{bmatrix}
= x \begin{bmatrix} 2 \ 3 \end{bmatrix} + y \begin{bmatrix} 1 \ -1 \end{bmatrix}.
$$
So solving the system means finding scalars $x, y$ that combine the columns of $A$ to reach $\mathbf{b}$.
Sometimes we want to save space further. We can put the coefficients and constants side by side in an augmented matrix:
$$
[A | \mathbf{b}] =
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}.
$$
This form is especially useful for elimination methods, where we manipulate rows without writing variables at each step.
This step-rewriting equations as matrix form-is the gateway into linear algebra. Once you can do it, you no longer think of systems of equations as isolated lines on paper, but as a unified object that can be studied with general tools. It opens the door to:
Every major idea flows from this compact representation.
Write the system
$$
\begin{cases}
4x - y = 7 \
-2x + 3y = 5
\end{cases}
$$
in matrix form.
For the system
$$
\begin{cases}
x + y + z = 6 \
2x - y + z = 3 \
x - y - z = -2
\end{cases}
$$
build the coefficient matrix, unknown vector, and constant vector.
Express the augmented matrix for the above system.
Challenge: Interpret the system in column view. What does it mean geometrically to express $(6, 3, -2)$ as a linear combination of the columns of the coefficient matrix?
By practicing these rewrites, you will see that linear algebra is not about juggling many equations-it is about seeing structure in one compact equation. This step transforms scattered equations into the language of matrices, where the real power begins.
Once a system of linear equations has been expressed as a matrix, the next step is to simplify that matrix into a form where the solutions become clear. The main tool for this simplification is the set of elementary row operations. These operations allow us to manipulate the rows of a matrix in systematic ways that preserve the solution set of the corresponding system of equations.
There are exactly three types of legal row operations, each with a clear algebraic meaning:
Row Swapping ($R_i \leftrightarrow R_j$): Exchange two rows. This corresponds to reordering equations in a system. Since the order of equations doesn't change the solutions, this operation is always valid.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \longrightarrow \quad
\begin{bmatrix}
3 & -1 & | & 4 \
2 & 1 & | & 5
\end{bmatrix}.
$$
Row Scaling ($R_i \to cR_i, ; c \neq 0$): Multiply all entries in a row by a nonzero constant. This is like multiplying both sides of an equation by the same number, which doesn't change its truth.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \longrightarrow \quad
\begin{bmatrix}
1 & \tfrac{1}{2} & | & \tfrac{5}{2} \
3 & -1 & | & 4
\end{bmatrix}.
$$
Row Replacement ($R_i \to R_i + cR_j$): Add a multiple of one row to another. This corresponds to replacing one equation with a linear combination of itself and another, a fundamental elimination step.
Example:
$$
\begin{bmatrix}
2 & 1 & | & 5 \
3 & -1 & | & 4
\end{bmatrix}
\quad \overset{R_2 \to R_2 - \tfrac{3}{2}R_1}{\longrightarrow} \quad
\begin{bmatrix}
2 & 1 & | & 5 \
0 & -\tfrac{5}{2} & | & -\tfrac{7}{2}
\end{bmatrix}.
$$
These three operations are the backbone of elimination because they do not alter the solution set of the system. Each is equivalent to applying an invertible transformation:
Thus, each operation is invertible, and the transformed system is always equivalent to the original.
Each elementary row operation can itself be represented by multiplying on the left with a special matrix called an elementary matrix.
For example:
Swapping rows 1 and 2 in a 2×2 system is done by
$$ E = \begin{bmatrix} 0 & 1 \ 1 & 0 \end{bmatrix}. $$
Scaling row 1 by 3 in a 2×2 system is done by
$$ E = \begin{bmatrix} 3 & 0 \ 0 & 1 \end{bmatrix}. $$
This perspective is crucial later for factorization methods like LU decomposition, where elimination is expressed as a product of elementary matrices.
System:
$$
\begin{cases}
x + 2y = 4 \
3x + 4y = 10
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & | & 4 \
3 & 4 & | & 10
\end{bmatrix}.
$$
Eliminate the $3x$ under the first pivot: $R_2 \to R_2 - 3R_1$.
$$
\begin{bmatrix}
1 & 2 & | & 4 \
0 & -2 & | & -2
\end{bmatrix}.
$$
Scale the second row: $R_2 \to -\tfrac{1}{2}R_2$.
$$
\begin{bmatrix}
1 & 2 & | & 4 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Eliminate above the pivot: $R_1 \to R_1 - 2R_2$.
$$
\begin{bmatrix}
1 & 0 & | & 2 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Solution: $x = 2, ; y = 1$.
Row operations do not alter the solution space:
Thus, row operations act like "reshaping the system" while leaving the intersection intact.
Row operations are the essential moves in solving linear systems by hand or computer. They:
Apply row operations to reduce
$$
\begin{bmatrix}
2 & 1 & | & 7 \
1 & -1 & | & 1
\end{bmatrix}
$$
to a form where the solution is obvious.
Show explicitly why swapping two equations in a system doesn't change its solutions.
Construct the elementary matrix for "add –2 times row 1 to row 3" in a 3×3 system.
Challenge: Prove that any elementary row operation corresponds to multiplication by an invertible matrix.
Mastering these operations equips you with the mechanical and conceptual foundation for the next stage: systematically reducing matrices to row-echelon form.
After introducing row operations, the natural question is: what are we trying to achieve by performing them? The answer is to transform a matrix into a standardized, simplified form where the solutions to the corresponding system of equations can be read off directly. Two such standardized forms are central in linear algebra: row-echelon form (REF) and reduced row-echelon form (RREF).
A matrix is in row-echelon form if:
Example of REF:
$$
\begin{bmatrix}
1 & 2 & 3 & | & 4 \
0 & 1 & -1 & | & 2 \
0 & 0 & 5 & | & -3 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Here, the pivots are the first 1 in row 1, the 1 in row 2, and the 5 in row 3. Each pivot is to the right of the one above it, and all entries below pivots are zero.
A matrix is in reduced row-echelon form if, in addition to the rules of REF:
Example of RREF:
$$
\begin{bmatrix}
1 & 0 & 0 & | & 3 \
0 & 1 & 0 & | & -2 \
0 & 0 & 1 & | & 1
\end{bmatrix}.
$$
This form is so simplified that solutions can be read directly: here, $x=3$, $y=-2$, $z=1$.
System:
$$
\begin{cases}
x + 2y + z = 4 \
2x + 5y + z = 7 \
3x + 6y + 2z = 10
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
2 & 5 & 1 & | & 7 \
3 & 6 & 2 & | & 10
\end{bmatrix}.
$$
Eliminate below first pivot (the 1 in row 1, col 1):
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & -1 \
0 & 0 & -1 & | & -2
\end{bmatrix}.
$$
This is now in REF.
Scale pivots and eliminate above them:
Final:
$$
\begin{bmatrix}
1 & 0 & 0 & | & 2 \
0 & 1 & 0 & | & 1 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
Solution: $x=2, y=1, z=2$.
Reduce
$$
\begin{bmatrix}
2 & 4 & | & 6 \
1 & 3 & | & 5
\end{bmatrix}
$$
to REF, then RREF.
Find the RREF of
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
2 & 3 & 4 & | & 8 \
1 & 2 & 3 & | & 5
\end{bmatrix}.
$$
Explain why two different elimination sequences can lead to different REF but the same RREF.
Challenge: Prove that every matrix has a unique RREF by considering the effect of row operations systematically.
Reaching row-echelon and reduced row-echelon forms transforms messy systems into structured ones, turning algebraic clutter into an organized path to solutions.
When reducing a matrix to row-echelon or reduced row-echelon form, certain positions in the matrix take on a special importance. These are the pivots-the leading nonzero entries in each row. Around them, the entire solution structure of a linear system is organized. Understanding pivots, the variables they anchor, and the freedom that arises from non-pivot columns is essential to solving linear equations systematically.
In row-echelon form, a pivot is the first nonzero entry in a row, moving from left to right. After scaling in reduced row-echelon form, each pivot is set to exactly 1.
Example:
$$
\begin{bmatrix}
1 & 2 & 0 & | & 5 \
0 & 1 & 3 & | & -2 \
0 & 0 & 0 & | & 0
\end{bmatrix}
$$
Columns with pivots are pivot columns. Columns without pivots correspond to free variables.
Example:
$$
\begin{bmatrix}
1 & 0 & 2 & | & 3 \
0 & 1 & -1 & | & 4
\end{bmatrix}.
$$
This corresponds to:
$$ x_1 + 2x_3 = 3, \quad x_2 - x_3 = 4. $$
Here:
Thus, $x_1$ and $x_2$ depend on $x_3$:
$$ x_1 = 3 - 2x_3, \quad x_2 = 4 + x_3, \quad x_3 \text{ free}. $$
The solution set is infinite, described by the freedom in $x_3$.
In 2D:
Thus, the number of free variables determines the dimension of the solution set.
The number of pivot columns equals the rank of the matrix.
If the coefficient matrix $A$ is $m \times n$:
This is the rank–nullity connection in action:
$$ \text{number of variables} = \text{rank} + \text{nullity}. $$
System:
$$
\begin{cases}
x + 2y + z = 4 \
2x + 5y + z = 7
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
2 & 5 & 1 & | & 7
\end{bmatrix}.
$$
Reduce:
$R_2 \to R_2 - 2R_1$ →
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & -1
\end{bmatrix}.
$$
Now:
Solution:
$$ x = 4 - 2y - z, \quad y = -1 + z, \quad z \text{ free}. $$
Substitute:
$$ (x, y, z) = (6 - 3z, ; -1 + z, ; z). $$
Solutions form a line in 3D parameterized by $z$.
In RREF, each pivot is scaled to 1, making it easy to isolate pivot variables. Without leading ones, equations may still be correct but harder to interpret.
For example:
$$
\begin{bmatrix}
2 & 0 & | & 6 \
0 & -3 & | & 9
\end{bmatrix}
$$
becomes
$$
\begin{bmatrix}
1 & 0 & | & 3 \
0 & 1 & | & -3
\end{bmatrix}.
$$
The solutions are immediately visible: $x=3, y=-3$.
Reduce
$$
\begin{bmatrix}
1 & 3 & 1 & | & 5 \
2 & 6 & 2 & | & 10
\end{bmatrix}
$$
and identify pivot and free variables.
For the system
$$ x + y + z = 2, \quad 2x + 3y + 5z = 7, $$
write the RREF and express the solution with free variables.
Compute the rank and number of free variables of a 3×5 matrix with two pivot columns.
Challenge: Show that if the number of pivots equals the number of variables, the system has either no solution or a unique solution, but never infinitely many.
Understanding pivots and free variables provides the key to classifying solution sets: unique, infinite, or none. This classification lies at the heart of solving linear systems.
A system of linear equations is called consistent if it has at least one solution. Consistency is the first property to check when working with a system, because before worrying about uniqueness or parametrization, we must know whether a solution exists at all. This section explains how to recognize consistent systems, how to solve them using row-reduction, and how to describe their solutions in terms of pivots and free variables.
Given a system $A\mathbf{x} = \mathbf{b}$:
Consistency depends on the relationship between the vector $\mathbf{b}$ and the column space of $A$:
$$ \mathbf{b} \in \text{Col}(A) \quad \iff \quad \text{system is consistent}. $$
If $\mathbf{b}$ cannot be written as a linear combination of the columns of $A$, the system has no solution.
To test consistency, reduce the augmented matrix $[A | \mathbf{b}]$ to row-echelon form.
If you find a row of the form:
$$ [0 ;; 0 ;; \dots ;; 0 ;|; c], \quad c \neq 0, $$
then the system is inconsistent (contradiction: 0 = c).
If no such contradiction appears, the system is consistent.
System:
$$
\begin{cases}
x + y = 2 \
x - y = 0
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
1 & -1 & | & 0
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - R_1$:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & -2 & | & -2
\end{bmatrix}.
$$
$R_2 \to -\tfrac{1}{2}R_2$:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & 1 & | & 1
\end{bmatrix}.
$$
$R_1 \to R_1 - R_2$:
$$
\begin{bmatrix}
1 & 0 & | & 1 \
0 & 1 & | & 1
\end{bmatrix}.
$$
Solution: $x = 1, ; y = 1$. Unique solution.
System:
$$
\begin{cases}
x + y + z = 3 \
2x + 2y + 2z = 6
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
2 & 2 & 2 & | & 6
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - 2R_1$:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 3 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
No contradiction, so consistent. Solution:
$$ x = 3 - y - z, \quad y \text{ free}, \quad z \text{ free}. $$
The solution set is a plane in $\mathbb{R}^3$.
System:
$$
\begin{cases}
x + y = 1 \
x + y = 2
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
1 & 1 & | & 2
\end{bmatrix}.
$$
Row reduce:
$R_2 \to R_2 - R_1$:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
0 & 0 & | & 1
\end{bmatrix}.
$$
Contradiction: $0 = 1$. Inconsistent, no solution.
In 2D:
In 3D:
Reduce the augmented matrix
$$
\begin{bmatrix}
1 & 2 & 1 & | & 5 \
2 & 4 & 2 & | & 10 \
3 & 6 & 3 & | & 15
\end{bmatrix}
$$
and determine if the system is consistent.
Classify the system as having unique, infinite, or no solutions:
$$
\begin{cases}
x + y + z = 2 \
x - y + z = 0 \
2x + 0y + 2z = 3
\end{cases}
$$
Explain geometrically what it means when the augmented matrix has a contradictory row.
Challenge: Show algebraically that a system is consistent if and only if $\mathbf{b}$ lies in the span of the columns of $A$.
Consistent systems mark the balance point between algebraic rules and geometric reality: they are where equations and space meet in harmony.
Not every system of linear equations has a solution. Some are inconsistent, meaning the equations contradict one another and no vector $\mathbf{x}$ can satisfy them all at once. Detecting such inconsistency early is crucial: it saves wasted effort trying to solve an impossible system and reveals important geometric and algebraic properties.
Consider the system:
$$
\begin{cases}
x + y = 1 \
x + y = 3
\end{cases}
$$
Clearly, the two equations cannot both be true. In augmented matrix form:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
1 & 1 & | & 3
\end{bmatrix}.
$$
Row reduction gives:
$$
\begin{bmatrix}
1 & 1 & | & 1 \
0 & 0 & | & 2
\end{bmatrix}.
$$
The bottom row says $0 = 2$, a contradiction. This is the hallmark of inconsistency: a row of zeros in the coefficient part, with a nonzero constant in the augmented part.
A system $A\mathbf{x} = \mathbf{b}$ is inconsistent if, after row reduction, the augmented matrix contains a row of the form:
$$ [0 ;; 0 ;; \dots ;; 0 ;|; c], \quad c \neq 0. $$
This indicates that all variables vanish from the equation, leaving an impossible statement like $0 = c$.
$$
\begin{cases}
x + y = 2 \
2x + 2y = 5
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
2 & 2 & | & 5
\end{bmatrix}.
$$
Row reduce:
$$
\begin{bmatrix}
1 & 1 & | & 2 \
0 & 0 & | & 1
\end{bmatrix}.
$$
Contradiction: no solution. Geometrically, the two equations are parallel lines that never intersect.
$$
\begin{cases}
x + y + z = 1 \
2x + 2y + 2z = 2 \
x + y + z = 3
\end{cases}
$$
The first and third equations already conflict: the same plane equation is forced to equal two different constants.
Augmented matrix reduces to:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 1 \
0 & 0 & 0 & | & 0 \
0 & 0 & 0 & | & 2
\end{bmatrix}.
$$
Contradiction: no solution. The "planes" fail to intersect in common.
Another way to detect inconsistency is using ranks.
Rule:
This rank condition is fundamental and works in any dimension.
$$
\begin{bmatrix}
1 & -1 & | & 2 \
2 & -2 & | & 5
\end{bmatrix}
$$
and decide if the system is consistent.
$$ x + y = 0, \quad x + y = 1 $$
is inconsistent.
$$
\begin{cases}
x + y + z = 2 \
2x + 2y + 2z = 4 \
3x + 3y + 3z = 5
\end{cases}
$$
Detecting inconsistency is not just about spotting contradictions-it connects algebra, geometry, and linear transformations, showing exactly when a system cannot possibly fit together.
Gaussian elimination is the systematic procedure for solving systems of linear equations by using row operations to simplify the augmented matrix. The goal is to transform the system into row-echelon form (REF) and then use back substitution to find the solutions. This method is the backbone of linear algebra computations and is the foundation of most computer algorithms for solving linear systems.
Suppose we have $n$ equations with $n$ unknowns.
System:
$$
\begin{cases}
x + 2y = 5 \
3x + 4y = 11
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 2 & | & 5 \
3 & 4 & | & 11
\end{bmatrix}.
$$
Pivot at (1,1) = 1.
Eliminate below: $R_2 \to R_2 - 3R_1$.
$$
\begin{bmatrix}
1 & 2 & | & 5 \
0 & -2 & | & -4
\end{bmatrix}.
$$
Back substitution: From row 2: $-2y = -4 \implies y = 2$. Substitute into row 1: $x + 2(2) = 5 \implies x = 1$.
Solution: $(x, y) = (1, 2)$.
System:
$$
\begin{cases}
x + y + z = 6 \
2x + 3y + z = 14 \
x - y + 2z = 2
\end{cases}
$$
Augmented matrix:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
2 & 3 & 1 & | & 14 \
1 & -1 & 2 & | & 2
\end{bmatrix}.
$$
Step 1: Pivot at (1,1). Eliminate below:
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & -2 & 1 & | & -4
\end{bmatrix}.
$$
Step 2: Pivot at (2,2). Eliminate below: $R_3 \to R_3 + 2R_2$.
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & 0 & -1 & | & 0
\end{bmatrix}.
$$
Step 3: Pivot at (3,3). Scale row: $R_3 \to -R_3$.
$$
\begin{bmatrix}
1 & 1 & 1 & | & 6 \
0 & 1 & -1 & | & 2 \
0 & 0 & 1 & | & 0
\end{bmatrix}.
$$
Back substitution:
Solution: $(x, y, z) = (4, 2, 0)$.
Elimination corresponds to progressively restricting the solution set:
Solve the system
$$
\begin{cases}
2x + y = 7 \
4x + 3y = 15
\end{cases}
$$
using Gaussian elimination.
Reduce
$$
\begin{bmatrix}
1 & 2 & -1 & | & 3 \
3 & 8 & 1 & | & 12 \
2 & 6 & 3 & | & 11
\end{bmatrix}
$$
to REF and solve.
Practice with a system that has infinitely many solutions:
$$ x + y + z = 4, \quad 2x + 2y + 2z = 8. $$
Challenge: Explain why Gaussian elimination always terminates in at most $n$ pivot steps for an $n \times n$ system.
Gaussian elimination transforms the complexity of many equations into an orderly process, making the hidden structure of solutions visible step by step.
Once Gaussian elimination reduces a system to row-echelon form (REF), the next step is to actually solve for the unknowns. This process is called back substitution: we begin with the bottom equation (which involves the fewest variables) and work our way upward, solving step by step. Back substitution is what converts the structured triangular system into explicit solutions.
A system in REF looks like this:
$$ \begin{bmatrix}
& * & * & * & | & * \
0 & * & * & * & | & * \
0 & 0 & * & * & | & * \
0 & 0 & 0 & * & | & *
\end{bmatrix}
$$
Each row corresponds to an equation with fewer variables than the row above.
The bottom equation has only one or two variables.
This triangular form makes it possible to solve "from the bottom up."
System after elimination:
$$
\begin{bmatrix}
1 & 2 & -1 & | & 3 \
0 & 1 & 2 & | & 4 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
This corresponds to:
$$
\begin{cases}
x + 2y - z = 3 \
y + 2z = 4 \
z = 2
\end{cases}
$$
Solution: $(x, y, z) = (5, 0, 2)$.
Not all systems reduce to unique solutions. If there are free variables (non-pivot columns), back substitution expresses pivot variables in terms of free ones.
Example:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & 1 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Equations:
$$
\begin{cases}
x + 2y + z = 4 \
y - z = 1
\end{cases}
$$
Solution set:
$$ (x, y, z) = (2 - 3t, ; 1 + t, ; t), \quad t \in \mathbb{R}. $$
Here $z = t$ is the free variable. The solutions form a line in 3D.
For a consistent system:
For the infinite-solution example above:
$$ (x, y, z) = (2, 1, 0) + t(-3, 1, 1). $$
This expresses the solution set as a base point plus a direction vector, making the geometry clear.
$$
\begin{bmatrix}
1 & -1 & 2 & | & 3 \
0 & 1 & 3 & | & 5 \
0 & 0 & 1 & | & 2
\end{bmatrix}.
$$
$$ x + y + z = 2, \quad 2x + 2y + 2z = 4. $$
Express the solution set of the above system in parametric vector form.
Challenge: For a 4×4 system with two free variables, explain why the solution set forms a plane in $\mathbb{R}^4$.
Back substitution completes the elimination process, translating triangular structure into explicit solutions, and shows how algebra and geometry meet in the classification of solution sets.
The concept of rank lies at the heart of linear algebra. It connects the algebra of solving systems, the geometry of subspaces, and the structure of matrices into one unifying idea. Rank measures the amount of independent information in a matrix: how many rows or columns carry unique directions instead of being repetitions or combinations of others.
The rank of a matrix $A$ is the number of pivots in its row-echelon form. Equivalently, it is:
All these definitions agree.
When solving a system with Gaussian elimination:
Example:
$$
\begin{bmatrix}
1 & 2 & 1 & | & 4 \
0 & 1 & -1 & | & 2 \
0 & 0 & 0 & | & 0
\end{bmatrix}.
$$
Here, there are 2 pivots. So:
A set of vectors is linearly independent if none can be expressed as a combination of the others.
Example:
$$
\begin{bmatrix}
1 & 2 & 3 \
2 & 4 & 6 \
3 & 6 & 9
\end{bmatrix}.
$$
Here, each row is a multiple of the first. Rank = 1, since only one independent row/column direction exists.
Consider $A\mathbf{x} = \mathbf{b}$.
Thus, rank classifies solution sets.
Rank tells us the dimension of the subspace spanned by rows or columns.
Example:
It is a remarkable fact that the number of independent rows = number of independent columns. This is not obvious at first glance, but it is always true. So we can define rank either by rows or by columns-it makes no difference.
Find the rank of
$$
\begin{bmatrix}
1 & 2 & 3 \
2 & 4 & 5 \
3 & 6 & 8
\end{bmatrix}.
$$
Solve the system
$$ x + y + z = 2, \quad 2x + 2y + 2z = 4, $$
and identify the rank of the coefficient matrix.
In $\mathbb{R}^3$, what is the geometric meaning of a 3×3 matrix of rank 2?
Challenge: Prove that the row rank always equals the column rank by considering the echelon form of the matrix.
Rank is the first truly unifying concept in linear algebra: it tells us how much independent structure a matrix contains and sets the stage for understanding spaces, dimensions, and transformations.
Gaussian elimination not only solves systems but also reveals a deeper structure: many matrices can be factored into simpler pieces. One of the most useful is the LU factorization, where a matrix $A$ is written as the product of a lower-triangular matrix $L$ and an upper-triangular matrix $U$. This factorization captures all the elimination steps in a compact form and allows systems to be solved efficiently.
If $A$ is an $n \times n$ matrix, then
$$ A = LU, $$
where:
This means:
Take
$$
A = \begin{bmatrix}
2 & 3 \
4 & 7
\end{bmatrix}.
$$
Elimination: $R_2 \to R_2 - 2R_1$.
Multiplier = 2 (used to eliminate entry 4).
Resulting $U$:
$$
U = \begin{bmatrix}
2 & 3 \
0 & 1
\end{bmatrix}.
$$
$L$:
$$
L = \begin{bmatrix}
1 & 0 \
2 & 1
\end{bmatrix}.
$$
Check:
$$
LU = \begin{bmatrix}
1 & 0 \
2 & 1
\end{bmatrix}
\begin{bmatrix}
2 & 3 \
0 & 1
\end{bmatrix}
= \begin{bmatrix}
2 & 3 \
4 & 7
\end{bmatrix} = A.
$$
$$
A = \begin{bmatrix}
2 & 1 & 1 \
4 & -6 & 0 \
-2 & 7 & 2
\end{bmatrix}.
$$
Step 1: Eliminate below pivot (row 1).
Step 2: Eliminate below pivot in column 2.
Result:
$$
L = \begin{bmatrix}
1 & 0 & 0 \
2 & 1 & 0 \
-1 & -1 & 1
\end{bmatrix}, \quad
U = \begin{bmatrix}
2 & 1 & 1 \
0 & -8 & -2 \
0 & 0 & 1
\end{bmatrix}.
$$
Thus $A = LU$.
Suppose $Ax = b$. If $A = LU$:
This two-step process is much faster than elimination from scratch each time, especially if solving multiple systems with the same $A$ but different $b$.
Sometimes elimination requires row swaps (to avoid division by zero or instability). Then factorization is written as:
$$ PA = LU, $$
where $P$ is a permutation matrix recording the row swaps. This is the practical form used in numerical computing.
LU decomposition separates the elimination process into:
Together, they represent the same linear transformation as $A$, but decomposed into simpler building blocks.
Factor
$$
A = \begin{bmatrix}
1 & 2 \
3 & 8
\end{bmatrix}
$$
into $LU$.
Solve
$$
\begin{bmatrix}
2 & 1 \
6 & 3
\end{bmatrix}
\begin{bmatrix}
x \ y
\end{bmatrix} =
\begin{bmatrix}
5 \ 15
\end{bmatrix}
$$
using LU decomposition.
Compute $\det(A)$ for
$$
A = \begin{bmatrix}
2 & 1 & 1 \
4 & -6 & 0 \
-2 & 7 & 2
\end{bmatrix}
$$
by using its LU factorization.
Challenge: Prove that if $A$ is invertible, then it has an LU factorization (possibly after row swaps).
LU factorization organizes elimination into a powerful tool: compact, efficient, and deeply tied to both the theory and practice of linear algebra.
Paths diverge or merge,
pivots mark the way forward,
truth distilled in rows.
Endless skies expand,
spaces within spaces grow,
freedom takes its shape.
Up to now, we have worked with vectors in $\mathbb{R}^2$, $\mathbb{R}^3$, and higher-dimensional Euclidean spaces. But the true power of linear algebra comes from abstracting away from coordinates. A vector space is not tied to arrows in physical space-it is any collection of objects that behave like vectors, provided they satisfy certain rules. These rules are called the axioms of vector spaces.
A vector space is a set $V$ equipped with two operations:
The magic is that as long as certain rules (axioms) hold, the objects in $V$ can be treated as vectors. They need not be arrows or coordinate lists-they could be polynomials, functions, matrices, or sequences.
Let $u, v, w \in V$ (vectors) and $a, b \in \mathbb{R}$ (scalars). The axioms are:
(These are sometimes listed as eight, with some grouped together, but the essence is the same.)
All of these satisfy the vector space axioms.
These examples show why the axioms are essential: without them, the structure breaks.
Every vector space must contain a zero vector. This is not optional. It is the "do nothing" element for addition. In $\mathbb{R}^n$, this is $(0,0,\dots,0)$. In polynomials, it is the zero polynomial. In function spaces, it is the function $f(x) = 0$.
For every vector $v$, we require $-v$. This ensures that equations like $u+v=w$ can always be rearranged to $u=w-v$. Without additive inverses, solving linear equations would not work.
Scalars come from a field: usually the real numbers $\mathbb{R}$ or the complex numbers $\mathbb{C}$. The choice of scalars matters:
Together, these rules make vector spaces stable and reliable mathematical objects.
The axioms of vector spaces provide the foundation on which the rest of linear algebra is built. Everything that follows-subspaces, independence, basis, dimension-grows naturally from this formal framework.
Once the idea of a vector space is in place, the next step is to recognize smaller vector spaces that live inside bigger ones. These are called subspaces. Subspaces are central in linear algebra because they reveal the internal structure of matrices and linear systems. Two special subspaces-the column space and the null space-play particularly important roles.
A subspace $W$ of a vector space $V$ is a subset of $V$ that is itself a vector space under the same operations. To qualify as a subspace, $W$ must satisfy:
That's it-no further checking of all ten vector space axioms is needed, because those are inherited from $V$.
In $\mathbb{R}^3$:
In the space of polynomials:
In function spaces:
Given a matrix $A$, the column space is the set of all linear combinations of its columns. Formally,
$$ C(A) = { A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n }. $$
Example:
$$
A = \begin{bmatrix}
1 & 2 \
2 & 4 \
3 & 6
\end{bmatrix}.
$$
The second column is just twice the first. So the column space is all multiples of $\begin{bmatrix}1 \ 2 \ 3\end{bmatrix}$, which is a line in $\mathbb{R}^3$. Rank = 1.
The null space (or kernel) of a matrix $A$ is the set of all vectors $\mathbf{x}$ such that
$$ A\mathbf{x} = 0. $$
Example:
$$
A = \begin{bmatrix}
1 & 2 & 3 \
4 & 5 & 6
\end{bmatrix}.
$$
Solve $A\mathbf{x} = 0$. This yields a null space spanned by one vector, meaning it is a line through the origin in $\mathbb{R}^3$.
Together, they capture the full behavior of a matrix.
Show that the set ${(x,y,0) : x,y \in \mathbb{R}}$ is a subspace of $\mathbb{R}^3$.
For
$$
A = \begin{bmatrix}
1 & 2 & 3 \
0 & 0 & 0 \
1 & 2 & 3
\end{bmatrix},
$$
find the column space and its dimension.
For the same $A$, compute the null space and its dimension.
Challenge: Prove that the null space of $A$ is always a subspace of $\mathbb{R}^n$.
Subspaces-especially the column space and null space-are the first glimpse of the hidden geometry inside every matrix, showing us which directions survive and which vanish.
The idea of a span captures the simplest and most powerful way to build new vectors from old ones: by taking linear combinations. A span is not just a set of scattered points but a structured, complete collection of all combinations of a given set of vectors. Understanding span leads directly to the concepts of bases, dimension, and the structure of subspaces.
Given vectors $v_1, v_2, \dots, v_k \in V$, the span of these vectors is
$$ \text{span}{v_1, v_2, \dots, v_k} = {a_1 v_1 + a_2 v_2 + \dots + a_k v_k : a_i \in \mathbb{R}}. $$
In $\mathbb{R}^2$:
In $\mathbb{R}^3$:
A set of vectors is a generating set (or spanning set) for a subspace if their span equals that subspace.
Different generating sets can span the same space. Some may be redundant, others minimal. Later, the concept of a basis refines this idea.
Consider the system $Ax=b$.
The concept of span transforms our perspective: instead of focusing on single vectors, we see the entire landscape of possibilities they generate.
Having introduced span and generating sets, the natural question arises: when are the vectors in a spanning set truly necessary, and when are some redundant? This leads to the idea of linear independence. It is the precise way to distinguish between essential vectors (those that add new directions) and dependent vectors (those that can be expressed in terms of others).
A set of vectors ${v_1, v_2, \dots, v_k}$ is linearly independent if the only solution to
$$ a_1 v_1 + a_2 v_2 + \dots + a_k v_k = 0 $$
is
$$ a_1 = a_2 = \dots = a_k = 0. $$
If there exists a nontrivial solution (some $a_i \neq 0$), then the vectors are linearly dependent.
$(1,0)$ and $(0,1)$ are independent.
$(1,0)$ and $(2,0)$ are dependent.
Any set of 3 vectors in $\mathbb{R}^2$ is dependent, since the dimension of the space is 2.
Put the vectors as columns in a matrix. Perform row reduction:
Example:
$$
\begin{bmatrix}
1 & 2 & 3 \
0 & 1 & 4 \
0 & 0 & 0
\end{bmatrix}.
$$
Here the third column has no pivot → the 3rd vector is dependent on the first two.
Linear independence is the tool that separates essential directions from redundant ones. It is the key to defining bases, counting dimensions, and understanding the structure of all vector spaces.
The concepts of span and linear independence come together in the powerful idea of a basis. A basis gives us the minimal set of building blocks needed to generate an entire vector space, with no redundancy. Once a basis is chosen, every vector in the space can be described uniquely by a list of numbers called its coordinates.
A basis of a vector space $V$ is a set of vectors ${v_1, v_2, \dots, v_k}$ that satisfies two properties:
In short: a basis is a spanning set with no redundancy.
These are called standard because they are the default way of describing coordinates.
One of the most important facts about bases is that they provide unique representations of vectors.
Given a basis ${v_1, \dots, v_k}$, any vector $x \in V$ can be written uniquely as:
$$ x = a_1 v_1 + a_2 v_2 + \dots + a_k v_k. $$
The coefficients $(a_1, a_2, \dots, a_k)$ are the coordinates of $x$ relative to that basis.
This uniqueness distinguishes bases from arbitrary spanning sets, where redundancy allows multiple representations.
Let basis = ${(1,0), (0,1)}$.
If we switch to a different basis, the coordinates change even though the vector itself does not.
Basis = ${(1,1), (1,-1)}$ in $\mathbb{R}^2$. Find coordinates of $x = (2,0)$.
Solve $a(1,1) + b(1,-1) = (2,0)$. This gives system:
$$ a + b = 2, \quad a - b = 0. $$
So $a=1, b=1$. Coordinates relative to this basis: $(1,1)$.
Notice: coordinates depend on basis choice.
For polynomials of degree ≤ 2: basis = ${1, x, x^2}$.
For continuous functions on $[0,1]$, one possible basis is the infinite set ${1, x, x^2, \dots}$.
This shows bases are not restricted to geometric vectors.
The number of vectors in a basis is the dimension of the vector space.
Dimension tells us how many independent directions exist in the space.
A basis is the minimal, elegant foundation of a vector space, turning the infinite into the manageable by providing a finite set of independent building blocks.
Dimension is one of the most profound and unifying ideas in linear algebra. It gives a single number that captures the "size" or "capacity" of a vector space: how many independent directions it has. Unlike length, width, or height in everyday geometry, dimension in linear algebra applies to spaces of any kind-geometric, algebraic, or even function spaces.
The dimension of a vector space $V$ is the number of vectors in any basis of $V$.
Since all bases of a vector space have the same number of elements, dimension is well-defined.
If $\dim V = n$, then:
$\dim(\mathbb{R}^2) = 2$.
$\dim(\mathbb{R}^3) = 3$.
The set of all polynomials of degree ≤ 2 has dimension 3.
The space of all $m \times n$ matrices has dimension $mn$.
Finite-dimensional spaces: $\mathbb{R}^n$, polynomials of degree ≤ $k$.
Infinite-dimensional spaces:
When solving $A\mathbf{x} = \mathbf{b}$:
The dimension of the column space = rank = number of independent directions in the outputs.
The dimension of the null space = number of free variables.
By the rank–nullity theorem:
$$ \dim(\text{column space}) + \dim(\text{null space}) = \text{number of variables}. $$
Thus, dimension = length of coordinate list.
Dimension is the measuring stick of linear algebra: it tells us how many independent pieces of information are needed to describe the whole space.
The rank–nullity theorem is one of the central results of linear algebra. It gives a precise balance between two fundamental aspects of a matrix: the dimension of its column space (rank) and the dimension of its null space (nullity). It shows that no matter how complicated a matrix looks, the distribution of information between its "visible" outputs and its "hidden" null directions always obeys a strict law.
Let $A$ be an $m \times n$ matrix (mapping $\mathbb{R}^n \to \mathbb{R}^m$):
$$ \text{rank}(A) + \text{nullity}(A) = n $$
where:
Think of a matrix as a machine that transforms input vectors into outputs:
This ensures nothing disappears mysteriously-every input direction is accounted for.
$$
A = \begin{bmatrix}
1 & 0 \
0 & 1 \
\end{bmatrix}.
$$
$$
A = \begin{bmatrix}
1 & 2 \
2 & 4 \
3 & 6 \
\end{bmatrix}.
$$
$$
A = \begin{bmatrix}
1 & 0 & 1 \
0 & 1 & 1
\end{bmatrix}.
$$
Row reduce $A$ to echelon form.
Pivots correspond to independent columns → count = rank.
Free variables correspond to null space directions → count = nullity.
Each column is either a pivot column or corresponds to a free variable, so:
$$ \text{rank} + \text{nullity} = \text{number of columns}. $$
The theorem guarantees the geometry of "surviving" and "vanishing" directions always adds up consistently.
Solving systems $Ax = b$:
Data compression: Rank identifies independent features; nullity shows redundancy.
Computer graphics: Rank–nullity explains how 3D coordinates collapse into 2D images: one dimension of depth is lost.
Machine learning: Rank signals how much real information a dataset contains; nullity indicates degrees of freedom that add nothing new.
Verify rank–nullity for
$$
A = \begin{bmatrix}
1 & 2 & 3 \
4 & 5 & 6
\end{bmatrix}.
$$
For a $4 \times 5$ matrix of rank 3, what is its nullity?
In $\mathbb{R}^3$, suppose a matrix maps all of space onto a line. What are its rank and nullity?
Challenge: Prove rigorously that the row space and null space are orthogonal complements, and use this to derive rank–nullity again.
The rank–nullity theorem is the law of balance in linear algebra: every input dimension is accounted for, either as a surviving direction (rank) or as one that vanishes (nullity).
Once a basis for a vector space is chosen, every vector in that space can be described uniquely in terms of the basis. These descriptions are called coordinates. Coordinates transform abstract vectors into concrete lists of numbers, making computation possible. Changing the basis changes the coordinates, but the underlying vector remains the same.
Given a vector space $V$ and a basis $B = {v_1, v_2, \dots, v_n}$, every vector $x \in V$ can be written uniquely as:
$$ x = a_1 v_1 + a_2 v_2 + \dots + a_n v_n. $$
The coefficients $(a_1, a_2, \dots, a_n)$ are the coordinates of $x$ with respect to the basis $B$.
This representation is unique because basis vectors are independent.
Standard basis: $B = {(1,0), (0,1)}$.
Non-standard basis: $B = {(1,1), (1,-1)}$.
Write $x = (3,5)$ as $a(1,1) + b(1,-1)$.
Solve:
$$ a+b = 3, \quad a-b = 5. $$
Adding: $2a = 8 \implies a = 4$. Subtracting: $2b = -2 \implies b = -1$.
Coordinates relative to this basis: $(4, -1)$.
The same vector looks different depending on the chosen basis.
Let $B = {(1,0,0), (1,1,0), (1,1,1)}$. Find coordinates of $x = (2,3,4)$.
Solve $a(1,0,0) + b(1,1,0) + c(1,1,1) = (2,3,4)$. This gives system:
$$ a+b+c = 2, \quad b+c = 3, \quad c = 4. $$
From $c=4$, we get $b+c=3 \implies b=-1$. Then $a+b+c=2 \implies a-1+4=2 \implies a=-1$. Coordinates: $(-1, -1, 4)$.
If $B = {v_1, \dots, v_n}$, form the basis matrix
$$ P = [v_1 \ v_2 \ \dots \ v_n]. $$
Then for a vector $x$, its coordinate vector $[x]_B$ satisfies
$$ P [x]_B = x. $$
Thus,
$$ [x]_B = P^{-1}x. $$
This shows coordinate transformation is simply matrix multiplication.
Suppose a vector has coordinates $[x]_B$ relative to basis $B$. If we switch to another basis $C$, we use a change-of-basis matrix to convert coordinates:
$$ [x]_C = (P_C^{-1} P_B) [x]_B. $$
This process is fundamental in computer graphics, robotics, and data transformations.
Coordinates relative to a basis are the bridge between geometry and algebra: they turn abstract spaces into numerical systems where computation, reasoning, and transformation become systematic and precise.
Every vector space allows multiple choices of basis, and each basis provides a different way of describing the same vectors. The process of moving from one basis to another is called a change of basis. To perform this change systematically, we use a change-of-basis matrix. This matrix acts as a translator between coordinate systems: it converts the coordinates of a vector relative to one basis into coordinates relative to another.
Let $V$ be a vector space with two bases:
Suppose a vector $x \in V$ has coordinates $[x]_B$ relative to $B$, and $[x]_C$ relative to $C$.
We want a matrix $P_{B \to C}$ such that:
$$ [x]C = P{B \to C} [x]_B. $$
This matrix $P_{B \to C}$ is the change-of-basis matrix from $B$ to $C$.
In matrix form:
$$ P_{B \to C} = \big[ [b_1]_C \ [b_2]_C \ \dots \ [b_n]_C \big]. $$
Let
To build $P_{B \to C}$:
Solve:
$$ (1,0) = a(1,1) + b(1,-1). $$
This gives system:
$$ a+b=1, \quad a-b=0. $$
Solution: $a=\tfrac{1}{2}, b=\tfrac{1}{2}$. So $(1,0) = \tfrac{1}{2}(1,1) + \tfrac{1}{2}(1,-1)$.
Next:
$$ (0,1) = a(1,1) + b(1,-1). $$
System:
$$ a+b=0, \quad a-b=1. $$
Solution: $a=\tfrac{1}{2}, b=-\tfrac{1}{2}$.
Thus:
$$
P_{B \to C} = \begin{bmatrix}
\tfrac{1}{2} & \tfrac{1}{2} \
\tfrac{1}{2} & -\tfrac{1}{2}
\end{bmatrix}.
$$
So for any vector $x$,
$$ [x]C = P{B \to C}[x]_B. $$
If $P_{B \to C}$ is the change-of-basis matrix from $B$ to $C$, then its inverse is the change-of-basis matrix in the opposite direction:
$$ P_{C \to B} = (P_{B \to C})^{-1}. $$
This makes sense: translating back and forth between languages should undo itself.
Let
$$ P_B = [b_1 \ b_2 \ \dots \ b_n], \quad P_C = [c_1 \ c_2 \ \dots \ c_n], $$
the matrices whose columns are basis vectors written in standard coordinates.
Then the change-of-basis matrix from $B$ to $C$ is:
$$ P_{B \to C} = P_C^{-1} P_B. $$
This formula is extremely useful because it reduces the problem to matrix multiplication.
Change-of-basis matrices give us the precise mechanism for switching perspectives. They ensure that although bases change, vectors remain invariant, and computations remain consistent.
So far, vector spaces and subspaces have always passed through the origin. But in many real-world situations, we deal with shifted versions of these spaces: planes not passing through the origin, lines offset from the zero vector, or solution sets to linear equations with nonzero constants. These structures are called affine subspaces. They extend the idea of subspaces by allowing "translation away from the origin."
An affine subspace of a vector space $V$ is a set of the form
$$ x_0 + W = {x_0 + w : w \in W}, $$
where:
Thus, an affine subspace is simply a subspace shifted by a vector.
Affine subspaces naturally arise as solution sets of linear equations.
Homogeneous system: $Ax = 0$.
Non-homogeneous system: $Ax = b$ with $b \neq 0$.
Solution set is affine.
If $x_p$ is one particular solution, then the general solution is:
$$ x = x_p + N(A), $$
where $N(A)$ is the null space.
Thus, the geometry of solving equations leads naturally to affine subspaces.
The dimension of an affine subspace is defined as the dimension of its direction subspace $W$.
Think of affine subspaces as "flat sheets" floating in space:
Show that the set of solutions to
$$ x+y+z=1 $$
is an affine subspace of $\mathbb{R}^3$. Identify its dimension.
Find the general solution to
$$ x+2y=3 $$
and describe it as an affine subspace.
Prove that the intersection of two affine subspaces is either empty or another affine subspace.
Challenge: Show that every affine subspace can be written uniquely as $x_0 + W$ with $W$ a subspace.
Affine subspaces are the natural setting for most real-world linear problems: they combine the strict structure of subspaces with the freedom of translation, capturing both direction and position.
Each basis a song,
dimension counts melodies,
the space breathes its form.
Maps preserve the line,
reflections ripple outward,
motion kept in frame.
A linear transformation is the heart of linear algebra. It is the rule that connects two vector spaces in a way that respects their linear structure: addition and scalar multiplication. Instead of thinking of vectors as static objects, linear transformations let us study how vectors move, stretch, rotate, project, or reflect. They give linear algebra its dynamic power and are the bridge between abstract theory and concrete applications.
A function $T: V \to W$ between vector spaces is called a linear transformation if for all $u, v \in V$ and scalars $a, b \in \mathbb{R}$ (or another field),
$$ T(au + bv) = aT(u) + bT(v). $$
This single condition encodes two rules:
If both are satisfied, the transformation is linear.
Scaling: $T(x) = 3x$ in $\mathbb{R}$. Every number is stretched threefold.
Rotation in the plane: $T(x,y) = (x\cos\theta - y\sin\theta, , x\sin\theta + y\cos\theta)$.
Projection: Projecting $(x,y,z)$ onto the $xy$-plane: $T(x,y,z) = (x,y,0)$.
Differentiation: On the space of polynomials, $T(p(x)) = p'(x)$.
Integration: On continuous functions, $T(f)(x) = \int_0^x f(t) , dt$.
All these are linear because they preserve addition and scaling.
Nonlinear rules break the structure of vector spaces.
Every linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ can be represented by a matrix.
If $T: \mathbb{R}^n \to \mathbb{R}^m$, then there exists an $m \times n$ matrix $A$ such that:
$$ T(x) = Ax. $$
The columns of $A$ are simply $T(e_1), T(e_2), \dots, T(e_n)$, where $e_i$ are the standard basis vectors.
Example: Let $T(x,y) = (2x+y, x-y)$.
$$ A = \begin{bmatrix} 2 & 1 \ 1 & -1 \end{bmatrix}. $$
Then $T(x,y) = A \begin{bmatrix} x \ y \end{bmatrix}$.
For $T: V \to W$:
$$ \dim(\ker T) + \dim(\text{im}(T)) = \dim(V). $$
Linear transformations reshape space:
The key feature: straight lines remain straight, and the origin stays fixed.
Truncated — view the full README on GitHub.
(top 24 of 34)
12,947 followers · starred Sep 2025
2,437 followers · starred Sep 2025
135 followers · starred Sep 2025
85 followers · starred Sep 2025