academy

Linear Algebra for AI: Path 1 – Foundation Builder

Path 1: Alicia - Foundation Builder. A complete guide for self-taught developers learning linear algebra for machine learning from scratch.

Ernesto Diaz-Aviles

• 8 min read
Linear Algebra for AI: Path 1 – Foundation Builder
Photo by Joshua Hoehne / Unsplash

Path 1: Alicia - Foundation Builder

Welcome, Alicia! 👋

You taught yourself Python through online courses or a bootcamp. You can build web apps, work with APIs, and use pandas for data analysis. But when you try to use scikit-learn or understand ML tutorials, the math feels like a foreign language. Terms like "matrix multiplication," "eigenvectors," and "gradient descent" make you anxious. You're smart and motivated, but you need to start from the actual beginning.

Your journey: 14 weeks, 4-5 hours/week

Your philosophy: Build intuition first, formalism later. Heavy visualization, immediate coding, constant "why should I care" answers.

Week 1-2: What Even Is a Vector?

Before touching any course, you need to see what these objects ARE.

🎥 3Blue1Brown: Essence of Linear Algebra (Videos 1-3)

Video 1: Vectors, what even are they?

What you'll learn: Vectors aren't scary math symbols. They're arrows. They have direction and length. You can think of them as coordinates, as movements, or as lists of numbers. All three views are correct.

Why you care: Every data point in ML is a vector. A customer is a vector of [age, income, purchases]. An image is a vector of pixel values. Understanding this unlocks everything.

After watching: Open Python, create a simple vector as a list: v = [3, 4]. Plot it as an arrow using matplotlib. Make it click visually.

Video 2: Linear combinations, span, and basis vectors

What you'll learn: When you multiply a vector by a number and add vectors together, you're making "linear combinations." This is the fundamental operation in all of ML. The "span" is all possible combinations you can make.

Why you care: ML models make predictions by taking linear combinations of features. "Basis vectors" are like the coordinate system—the building blocks everything else is made from.

After watching: In Python, take two vectors and make different combinations. Plot them. See that you can reach different points by changing the multipliers.

Video 3: Linear transformations and matrices

What you'll learn: A matrix is a transformation machine. It takes vectors as input and spits out transformed vectors. It rotates, stretches, squishes, or flips space.

Why you care: Every neural network layer is a matrix transformation. Every image filter is a matrix. Once you see matrices as transformations, not just grids of numbers, ML makes so much more sense.

After watching: Create a 2x2 matrix in NumPy. Multiply it by several vectors. Plot the original vectors and transformed vectors side-by-side. Watch the transformation happen.

📚 Khan Academy: Vectors Basics

Work through: Vectors and spaces section

Focus on:

  • Vector intro (what they are, how to draw them)
  • Adding vectors graphically and algebraically
  • Multiplying vectors by scalars
  • Unit vectors

Why Khan Academy: Practice problems with instant feedback. Do 10-15 problems until adding and scaling vectors feels automatic.

Coding exercise: Implement vector addition and scalar multiplication in pure Python (no NumPy). Then learn the NumPy way. Understand both.

Week 3-4: Matrices as Data

Now you'll start seeing matrices everywhere in your work.

🎥 3Blue1Brown: Essence (Videos 4-7)

Video 4: Matrix multiplication as composition

What you'll learn: Multiplying matrices is like doing one transformation, then another. The order matters! AB ≠ BA.

Why you care: Neural networks are compositions of transformations. Understanding this makes backpropagation less mysterious.

After watching: Create two transformation matrices. Apply them in both orders to a vector. See the different results.

Video 5: Three-dimensional linear transformations

What you'll learn: Everything extends to 3D and beyond. A 3x3 matrix transforms 3D space. Same principles.

Why you care: Real data has hundreds or thousands of dimensions. This shows you the geometric intuition carries over.

Video 6: The determinant

What you'll learn: The determinant tells you how much a transformation stretches or squishes space. If it's zero, the transformation squishes everything into a lower dimension.

Why you care: When training ML models fails, often the determinant of some matrix is near zero. Now you'll understand why that's a problem.

Video 7: Inverse matrices, column space and null space

What you'll learn: The inverse "undoes" a transformation. The column space is all possible outputs. The null space is inputs that get squished to zero.

Why you care: This is fundamental to understanding what ML models CAN and CANNOT learn.

💻 Hands-On Project: Image Transformation

Build this: Write code that loads an image, represents it as a matrix, and applies transformations:

  1. Rotation (rotation matrix)
  2. Scaling (diagonal matrix)
  3. Shearing (off-diagonal elements)

Tools: Python with NumPy, matplotlib, PIL/Pillow

Why: You'll SEE transformations happen to real data. This cements the geometric intuition.

Starter code structure:

python

import numpy as np
from PIL import Image
import matplotlib.pyplot as plt

# Load image, convert to matrix
# Create transformation matrix
# Apply transformation
# Display before/after

Week 5-7: Gentle Introduction to Real Linear Algebra

Now you're ready for actual instruction, but we'll be very selective.

🎓 Gilbert Strang 18.06: Selected Lectures (Watch at 1.5x speed)

Important: You're NOT watching the full course. Just these specific lectures, in this order.

Lecture 1: The Geometry of Linear Equations

What you'll learn: Systems of equations can be viewed three ways. The "column picture" view is the most important for ML.

Before watching: Review 3Blue1Brown videos 1-3

After watching: Solve a small 2x2 system by hand, then with NumPy's np.linalg.solve(). Understand both.

Why you care: Every time an ML model "fits" data, it's solving a system of equations.

Lecture 2: Elimination with Matrices

What you'll learn: How to systematically solve equations using elimination. This is what computers actually do.

After watching: Implement simple Gaussian elimination in Python (don't worry about edge cases, just the basic algorithm).

Why you care: Understanding the algorithm helps you know when solving systems will be easy or hard.

Lecture 3: Multiplication and Inverse Matrices

What you'll learn: Multiple ways to think about matrix multiplication. Each view is useful in different contexts.

After watching: Implement matrix multiplication yourself (nested loops). Time it. Then use NumPy and see the speedup. Appreciate optimization!

Skip to Lecture 15: Projections onto Subspaces

What you'll learn: Projection is finding the closest point in a subspace. This is the geometric meaning of linear regression.

Why you care: When you "fit a line" to data, you're projecting onto the space of all possible lines.

After watching: Implement simple linear regression from scratch using the projection formula. Compare with scikit-learn's LinearRegression.

Lecture 16: Projection Matrices and Least Squares

What you'll learn: Least squares minimizes error. The formula P = A(A^T A)^(-1)A^T looks scary but has beautiful geometric meaning.

Why you care: This is the foundation of regression, the most-used ML technique in industry.

After watching: Build a least-squares solver. Test it on real data (housing prices, anything from Kaggle).

📚 Khan Academy: Practice Problems

Work through:

Goal: Build computational fluency. Do 20-30 problems total.

Week 8-10: Machine Learning Connections

Now we connect everything to actual ML.

🎥 3Blue1Brown: Essence (Videos 14-15)

Video 14: Eigenvectors and eigenvalues

What you'll learn: Eigenvectors are special vectors that only get stretched, never rotated, by a transformation. Eigenvalues tell you the stretch factor.

Why you care: Principal Component Analysis (PCA) finds eigenvectors of your data. These are the "most important directions."

After watching: Compute eigenvectors of a 2x2 matrix by hand, then with NumPy. Visualize them.

Video 15: Abstract vector spaces

What you'll learn: Functions can be vectors! Polynomials can be vectors! This generalizes everything.

Why you care: Neural networks operate in abstract vector spaces. This video expands your thinking.

🎓 Strang 18.065: ML Applications (Selected Lectures)

These lectures are specifically about ML. Watch carefully.

Lecture 1: The Column Space of A Contains All Vectors Ax

What you'll learn: The column space is all possible predictions your model can make. If the target isn't in the column space, you need more features or a different model.

Why you care: This explains why sometimes adding more data doesn't help—you need different features.

After watching: Take a dataset, compute its column space (using SVD). Visualize which outputs are reachable.

Lecture 6: Singular Value Decomposition (SVD)

What you'll learn: SVD breaks any matrix into rotation × stretch × rotation. It's the most important matrix factorization in ML.

Why you care: Image compression, recommender systems, dimensionality reduction, noise removal—all use SVD.

After watching: Compress an image using SVD. Keep only top 10, 20, 50 singular values. See the quality-size tradeoff.

Code example:

python

U, S, Vt = np.linalg.svd(image_matrix)
# Keep top k values
compressed = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]

Lecture 7: Eckart-Young: The Closest Rank k Matrix to A

What you'll learn: SVD gives you the mathematically optimal low-rank approximation. You can't do better.

Why you care: PCA, recommendation systems, and many compression techniques are just applications of this theorem.

Week 11-12: Neural Networks Basics

Time to understand deep learning.

🎥 3Blue1Brown: Neural Networks Series

This series is INCREDIBLE for intuition. Watch all of it (4 videos, ~1 hour).

  • Video 1: What neural networks are
  • Video 2: Gradient descent
  • Video 3: Backpropagation intuition
  • Video 4: Backpropagation calculus

After watching: You'll understand that neural networks are just:

1. Matrix multiplication

2. Non-linear activation

3. Repeat

4. Compute gradient

5. Update weights

🎓 Strang 18.065: Neural Network Lectures

Lecture 27: Backpropagation: Find Partial Derivatives

What you'll learn: Backpropagation is just the chain rule, applied to matrices. It's not magic.

Why you care: Every deep learning framework (PyTorch, TensorFlow) does this automatically. But understanding it helps you debug.

After watching: Implement a tiny 2-layer network from scratch. Forward pass and backward pass. No frameworks, just NumPy.

Week 13-14: Final Project & Real ML

💻 Capstone Project: Build Something Real

Choose one (or do multiple if you're motivated):

Option 1: Handwritten Digit Recognition

  • Dataset: MNIST (included in scikit-learn)
  • Build: Implement simple neural network from scratch with NumPy
  • Then: Use scikit-learn's MLPClassifier, compare performance
  • Visualize: Plot the weight matrices, see what patterns it learned

Option 2: Movie Recommendation

  • Dataset: MovieLens 100K
  • Build: Matrix factorization using SVD
  • Implement: Basic collaborative filtering
  • Evaluate: Predict ratings for held-out movies

Option 3: Image Compression

  • Dataset: Your own photos
  • Build: SVD-based compression with adjustable quality
  • Create: Interactive tool where user can slide quality slider
  • Visualize: Singular values drop-off, reconstruction at different ranks

Option 4: Linear Regression Deep Dive

  • Dataset: Boston Housing or California Housing
  • Build: Linear regression three ways: (1) normal equations, (2) gradient descent from scratch, (3) scikit-learn
  • Compare: Speed, accuracy, when each method works best
  • Visualize: Gradient descent path in parameter space

🎯 Tools & Resources Checklist

You should now be comfortable with:

  • ✅ NumPy for matrix operations
  • ✅ Matplotlib for visualization
  • ✅ Scikit-learn basics (fit, predict, transforms)
  • ✅ Reading ML documentation and understanding the math
  • ✅ Implementing simple algorithms from scratch
  • ✅ Debugging ML code (shapes, dimensions, gradients)

Next steps after this path:

  1. Deepen ML: Take Andrew Ng's ML course on Coursera (now the math won't scare you!)
  2. Deep learning: Fast.ai course (practical deep learning)
  3. Practice: Kaggle competitions (start with "Getting Started" competitions)
  4. Have a look to Path 2: Beatriz - Rapid Learner

Primary Video Resources:

Practice & Supplements:

Textbooks (Optional):

Coding Resources:

💡 Study Strategies for Self-Taught Learners

Dealing with Math Anxiety:

  • It's normal to not understand something the first time
  • If stuck for 30+ minutes, move on and return later
  • Post questions on r/learnmachinelearning or r/learnmath
  • Watch videos at different speeds (0.75x for hard parts, 1.5x for review)

Time Management (4-5 hours/week):

  • 2 hours: Video lectures (weekday evenings, 30-min sessions)
  • 1.5 hours: Practice problems (can be broken up)
  • 1.5 hours: Coding projects (weekend mornings)

Active Learning Techniques:

  • Code immediately: After learning a concept, code it that same day
  • Teach it: Explain to a friend, rubber duck, or write a blog post
  • Visualize everything: Plot vectors, matrices, transformations constantly
  • Test yourself: Before watching a solution, try the problem yourself

Common Pitfalls to Avoid:

  • ❌ Watching videos passively without coding
  • ❌ Trying to memorize formulas without understanding
  • ❌ Skipping the practice problems
  • ❌ Comparing yourself to CS PhD students (you're on your own path!)
  • ❌ Getting discouraged by what you don't know (focus on progress)

Building Confidence:

  • Keep a "wins" journal: Every day, write one thing you understood
  • Revisit week 1 materials after finishing—you'll be amazed how easy they seem
  • Join study groups (Discord, Reddit, local meetups)
  • Remember: Every expert was once a beginner

When You Feel Stuck:

  1. Re-watch the 3Blue1Brown video on that topic
  2. Try coding a simpler version of the concept
  3. Draw pictures and diagrams
  4. Ask on StackOverflow or math.stackexchange.com
  5. Take a break—sometimes your brain needs processing time

Community Resources:

You've got this! Linear algebra seemed impossible at first, but in 14 weeks you'll understand the math behind AI. You're building a superpower. 🚀