Machine Learning — Complete Course from the 21 Lecture PDFs

Source folder: ML_Notes (Dr. Tanmay Basu’s Machine Learning, Summer 2024 lecture notes, plus the Bishop PCA excerpt). All 21 PDFs were processed and unified into 54 concepts across 10 modules. Each concept has: intuition, formal definition, mathematics/derivation, key insights, exactly 5 worked examples (very easy → very hard), common mistakes, exam traps, connections, mastery checklist.

How to read. Follow the modules in order — each assumes the previous ones. Example 5 of each concept is the “exam-level” one. Most numerical answers (all the data-heavy ones: trees, Bayes/NB, regression, SVM, clustering, PCA, perceptron/backprop) were recomputed with a script; the remaining simple ones were checked by hand. If you spot a slip, treat the derivation, not the final number, as authoritative.

Course map

#	File	Concepts	Source lectures
0	`01_Foundations.md`	C1 What is ML · C2 Supervised (classification/regression) · C3 Unsupervised & feature engineering	Lec 2
1	`02_Decision_Trees_and_Random_Forests.md`	C4 Entropy & IG · C5 Gini · C6 Gain ratio · C7 Stopping & overfitting · C8 Pruning (REP, CCP, EBP) · C9 ID3/C4.5/CART, merits & limits · C10 Random forest	Lec 3–4
2	`03_Evaluation_and_Model_Selection.md`	C11 Confusion matrix, accuracy, P/R/F · C12 Macro/micro, sensitivity/specificity, imbalance · C13 Cross-validation (LOO, $k$ -fold)	Lec 5, 6
3	`04_Bayes_Decision_Theory_and_Naive_Bayes.md`	C14 Bayes rule · C15 Bayes error & optimality · C16 Chernoff/Bhattacharyya · C17 Minimum distance classifier · C18 Linear discriminant · C19 Naive Bayes · C20 Gaussian NB · C21 Multinomial NB + Laplace · C22 Bernoulli NB	Lec 7, 8
4	`05_k_Nearest_Neighbour.md`	C23 kNN · C24 Distance-weighted kNN · C25 Tweaked kNN	Lec 9
5	`06_Regression_and_Logistic_Models.md`	C26 Linear regression · C27 Logistic regression · C28 Softmax · C29 Generative vs discriminative	Lec 10, 11
6	`07_Support_Vector_Machines.md`	C30 Hard-margin geometry · C31 Lagrange dual & KKT · C32 Soft margin · C33 Kernel trick · C34 Valid kernels	Lec 13, 14, 15
7	`08_Clustering.md`	C35 Clustering formulation · C36 k-means · C37 Bisecting & Buckshot · C38 k-medoids · C39 Agglomerative linkages · C40 Divisive & DBSCAN · C41 Cluster F-measure · C42 NMI	Lec 16–19
8	`09_Principal_Component_Analysis.md`	C43 Orthonormal bases · C44 PCA max-variance · C45 PCA min-error · C46 Algorithm, standardisation, whitening · C47 PCA for $N < D$ · C48 Applications & Fisher comparison	Lec 20, Bishop §12.1
9	`10_Perceptrons_and_Neural_Networks.md`	C49 Perceptron · C50 Convergence theorem · C51 Batch perceptron · C52 XOR & MLP · C53 Feedforward architecture · C54 Backpropagation	Lec 21, 22, 23

PDF Coverage Checklist

PDF	Topics Covered	Concepts Covered (where)	Status
Lec2 Introduction to Machine Learning	Samuel/Mitchell definitions; supervised (classification, regression); unsupervised (clustering, density estimation); feature selection vs extraction; applications	C1–C3 (`01`)	✓ Complete
Lec3-4 Decision Tree and Random Forest Classifiers	Tree structure; weather dataset; entropy/IG; Gini; gain ratio; stopping criteria; overfitting; REP, cost-complexity, error-based pruning; ID3, C4.5, CART; merits/limits; random forest, OOB (all figures 3.1–3.5 checked)	C4–C10 (`02`); weather data worked end-to-end	✓ Complete
Lec5 Evaluation Criteria for Data Classification	Confusion matrix; accuracy; precision/recall/F; macro vs micro averaging; class imbalance; sensitivity/specificity	C11–C12 (`03`)	✓ Complete
Lec6 Parameter Tuning for Data Classification	Cross-validation; leave-one-out; $k$ -fold (fold conditions i–iv); repeated runs	C13 (`03`)	✓ Complete
Lec7 Bayes Decision Rule for Data Classification	Priors/posteriors; Bayes rule; error probability and optimality; equal-variance Gaussian error $Φ (\cdot)$ ; $min (u, v) \leq u^{α} v^{1 - α}$ ; Chernoff/Bhattacharyya; minimum distance classifier; linear discriminant	C14–C18 (`04`)	✓ Complete
Lec8 Naive Bayes Classifier	Conditional independence; MAP; log-space; Gaussian, multinomial (Laplace), Bernoulli NB	C19–C22 (`04`)	✓ Complete
Lec9 k-Nearest Neighbor Decision Rule	kNN algorithm; choice of $k$ ; 1-NN vs Bayes error; distance-weighted kNN; TkNN with $β$ ; the 9-element sequence example	C23–C25 (`05`)	✓ Complete
Lec10 Linear Regression	Least squares; normal equations; basis expansions	C26 (`06`)	✓ Complete
Lec11 Logistic Regression	Sigmoid; likelihood/cross-entropy; gradient; softmax; one-hot; discriminative vs generative	C27–C29 (`06`)	✓ Complete
Lec13 SVM (Hard Margin)	Separability/convex hull; margin $2/∥ W ∥$ ; support vectors; Lagrangian, dual, KKT; computing $W, b$	C30–C31 (`07`)	✓ Complete
Lec14 SVM (Soft Margin)	Slack variables; objective with $C$ ; Lagrangian with $α, μ$ ; box constraint; KKT cases; role of $C$	C32 (`07`)	✓ Complete
Lec15 SVM Using Kernels	Feature map; kernel trick; polynomial-kernel derivation; symmetric/PSD/Mercer; polynomial, Gaussian, sigmoid kernels; RBF network view; the $ϕ$ example with $(8, 10)$ etc.	C33–C34 (`07`)	✓ Complete
Lec16 Overview of Data Clustering	Partition axioms; hierarchical vs partitional; loss $L$ ; convex-shaped clusters; half-spaces	C35 (`08`)	✓ Complete
Lec17 Partitional Clustering Technique	Lloyd’s k-means; MacQueen variant; bisecting k-means; Buckshot; k-medoids (two medoid definitions)	C36–C38 (`08`)	✓ Complete
Lec18 Hierarchical Clustering Technique	Agglomerative; single/complete/average linkage; chaining; divisive; DBSCAN	C39–C40 (`08`)	✓ Complete
Lec19 Evaluation Criteria for Clustering	Cluster F-measure; mutual information; NMI	C41–C42 (`08`)	✓ Complete
Lec20 PCA for Dimensionality Reduction	PCA algorithm; orthonormal bases; min-error derivation; Lagrangian; eigen-decomposition; appendix expansion	C43–C46 (`09`)	✓ Complete
PCA_Bishop_Book (§12.1, pp. 561–570)	Max-variance formulation; min-error formulation; applications (compression, pre-processing, standardising, whitening, visualisation); PCA vs Fisher; PCA for $N < D$ ; probabilistic PCA (introduced)	C44–C48 (`09`)	✓ Complete
Lec21 Perceptron Learning Algorithm	Perceptron; update rule; convergence proof; salient points; batch version and cost $J$	C49–C51 (`10`)	✓ Complete
Lec22 Multilayer Perceptron Learning Algorithm	OR/AND perceptrons; XOR problem; two-layer perceptron; hidden-layer mapping	C52 (`10`)	✓ Complete
Lec23 ANN and Backpropagation	Architecture; notation; forward computation; squared-error risk; $γ$ recursion; backpropagation algorithm; remarks	C53–C54 (`10`)	✓ Complete

Method notes (honesty about the processing)

Text of every PDF was extracted programmatically and read in full. Figures were spot-checked as images (Lec 3-4 Figs 3.3/3.5; Lec 16, 18, 22, 23 figures); others were reconstructed from captions and surrounding text, and all agree with the written content.
The Bishop excerpt is a scanned page image whose OCR text is garbled; its mathematics (eqs. 12.1–12.30) was rebuilt and cross-checked against Lec 20, which derives the same results.
§12.2 “Probabilistic PCA” is only introduced on the last page provided; it is described but not developed.

Source errata and clarifications (corrected silently in the text; listed here for transparency)

Lecture	Issue in the notes	Correct statement used here
3–4	$E (X) = - \frac{9}{5} lo g \frac{9}{5} - \dots$	$E = - \frac{9}{14} lo g \frac{9}{14} - \frac{5}{14} lo g \frac{5}{14} = 0.940$
3–4	Gain-ratio denominator written $E (X, f)$	Split information $-\sum\frac{
3–4	”Gini index is used to prune” (CART)	CART splits by Gini, prunes by cost-complexity
3–4	" $I G (w in d) = 0.0.048$ "	$0.048$
3–4	Random forest “low variance and bias”	Mainly reduces variance; bias stays about that of a deep tree
7	”minimum of posteriors” in $R (h)$	Minimum of the joint $p (w_{i}) f_{i}$
9	1-NN error “twice Bayes”	$\leq 2 R^{} (1 - R^{})$ (as $n \to \infty$ )
11	”loss function is linear in $β$ “	Boundary linear; loss convex (not linear); sigmoid range is $(0, 1)$ for $x \in (- \infty, \infty)$
13	Min $\frac{1}{2} W^{T} W$ called the “dual”	Equivalent primal; true dual is $L_{D} (α)$ ; " $0 \leq α \leq \infty$ " ⇒ $α \geq 0$
13/14	” $α > 0$ only for SVs”; margin violators “are not SVs”	All points with $α_{i} > 0$ are SVs; $α_{i} = C$ are violators; a point on the margin can have $α_{i} = 0$
14	$C \in (0, α]$	$C \in (0, \infty]$
15	Gaussian kernel $e^{+ ∥ X - Z ∥^{2} /2 σ^{2}}$	$e^{- ∥ X - Z ∥^{2} /2 σ^{2}}$ (otherwise not PSD)
15	Example lists “positive class” twice; $ϕ^{'}$ 7-vector has a repeated component; polynomial kernel " $p > 2$ "	Second list is the negative class; verify $ϕ^{'}$ before use; $p \geq 1$ integer (the lecture uses $p = 2$ )
17	Medoid defined as minimax and as minimum-average-distance	Both forms discussed; they can differ on ties
20	$Z^{'} = Z E^{T}$ ; “least $(d - m)$ eigenvalues”; independence definition garbled	$Z^{'} = Z E$ ; least $(m - d)$ ; independence ⇔ only the trivial combination is $0$
21	”Cannot find a bound on the number of iterations”	A bound $R^{2} ∥ W^{*} ∥^{2} / γ^{2}$ exists but depends on the unknown margin; sign in 21.15 should give $W + η \sum y X$
23	Mixed indexing in $S_{j}^{l}$ ; (23.6) per-sample vs (23.7) summed	Online vs batch gradient descent; $γ^{L} = (y - z) f^{'}$ assumes squared error

One-page formula sheet

Information. $E = - \sum p_{i} lo g_{2} p_{i}$ ; $I G = E (X) - \sum \frac{∣ X _{v} ∣}{∣ X ∣} E (X_{v})$ ; $G ini = 1 - \sum p_{i}^{2}$ ; $GR = I G / S pl i t I n f o$ . Evaluation. $P = \frac{T P}{T P + F P}$ , $R = \frac{T P}{T P + F N}$ , $F = \frac{2 T P}{2 T P + F P + F N}$ ; spec $= \frac{T N}{T N + F P}$ . Bayes. decide $ar g max_{i} p (w_{i}) p (X ∣ w_{i})$ ; equal-var Gaussian error $Φ (- d /2)$ ; NB: $ar g max [ln p (w_{k}) + \sum ln p (x_{i} ∣ w_{k})]$ ; Laplace $\frac{F _{ik} + 1}{\sum F _{j k} + m}$ . kNN. vote of $k$ nearest; weights $\frac{d _{k} - d _{j}}{d _{k} - d _{1}}$ ; TkNN: stop when margin $= β$ . Regression. $β = (Z^{T} Z)^{- 1} Z^{T} Y$ ; logistic $\partial L / \partial β_{j} = \sum x_{ij} (p_{i} - y_{i})$ ; softmax gradient $\overset{y}{^} - y$ . SVM. margin $\frac{2}{∥ W ∥}$ ; $W = \sum α_{i} y_{i} X_{i}$ ; $\sum α_{i} y_{i} = 0$ ; $0 \leq α_{i} \leq C$ ; $K = (1 + X^{T} Z)^{p}$ , $e^{- ∥ X - Z ∥^{2} /2 σ^{2}}$ . Clustering. Lloyd: assign nearest, update means; linkages min/max/mean; NMI $= \frac{M I}{E ( R ) E ( S )}$ . PCA. $S u = λ u$ ; $J_{m i n} = \sum_{j > M} λ_{j}$ ; $Z^{'} = Z E$ ; whitening $Λ^{- 1/2} U^{T} (X - \overset{ˉ}{X})$ ; $N < D$ : $u_{i} = X^{T} v_{i} / N λ_{i}$ . Neural nets. perceptron $W \leftarrow W + y X^{'}$ ; mistake bound $R^{2} ∥ W^{*} ∥^{2} / γ^{2}$ ; backprop $γ^{L} = (y - z) f^{'}$ , $γ_{j}^{l} = f^{'} (S_{j}^{l}) \sum_{r} γ_{r}^{l + 1} w_{j r}^{l}$ , $\partial J / \partial w_{ij}^{l} = γ_{j}^{l + 1} y_{i}^{l}$ .

Vault

Explorer

00_INDEX_and_Coverage