01_Foundations

Module 0 — Foundations of Machine Learning

Source: Lec 2 — Introduction to Machine Learning. · Next: Module 1 — Decision Trees · Index: 00_INDEX_and_Coverage

Notation used throughout the course

• Data point: ${\vec{X}}_i=[x_{i1},\dots ,x_{im}{]}^T\in {\mathbb{R}}^m$ ($m$ features).
• $n$ = number of training instances. $y_i$ = class label (or target). ${\vec{X}}_0$ = test point.
• Labels: logistic regression uses $\{0,1\}$; SVM and perceptron use $\{-1,+1\}$.

How each concept is laid out

Intuition → definition → maths → insights → 5 worked examples (very easy → very hard) → mistakes → exam traps → connections → checklist. Each hard example has a collapsible Python check so you can reproduce the answer yourself.

---

C1. What Is Machine Learning? (Samuel, Mitchell, the three paradigms)

Intuition

A normal program = rules written by a human. A learning program = rules extracted from examples. A child learns “dog” by meeting many dogs, not by reading a definition. ML is that, but with a score you can measure.

flowchart LR
    D[Data] --> Q{What feedback<br/>does the learner get?}
    Q -->|correct answer<br/>per example| S[Supervised<br/>classification · regression]
    Q -->|no answers| U[Unsupervised<br/>clustering · density estimation]
    Q -->|delayed reward| R[Reinforcement<br/>mentioned only]

Formal definitions

• Samuel (1959): ML is the field of study that gives computers the ability to learn without being explicitly programmed.
• Mitchell (1997): A program learns from experience $E$ with respect to a task $T$ and performance measure $P$ if its performance on $T$, measured by $P$, improves with experience $E$.
• Paradigms: Supervised (labelled data), Unsupervised (no labels), Reinforcement (reward signal; mentioned, not studied in this course).

Mathematical formulation

Let $P_T(E)$ denote performance on task $T$ after experience $E$. Learning with respect to $(T,P)$ means: for experiences $E_1\subset E_2$, $P_T(E_2)\ge P_T(E_1)\text{(typically strictly, and measured on data not used for training).}$

Derivation / reasoning

Why must $P$ be measured on unseen data? If $P$ is measured on the training set, a lookup table gets a perfect score with zero “understanding”. Mitchell’s definition is meaningful only when $P$ reflects the task in general, i.e. on new instances. This is the seed of the whole generalisation / overfitting theme (C7, C13).

Key insights

1. Always identify the triple $(T,P,E)$ before choosing an algorithm.
2. $E$ determines the paradigm: labelled → supervised; unlabelled → unsupervised.
3. “Improves with experience” is a testable, quantitative statement.

Worked examples

Example 1 — Very Easy · The spam filter

• Problem: An email app learns to flag spam by watching which emails the user drags to the spam folder. Identify $T$, $P$, $E$.
• Asked: The three components.
• Concept: Mitchell’s definition.
• Approach: Task = what the program does; experience = what it watches; performance = how success is scored.
• Solution: $T$ = classify incoming emails as spam / not spam. $E$ = a database of emails with the user’s spam / not-spam labels. $P$ = fraction of new emails classified correctly.
• Final answer: $T$: spam classification; $E$: labelled emails; $P$: accuracy on new emails.

Key lesson

The three parts are three different objects; don’t merge them.

Example 2 — Easy · Name the paradigm

• Problem: Label each as supervised, unsupervised or reinforcement: (a) predicting house prices from past sales with sale prices; (b) a shop grouping customers by purchase behaviour, no labels; (c) a robot learning to walk, receiving $+1$ when it moves forward and $-1$ when it falls; (d) tagging photos using 10 000 photos already tagged by humans.
• Asked: Paradigm for each.
• Concept: Paradigms are defined by the type of feedback.
• Approach: Ask “what feedback does the learner receive?”
• Solution: (a) Each training house has its price (the target) → supervised (regression). (b) No target → unsupervised (clustering). (c) Feedback is a delayed scalar reward, not a correct answer → reinforcement. (d) Human tags are the answer key → supervised (classification).
• Final answer: (a) supervised, (b) unsupervised, (c) reinforcement, (d) supervised.

Key lesson

Supervision = a correct answer per example. A reward only says “good/bad”, not “what you should have done”.

Example 3 — Moderate · The digit recogniser’s learning curve

• Problem: A handwritten-digit program is evaluated on a fixed test set of 1000 images. After training on $E_1=500$ images its test accuracy is 71 %; after $E_2=2000$ images, 88 %; after $E_3=8000$ images, 87.5 %. Does it “learn” per Mitchell? Over which range?
• Asked: Whether $P$ improves with $E$.
• Concept: Mitchell’s definition (improvement, not necessarily monotone forever).
• Approach: Compare $P$ at successive experience levels, and compare each difference with sampling noise.
• Solution: $P(E_1)=0.71$, $P(E_2)=0.88$, $P(E_3)=0.875$. $P(E_2)-P(E_1)=0.17>0$: clear improvement. $P(E_3)-P(E_2)=-0.005$: a tiny decline, smaller than one standard error of a 1000-image test set, $\sqrt{0.88\cdot 0.12/1000}\approx 0.010$.
• Final answer: Yes, it learns from 500 → 2000 images; from 2000 → 8000 the change is statistically indistinguishable from zero (performance has plateaued).

accuracy
 90% |            ●────────●   ← plateau (difference < 1 s.e.)
 80% |
 70% |   ●
     +---+--------+--------+---→ training images
        500     2000     8000

Key lesson

Learning curves saturate. Compare differences with sampling noise before claiming “worse”.

Example 4 — Hard · Memoriser vs generaliser

• Problem: Two programs are trained on the same 1000 labelled digit images. Program A stores every training image with its label; on a test image it returns the stored label if an identical image exists, otherwise it guesses “0”. Program B fits a smooth classifier. Results — A: train 100 %, test 10 %. B: train 94 %, test 92 %. Which has learned in Mitchell’s sense, and what should $P$ be?
• Asked: Which program learned; what $P$ must measure.
• Concept: $P$ must be measured on unseen instances of $T$.
• Approach: Evaluate the two programs under both choices of $P$.
• Solution: (i) If $P$ = training accuracy: A scores 100 %, B 94 % → A “wins” — absurd, A does not generalise. (ii) If $P$ = test accuracy: A = 10 % (a random guess among 10 digits), B = 92 %. A improves with experience only if a test image happens to equal a stored one; for new handwriting, no. B improves with experience.
• Final answer: B has learned; the correct $P$ is accuracy on unseen instances.

Key lesson

Memorisation is not learning; generalisation is the point.

Example 5 — Very Hard · The leaking test set

• Problem: A team reports accuracy on a fixed 200-point test set for training sizes $n=100,400,1600$. Program X: $0.60,0.72,0.80$. Program Y: $0.70,0.70,0.70$. Program Z: $0.55,0.60,0.58$. Z’s test set accidentally contains 20 points that also appear in the training set once $n=400$. (a) Which programs learn? (b) What is Z’s accuracy on the non-leaked 180 test points at $n=400$, assuming Z gets all 20 leaked points right?
• Asked: Learning judgement and corrected accuracy.
• Concept: Mitchell’s definition + leakage invalidates $P$.
• Approach: (a) look at the trend; (b) remove leaked points from numerator and denominator.
• Solution: (a) X: strictly increasing → learns. Y: constant → does not improve → not learning (a fixed rule). Z: noisy and non-monotone, $0.55\to 0.60\to 0.58$. (b) Correct answers at $n=400$: $0.60\times 200=120$. Remove the 20 leaked (all correct): $120-20=100$ correct among $180$ clean points $\Rightarrow 100/180=0.5556$. At $n=100$ there is no leakage: $0.55$. So clean accuracy goes $0.55\to 0.556$: practically no improvement; Z’s apparent gain was leakage.
• Final answer: X learns; Y and Z do not (Z’s clean accuracy $\approx 0.556$ vs $0.55$).

Python check

import numpy as np
print("s.e. (Ex 3):", np.sqrt(0.88 * 0.12 / 1000))      # 0.0103
print("Z clean accuracy (Ex 5):", (0.60 * 200 - 20) / 180)  # 0.5556

Key lesson

A test set contaminated by training data silently inflates $P$. Always check that train and test are disjoint.

Common mistakes

• Treating training accuracy as performance.
• Saying “learning requires labels” (unsupervised learning has none).

Exam traps

• “Define ML” questions: give both Samuel and Mitchell; Mitchell is the formal one.
• The definition says improves; a constant program does not learn.

Connections: Evaluation (C11–C13), overfitting (C7), supervised / unsupervised split (C2–C3 below).

Mastery checklist

• State Samuel’s and Mitchell’s definitions.
• Identify $(T,P,E)$ for any story.
• Explain why $P$ must be on unseen data.

---

C2. Supervised Learning: Classification vs Regression

Intuition

Supervised learning = learning with an answer key. You get $(X_i,y_i)$ pairs and must predict $y$ for new $X$. If the answer is a category (spam / not spam) it is classification; if it is a number (price) it is regression.

flowchart LR
    T[(Training pairs<br/>X_i, y_i)] --> L[Learner]
    L --> H["hypothesis h(X)"]
    N[New X_0] --> H
    H --> P["prediction ŷ"]
    P -.compare on held-out data.-> E[estimate of true risk]

Formal definition

Given training data $\{({\vec{X}}_i,y_i){\}}_{i=1}^n$, ${\vec{X}}_i\in {\mathbb{R}}^m$, learn a function $h$ (the inferred function) so that $h({\vec{X}}_0)$ estimates the target of a new ${\vec{X}}_0\in {\mathbb{R}}^m$.

• Classification: $y_i\in \{1,\dots ,c\}$ (discrete).
• Regression: $y_i\in \mathbb{R}$ (continuous).

Mathematical formulation

Empirical risk for loss $\ell$: $\hat{R}(h)=\frac{1}{n}{\sum }_{i=1}^n\ell (h({\vec{X}}_i),y_i),$ with ${\ell }_{0\text{-}1}(\hat{y},y)=1[\hat{y}\ne y]$ for classification and ${\ell }_{sq}(\hat{y},y)=(\hat{y}-y{)}^2$ for regression.

Derivation

Why not just minimise $\hat{R}$ to zero? A lookup table does it. We want small true risk $R(h)=\mathbb{E}[\ell (h(X),y)]$; $\hat{R}$ is only an estimate, so we evaluate on held-out data.

Key insights

• Same input, different target type → different algorithm family.
• Every classifier in this course (trees, Bayes, kNN, logistic, SVM, perceptron, NN) fits this template.

Worked examples

Example 1 — Very Easy · Classification or regression?

• Problem: Classify as classification or regression: (a) predict a patient’s blood-sugar level; (b) predict whether a tumour is malignant; (c) predict tomorrow’s rainfall in mm; (d) predict whether it will rain tomorrow.
• Asked: Task type.
• Concept: Target type.
• Approach: Discrete vs continuous target.
• Solution: (a) blood sugar is a real number → regression. (b) malignant / benign → classification. (c) mm of rain is a real number → regression. (d) rain / no rain → classification.
• Final answer: (a) regression; (b) classification; (c) regression; (d) classification.

Key lesson

Look at $y$, not $X$. The same weather features can feed either task.

Example 2 — Easy · Reading a data table

• Problem: A shop records (age, income in ₹ thousand) and whether the customer buys: $(25,30)\to$no, $(40,80)\to$yes, $(35,60)\to$yes. State $n$, $m$, ${\vec{X}}_2$, $y_2$.
• Asked: Notation.
• Concept: Training-set notation.
• Approach: Count rows and feature columns.
• Solution: Rows = 3 → $n=3$. Feature columns = 2 → $m=2$. ${\vec{X}}_2=[40,80{]}^T$, $y_2=\text{yes}$.
• Final answer: $n=3,m=2,{\vec{X}}_2=[40,80{]}^T,y_2=\text{yes}$.

Key lesson

$m$ counts features only; the target is not a feature.

Example 3 — Moderate · Scoring a study-time model

• Problem: Hours studied $x$ vs marks out of 10 $y$: $(0,1),(1,3),(2,5),(3,8)$. A tutor’s rule is $h(x)=2x+1$. Compute the empirical squared loss ${\hat{R}}_{sq}$ and the number of points a rounding classifier (”$h(x)$ equals $y$”) gets right.
• Asked: ${\hat{R}}_{sq}(h)$ and the match count.
• Concept: Empirical risk.
• Approach: Compute predictions and residuals.
• Solution: $h(0)=1,h(1)=3,h(2)=5,h(3)=7$. Residuals $y-h$: $0,0,0,1$. Squared: $0,0,0,1$. ${\hat{R}}_{sq}=\frac{1}{4}(0+0+0+1)=0.25$. Matches: first three equal, fourth $7\ne 8$ → 3 correct of 4.
• Final answer: ${\hat{R}}_{sq}=0.25$; 3 / 4 correct.

Key lesson

The same predictions can be scored by different losses (0.25 squared loss vs 25 % 0-1 error).

Example 4 — Hard · A regressor that acts as a classifier

• Problem: House prices (₹ lakh): sizes (m²) $50,60,70,80,90$; prices $30,34,41,45,50$. A manager wants “cheap” ($<40$) vs “expensive”. Option 1: regress price, then threshold at 40. Option 2: train a classifier directly. The regressor is $\hat{p}(s)=0.5s+5$. Compute the threshold classifier’s accuracy and find the decision boundary $s^{\ast }$.
• Asked: Accuracy and boundary.
• Concept: A regressor induces a classifier.
• Approach: Predict, threshold, compare with true labels.
• Solution: Predictions: $s=50\to 30$; $60\to 35$; $70\to 40$; $80\to 45$; $90\to 50$. True labels (price $<40$ → cheap): $30$ C, $34$ C, $41$ E, $45$ E, $50$ E. Predicted labels with rule “predicted $<40$ is cheap”: $30$ C, $35$ C, $40$ E (not $<40$), $45$ E, $50$ E → all five match → accuracy $5/5$. Boundary: $0.5s+5=40\Rightarrow s^{\ast }=70$ (ties at 70 go to expensive).
• Final answer: Accuracy 100 %, boundary $s^{\ast }=70$.

Python check

import numpy as np
s = np.array([50, 60, 70, 80, 90]); price = np.array([30, 34, 41, 45, 50])
pred = 0.5 * s + 5
print(pred, np.mean((pred < 40) == (price < 40)))   # [30. 35. 40. 45. 50.] 1.0

Key lesson

A regression model contains classification information, but the tie-breaking rule at the boundary must be fixed in advance.

Example 5 — Very Hard · The flashcard memoriser

• Problem: A “lookup” learner stores training pairs; for a test $x$ already stored it returns the stored $y$, otherwise it returns the majority training label. Training set: $(1,A),(2,A),(3,A),(4,B)$. Test set: $(1,A),(5,B),(6,B),(7,A)$. Compute training 0-1 risk, test 0-1 risk, and compare with the constant predictor “A”.
• Asked: Three error rates.
• Concept: Generalisation gap.
• Approach: Find the majority label first, then evaluate each point.
• Solution: Majority label = A (3 vs 1). Training: every stored point is returned exactly → 0 errors → risk 0. Test: $x=1$ stored → A, correct. $x=5$: unseen → A, true B → error. $x=6$: A, true B → error. $x=7$: A, true A → correct. Test risk $=2/4=0.5$. Constant “A”: test predictions all A → correct on $(1,A)$ and $(7,A)$ → risk 0.5. Training risk of constant A: 1 error of 4 → 0.25.
• Final answer: Lookup: train 0, test 0.5. Constant A: train 0.25, test 0.5. The lookup learner has a gap of 0.5 and no test advantage.

Learner	Train risk	Test risk	Gap
Lookup	0	0.5	0.5
Constant “A”	0.25	0.5	0.25

Key lesson

Zero training error can coexist with no generalisation. The gap is what matters.

Common mistakes

• Using the wrong loss (squared error on categorical labels).
• Counting the target as a feature.

Exam traps

• Labels coded $\{0,1\}$ vs $\{-1,+1\}$ change formulas (logistic uses $\{0,1\}$; SVM and perceptron use $\{-1,+1\}$).
• “Logistic regression” is a classifier.

Connections: Every later module. Regression → C26; classification → C4–C10, C14–C22, C23–C25, C27, C30–C34, C49+.

Mastery checklist

• Define classification and regression.
• Write the empirical risk for 0-1 and squared loss.
• Explain train vs test risk.

---

C3. Unsupervised Learning & Feature Engineering

Intuition

No answer key. Two things you can still do: group similar items (clustering) or describe how the data is spread (density estimation). Separately, you can clean up the features: keep the useful ones (selection) or build new, better ones (extraction).

flowchart TD
    F[Raw features] --> A{Use labels?}
    A -->|yes| S["Feature selection<br/>keep original columns<br/>χ² · information gain"]
    A -->|no| X["Feature extraction<br/>build new columns<br/>PCA · entropy-based"]

Formal definitions

• Clustering: partition data into groups; high similarity within, low across.
• Density estimation: estimate the distribution $p(\vec{X})$.
• Feature selection (supervised): score each feature using class labels (e.g. ${\chi }^2$ statistic, information gain), keep the top-scoring ones.
• Feature extraction (unsupervised): build new features from old ones without labels (e.g. PCA, entropy-based methods).

Mathematical formulation

${\chi }^2$ for a feature-vs-class contingency table with observed counts $O_{ij}$ and expected counts $E_{ij}=\frac{({\text{row}}_i)({\text{col}}_j)}{N}$: ${\chi }^2={\sum }_{i,j}\frac{(O_{ij}-E_{ij}{)}^2}{E_{ij}}.$ Information gain of a feature $f$ is defined in C4.

Derivation

Expected counts $E_{ij}$ are what we would see if feature and class were independent: $P({\text{row}}_i,{\text{col}}_j)=P({\text{row}}_i)P({\text{col}}_j)\Rightarrow E_{ij}=N\cdot \frac{r_i}{N}\cdot \frac{c_j}{N}=\frac{r_i{c}_j}{N}$. A large deviation from independence means the feature carries information about the class, so it gets a high score.

Key insights

• Selection keeps original features (interpretable); extraction creates new ones.
• Supervised ⇒ selection (uses labels); unsupervised ⇒ extraction (does not).
• Variance is not relevance (see Examples 4 and 5).

Worked examples

Example 1 — Very Easy · Which bucket is PCA?

• Problem: Is PCA feature selection or feature extraction? Is it supervised?
• Asked: Category.
• Concept: Taxonomy.
• Approach: PCA builds new axes and ignores labels.
• Solution: New features → extraction; no labels → unsupervised.
• Final answer: Unsupervised feature extraction.

Key lesson

Selection = keep; extraction = build.

Example 2 — Easy · Pick the best weather features

• Problem: For the play-tennis data, the information-gain scores are outlook 0.247, temperature 0.029, humidity 0.152, wind 0.048. Select the top 2 features.
• Asked: Top-2.
• Concept: Score-based selection.
• Approach: Sort descending.
• Solution: $0.247>0.152>0.048>0.029$. Top two: outlook, humidity.
• Final answer: {outlook, humidity}.

Key lesson

Selection is just ranking by a score and cutting at a budget.

Example 3 — Moderate · Does the word “free” signal spam?

• Problem: Among 100 emails: contains “free” & spam = 30, “free” & ham = 10, no “free” & spam = 20, no “free” & ham = 40. Compute ${\chi }^2$.
• Asked: ${\chi }^2$.
• Concept: ${\chi }^2$ feature score.
• Approach: Row / column totals → expected counts → sum.
• Solution: Row totals: present $=40$, absent $=60$. Column totals: spam $=50$, ham $=50$; $N=100$. $E_{11}=40\cdot 50/100=20$, $E_{12}=20$, $E_{21}=60\cdot 50/100=30$, $E_{22}=30$. Deviations: $30-20=10,10-20=-10,20-30=-10,40-30=10$. ${\chi }^2=\frac{100}{20}+\frac{100}{20}+\frac{100}{30}+\frac{100}{30}=5+5+3.333+3.333=16.667$.

	spam	ham	row total
”free” present	30 (E=20)	10 (E=20)	40
”free” absent	20 (E=30)	40 (E=30)	60
col total	50	50	100

• Final answer: ${\chi }^2\approx 16.67$ (strongly dependent on class; high score).

Python check

import numpy as np
from scipy.stats import chi2_contingency
table = np.array([[30, 10], [20, 40]])
print(chi2_contingency(table, correction=False)[0])   # 16.667  (no Yates correction)

Key lesson

Expected counts come from independence: row total × column total ÷ N.

Example 4 — Hard · Duplicate sensors and a noisy one

• Problem: Features $x_1$ and $x_2$ are exact duplicate sensors and both perfectly predict the class; $x_3$ is pure noise. A selector picks the top-2 by information gain. A PCA extractor keeps 1 component. Compare what each keeps. Four points $(x_1,x_2,x_3)$: $(0,0,5)$ class 0; $(0,0,-5)$ class 0; $(1,1,5)$ class 1; $(1,1,-5)$ class 1.
• Asked: Which representation is better and why.
• Concept: Redundancy, and variance ≠ relevance.
• Approach: Compute IG per feature; compute covariance.
• Solution: Class entropy $=1$ bit. Split on $x_1$: $x_1=0\to$ both class 0 (entropy 0); $x_1=1\to$ both class 1 (entropy 0); IG$(x_1)=1-0=1$; same for $x_2$. For $x_3$: $x_3=5\to$ one of each class (entropy 1); same for $-5$; IG$(x_3)=1-1=0$. The selector keeps $\{x_1,x_2\}$ — relevant but redundant. PCA: centred $x_1,x_2\in \{\pm 0.5\}$ with variance $0.25$ each and covariance $0.25$; $x_3$ has variance $25$ and zero covariance with the others. The largest variance direction is $x_3$ — pure noise. One-component PCA keeps noise and discards the class information.
• Final answer: Selection keeps two redundant but relevant features; one-component PCA keeps only the noise axis. Here selection wins.

Python check

import numpy as np
X = np.array([[0,0,5],[0,0,-5],[1,1,5],[1,1,-5]], float)
S = np.cov(X.T, bias=True)               # 1/n convention used in these notes
vals, vecs = np.linalg.eigh(S)
print(np.round(S, 3)); print(vals[-1], np.round(vecs[:, -1], 3))   # 25.0, ±[0,0,1]

Key lesson

Unsupervised extraction maximises variance, not relevance.

Example 5 — Very Hard · Vibration vs temperature on a machine

• Problem: A factory logs two readings per machine: $x_1$ = vibration (swings widely) and $x_2$ = temperature offset (tiny). Points (state): $(-10,-1)$ healthy, $(10,-1)$ healthy, $(-10,1)$ faulty, $(10,1)$ faulty. (a) Compute PCA’s first direction and the 1-D projected class separation. (b) Compute the information gain of splitting on the sign of $x_1$ and on the sign of $x_2$. (c) Which is better, and what is the lesson?
• Asked: PCA direction, IGs, conclusion.
• Concept: Variance ≠ discriminability.
• Approach: Covariance matrix → eigenvectors → project; then entropy.
• Solution: (a) Means: ${\bar{x}}_1=0,{\bar{x}}_2=0$. $\mathrm{Var}(x_1)=\frac{1}{4}(100\cdot 4)=100$; $\mathrm{Var}(x_2)=\frac{1}{4}(1\cdot 4)=1$; $\mathrm{Cov}=\frac{1}{4}[(-10)(-1)+(10)(-1)+(-10)(1)+(10)(1)]=\frac{1}{4}[10-10-10+10]=0$. $S=\mathrm{diag}(100,1)$, eigenvalues $100,1$; first PC $=[1,0{]}^T$. Projection onto it gives coordinates $-10,10,-10,10$ for the classes in the order healthy, healthy, faulty, faulty → both classes occupy both values → total overlap. (b) Splitting on $x_1$: groups $\{(-10,-1),(-10,1)\}$ and $\{(10,-1),(10,1)\}$, each with one of each class → entropy 1 each → IG$=1-1=0$. Splitting on $x_2$: groups all healthy and all faulty → entropy 0 → IG$=1$. (c) PCA’s direction has IG 0; the low-variance axis has IG 1.

 x2
  1 |  F(-10,1)            F(10,1)
    |
  0 +-----------------------------→ x1   ← PCA axis (var 100): classes overlap
    |
 -1 |  H(-10,-1)           H(10,-1)

• Final answer: PCA’s direction is $[1,0{]}^T$ (no class information); $x_2$ separates perfectly.

Python check

import numpy as np
from sklearn.feature_selection import mutual_info_classif
X = np.array([[-10,-1],[10,-1],[-10,1],[10,1]], float); y = np.array([0,0,1,1])
S = np.cov(X.T, bias=True); vals, vecs = np.linalg.eigh(S)
print(vals, np.round(vecs[:, -1], 3))                       # [1. 100.]  ±[1,0]
print(mutual_info_classif(np.sign(X), y, discrete_features=True) / np.log(2))   # [0. 1.]

Key lesson

When classification is the goal, use supervised selection — or a supervised projection such as Fisher’s discriminant (C48).

Common mistakes

• Believing the highest-variance feature is the most useful.
• Mixing up “selection” and “extraction”.

Exam traps

• ${\chi }^2$ and IG both need labels → supervised.
• Feature extraction may be applied before a supervised learner, but is itself unsupervised.

Connections: IG (C4), PCA (C43–C48), clustering (C35–C42).

Mastery checklist

• Distinguish selection vs extraction and supervised vs unsupervised.
• Compute ${\chi }^2$ from a $2\times 2$ table.
• Explain why PCA can discard class-relevant directions.