Module 2 — Evaluating Classifiers and Tuning Parameters

Sources: Lec 5 — Evaluation Criteria for Data Classification; Lec 6 — Parameter Tuning (Cross-Validation). · Prev: Module 1 · Next: Module 3 — Bayes · Index: 00_INDEX_and_Coverage

Python helpers — run once, checks below reuse them

import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score
 
def expand(M):
    """Confusion matrix (rows actual, cols predicted) -> (y_true, y_pred) label lists."""
    M = np.array(M); yt, yp = [], []
    for a in range(len(M)):
        for b in range(len(M)):
            yt += [a] * M[a, b]; yp += [b] * M[a, b]
    return yt, yp

C11. The Confusion Matrix, Accuracy, Precision, Recall, F-measure

Intuition

A classifier can be wrong in two different ways: crying wolf (false alarm) and missing the wolf (miss). The confusion matrix counts all four outcomes.

Accuracy lumps everything together.

Precision: “when I say positive, am I right?”

Recall: “of all real positives, how many did I find?”

F-measure combines the two.

Formal definition

	Predicted +	Predicted −
Actual +	TP	FN
Actual −	FP	TN

$Accuracy = \frac{T P + T N}{T P + F P + F N + T N}, Precision = \frac{T P}{T P + F P}, Recall = \frac{T P}{T P + F N},$ $F = \frac{2 \cdot Precision \cdot Recall}{Precision + Recall} .$

Memory aid

Precision looks down the predicted-positive column (TP + FP). Recall looks along the actual-positive row (TP + FN). TN appears in neither.

Mathematical properties

$F$ is the harmonic mean of precision $P$ and recall $R$ : $\frac{1}{F} = \frac{1}{2} (\frac{1}{P} + \frac{1}{R})$ .
$F \in [0, 1]$ ; $F = 1$ iff $P = R = 1$ ; $F = 0$ if $P = 0$ or $R = 0$ .
Equivalent count form: $F = \frac{2 T P}{2 T P + F P + F N}$ .
$min (P, R) \leq F \leq P R \leq \frac{P + R}{2}$ .

Derivation of the count form

$P = \frac{T P}{T P + F P}$ , $R = \frac{T P}{T P + F N}$ . Then $F = \frac{2 P R}{P + R} = \frac{2 \frac{T P ^{2}}{( T P + F P ) ( T P + F N )}}{\frac{T P ( T P + F N ) + T P ( T P + F P )}{( T P + F P ) ( T P + F N )}} = \frac{2 T P ^{2}}{T P ( 2 T P + F N + F P )} = \frac{2 T P}{2 T P + F P + F N} . ∎$

Key insights

TN does not appear in precision, recall, or $F$ — they ignore the (often huge) negative class.
Closer $P$ and $R$ ⇒ higher $F$ for a given sum.
Convention: if $T P + F P = 0$ precision is taken as $0$ (undefined $0/0$ ).

Worked examples

Example 1 — Very Easy · A spam filter’s accuracy

Problem: A spam filter handles 100 emails: $T P = 40$ spam caught, $F P = 10$ good mails wrongly flagged, $F N = 20$ spam missed, $T N = 30$ good mails delivered. Compute accuracy.
Asked: Accuracy.
Concept: Accuracy formula.
Approach: Correct / total.
Solution: Correct $= 40 + 30 = 70$ ; total $= 40 + 10 + 20 + 30 = 100$ .
Final answer: $0.70$ .

Key lesson

Accuracy uses the diagonal of the confusion matrix.

Example 2 — Easy · Precision and recall of the same filter

Problem: Same matrix. Compute precision and recall.
Asked: $P, R$ .
Concept: Definitions.
Approach: Row / column totals of the positive class.
Solution: $P = \frac{40}{40 + 10} = 0.80$ (of 50 flagged, 40 really spam). $R = \frac{40}{40 + 20} = 0.667$ (of 60 spam, 40 caught).
Final answer: $P = 0.80$ , $R = 0.667$ .

Key lesson

Precision divides by the predicted-positive column; recall by the actual-positive row.

Example 3 — Moderate · F two ways

Problem: Same matrix. Compute $F$ two ways and verify $min (P, R) \leq F \leq \frac{P + R}{2}$ .
Asked: $F$ .
Concept: Harmonic mean.
Approach: Formula and count form.
Solution: $F = \frac{2 ( 0.8 ) ( 0.6667 )}{0.8 + 0.6667} = \frac{1.0667}{1.4667} = 0.7273$ . Count form: $\frac{80}{80 + 10 + 20} = \frac{80}{110} = 0.7273$ ✓. Bounds: $0.667 \leq 0.727 \leq 0.733$ ✓.
Final answer: $F = 0.727$ .

Python check

yt, yp = expand([[40, 20], [10, 30]])          # rows = actual (spam, ham), cols = predicted
yt = [1 - t for t in yt]; yp = [1 - p for p in yp]       # make "spam" the positive class (1)
print(precision_score(yt, yp), recall_score(yt, yp), f1_score(yt, yp))   # 0.8  0.6667  0.7273

Key lesson

The harmonic mean sits below the arithmetic mean, pulled toward the smaller value.

Example 4 — Hard · The accuracy paradox (fraud detection)

Problem: Out of 1000 transactions, 20 are fraud. Classifier A predicts “not fraud” for everything. Classifier B has $T P = 15, F N = 5, F P = 30, T N = 950$ . Compare accuracy, $P$ , $R$ , $F$ .
Asked: Which is better.
Concept: Accuracy paradox.
Approach: Fill matrices.
Solution: A: $T P = 0, F N = 20, F P = 0, T N = 980$ . Accuracy $= 0.98$ . $R = 0$ ; $P = \frac{0}{0} \to 0$ by convention; $F = 0$ . B: accuracy $= \frac{965}{1000} = 0.965$ ; $P = \frac{15}{45} = 0.333$ ; $R = \frac{15}{20} = 0.75$ ; $F = \frac{30}{30 + 30 + 5} = 0.4615$ .
Final answer: A has higher accuracy (0.98 vs 0.965) but is useless (F = 0); B is far better (F = 0.46).

	Accuracy	$P$	$R$	$F$
A (always “legit”)	0.980	0	0	0
B	0.965	0.333	0.750	0.462

Key lesson

On imbalanced data, accuracy hides total failure on the rare class.

Example 5 — Very Hard · Where to cut a ranked list

Problem: A search engine ranks ten pages by score (highest first). Their true relevance is P,P,N,P,N,N,P,N,N,N (4 relevant). Predicting “relevant” for the top $k$ pages, compute $F$ for each $k$ and choose the best $k$ .
Asked: $ar g max_{k} F (k)$ .
Concept: $F = \frac{2 T P}{2 T P + F P + F N}$ ; threshold selection.
Approach: Track $T P, F P$ cumulatively; $F N = 4 - T P$ .
Solution:

$k$	TP	FP	FN	$F$
1	1	0	3	0.400
2	2	0	2	0.667
3	2	1	2	0.571
4	3	1	1	0.750
5	3	2	1	0.667
6	3	3	1	0.600
7	4	3	0	0.727
8	4	4	0	0.667
9	4	5	0	0.615
10	4	6	0	0.571

Final answer: Best $k = 4$ , $F = 0.75$ (not $k = 7$ where recall is 1).

Python check

labels = np.array([1, 1, 0, 1, 0, 0, 1, 0, 0, 0])
for k in range(1, 11):
    print(k, round(f1_score(labels, [1] * k + [0] * (10 - k)), 3))   # peaks at k = 4 (0.75)

Key lesson

$F$ is not monotone in the threshold; the best $F$ balances precision and recall.

Common mistakes

Mixing the roles of FP and FN in precision / recall.

Quoting accuracy alone on imbalanced data.

Exam traps

“Precision = TP/(TP+FN)” is wrong — that is recall.

$F$ is $0$ when $T P = 0$ .

Connections: Sensitivity / specificity and averaging (C12), clustering F-measure (C41), cross-validation scoring (C13).

Mastery checklist

Draw the confusion matrix and fill it.
Compute $A, P, R, F$ and the count form.
Explain the accuracy paradox.

C12. Macro/Micro Averaging, Sensitivity & Specificity, Class Imbalance

Intuition

With $m$ classes there is one $(T P_{i}, F P_{i}, F N_{i})$ per class (treat class $i$ as “positive”, the rest “negative”). Two ways to summarise:

Macro — average the per-class scores: every class counts equally, so rare classes matter.

Micro — pool the counts, then compute: every data point counts equally, so big classes dominate.

Formal definitions

$P_{ma cr o} = \frac{1}{m} \sum_{i = 1}^{m} \frac{T P _{i}}{T P _{i} + F P _{i}}, R_{ma cr o} = \frac{1}{m} \sum_{i = 1}^{m} \frac{T P _{i}}{T P _{i} + F N _{i}},$ $P_{mi cr o} = \frac{\sum _{i} T P _{i}}{\sum _{i} T P _{i} + \sum _{i} F P _{i}}, R_{mi cr o} = \frac{\sum _{i} T P _{i}}{\sum _{i} T P _{i} + \sum _{i} F N _{i}} .$ $Sensitivity = \frac{T P}{T P + F N} (recall, TPR), Specificity = \frac{T N}{T N + F P} (TNR) .$

Mathematical result (derived)

In single-label multiclass classification, every misclassified item is a false positive for exactly one class and a false negative for exactly one class, so $\sum_{i} F P_{i} = \sum_{i} F N_{i}$ . Hence $P_{mi cr o} = R_{mi cr o} = \frac{\sum _{i} T P _{i}}{N} = accuracy$ .

Derivation

Let $M_{ab}$ = count with actual $a$ , predicted $b$ . $F P_{b} = \sum_{a \neq = b} M_{ab}$ , $F N_{a} = \sum_{b \neq = a} M_{ab}$ . $\sum_{b} F P_{b} = \sum_{b} \sum_{a \neq = b} M_{ab} = \sum_{a \neq = b} M_{ab}$ , and $\sum_{a} F N_{a}$ is the same off-diagonal sum. ∎

Key insights

Macro: “per-class average”, insensitive to class frequency. Micro: “per-data-point average”.
Sensitivity and specificity are the standard medical pair: how many sick patients are found, how many healthy ones are cleared.

Worked examples

Example 1 — Very Easy · A disease screening test

Problem: A test on 1000 people: $T P = 15, F N = 5, F P = 30, T N = 950$ . Sensitivity and specificity?
Asked: Both.
Concept: Definitions.
Approach: Direct.
Solution: Sensitivity $= \frac{15}{20} = 0.75$ (3 of 4 sick people found). Specificity $= \frac{950}{950 + 30} = 0.9694$ .
Final answer: $0.75$ and $0.969$ .

Key lesson

High specificity can coexist with mediocre sensitivity.

Example 2 — Easy · Per-class counts for support tickets

Problem: A model sorts support tickets into A = billing, B = technical, C = legal. Confusion matrix (rows actual, columns predicted): $50545222534$ . Compute $T P_{i}, F P_{i}, F N_{i}$ and per-class precision / recall.
Asked: Per-class counts and scores.
Concept: One-vs-rest counts.
Approach: Diagonal; column sums minus diagonal; row sums minus diagonal.
Solution: Row totals: $60, 30, 10$ . Column totals: $59, 29, 12$ . $T P = (50, 22, 4)$ . $F P = col - T P = (9, 7, 8)$ . $F N = row - T P = (10, 8, 6)$ .

class	TP	FP	FN	precision	recall
A billing	50	9	10	0.8475	0.8333
B technical	22	7	8	0.7586	0.7333
C legal	4	8	6	0.3333	0.4000

Final answer: As in the table.

Key lesson

Class C (rare, 10 tickets) is much worse than A — the overall number hides it.

Example 3 — Moderate · Macro vs micro

Problem: For the matrix of Example 2 compute macro and micro precision and recall.
Asked: Four numbers.
Concept: Averaging.
Approach: Macro: mean of per-class; micro: pooled.
Solution: Macro $P = \frac{0.8475 + 0.7586 + 0.3333}{3} = 0.6465$ . Macro $R = \frac{0.8333 + 0.7333 + 0.4}{3} = 0.6556$ . Micro: $\sum T P = 76$ , $\sum F P = 24$ , $\sum F N = 24$ → $P_{mi cr o} = R_{mi cr o} = \frac{76}{100} = 0.76$ .
Final answer: Macro $(0.647, 0.656)$ ; micro $(0.76, 0.76)$ .

Python check

yt, yp = expand([[50, 5, 5], [5, 22, 3], [4, 2, 4]])
for avg in ("macro", "micro"):
    print(avg, precision_score(yt, yp, average=avg), recall_score(yt, yp, average=avg))
# macro 0.6465 0.6556   micro 0.76 0.76

Key lesson

Macro is lower because the weak rare class gets a full third of the weight.

Example 4 — Hard · A defect detector that never says “defect”

Problem: A factory line has 980 “good” and 20 “defective” items; the detector labels everything good. Compute accuracy, micro $R$ , macro $R$ , and the sensitivity and specificity of the defective class.
Asked: Five numbers.
Concept: Which metric exposes imbalance.
Approach: Per-class recalls.
Solution: Good: $T P = 980, F N = 0$ → $R = 1$ . Defective: $T P = 0, F N = 20$ → $R = 0$ . Accuracy $= 0.98$ . Micro $R = \frac{980}{1000} = 0.98$ . Macro $R = \frac{1 + 0}{2} = 0.5$ . Sensitivity (defective) $= 0$ ; specificity (defective) $= \frac{980}{980} = 1$ .
Final answer: Accuracy $0.98$ , micro $0.98$ , macro $0.5$ , sensitivity $0$ , specificity $1$ .

Key lesson

Macro averaging (and sensitivity) reveal failure on the rare class; accuracy and micro do not.

Example 5 — Very Hard · Improving only the rare class

Problem: Starting from the matrix of Example 2, an improved model re-labels 3 class-C tickets that were predicted A as predicted C: the C row becomes $[1, 2, 7]$ (rows A, B unchanged). Recompute micro and macro recall and precision, and compare the improvement.
Asked: New metrics and change.
Concept: Sensitivity of macro vs micro to a rare class.
Approach: Recount.
Solution: New matrix $50515222537$ . Column totals: $56, 29, 15$ . $T P = (50, 22, 7)$ , $F P = (6, 7, 8)$ , $F N = (10, 8, 3)$ . Precision: $0.8929, 0.7586, 0.4667$ → macro $P = 0.7060$ . Recall: $0.8333, 0.7333, 0.7$ → macro $R = 0.7556$ . Micro: $\sum T P = 79$ → $P_{mi cr o} = R_{mi cr o} = 0.79$ .

	before	after	change
micro $P = R$	0.76	0.79	+0.03
macro $P$	0.646	0.706	+0.060
macro $R$	0.656	0.756	+0.100

Final answer: Micro $0.79$ (+3 pts); macro $P = 0.706$ (+6 pts), macro $R = 0.756$ (+10 pts).

Python check

yt, yp = expand([[50, 5, 5], [5, 22, 3], [1, 2, 7]])
for avg in ("macro", "micro"):
    print(avg, round(precision_score(yt, yp, average=avg), 4), round(recall_score(yt, yp, average=avg), 4))
# macro 0.7060 0.7556   micro 0.79 0.79

Key lesson

Improving one small class barely moves micro scores but visibly moves macro scores.

Common mistakes

Averaging per-class accuracies and calling it macro precision.

Expecting micro $P \neq =$ micro $R$ in single-label problems.

Exam traps

Sensitivity ≡ recall ≡ TPR; specificity is not precision.

Micro = per-data-point average; macro = per-class average.

Connections: C11, class imbalance, cross-validation (C13), clustering evaluation (C41).

Mastery checklist

Extract $T P_{i}, F P_{i}, F N_{i}$ from a multiclass matrix.
Compute macro vs micro.
Prove micro $P$ = micro $R$ = accuracy.

C13. Cross-Validation: Leave-One-Out and $k$ -Fold

Intuition

Evaluating on training data flatters the model; setting aside one test set wastes data and gives a lucky-or-unlucky estimate. Cross-validation (CV) rotates the roles: every point is a test point exactly once, while the model is never trained on the point it is tested on. CV is also how hyper-parameters (like $k$ in kNN, $C$ in SVM, tree depth) are chosen.

 3-fold CV on 9 points            round 1: [ TEST ][ train ][ train ]
 ┌─────┬─────┬─────┐              round 2: [ train ][ TEST ][ train ]
 │ Z1  │ Z2  │ Z3  │              round 3: [ train ][ train ][ TEST ]
 └─────┴─────┴─────┘              → pool the 9 predictions, then score once

Formal definition

Data $Z = {X_{1}, \dots, X_{n}}$ , labels $y_{i} \in {1, \dots, c}$ .

Leave-One-Out (LOO): for $i = 1.. n$ : train on $Z ∖ {X_{i}}$ , predict $y_{i}^{'}$ for $X_{i}$ . Score using ${y_{i}}$ vs ${y_{i}^{'}}$ (e.g. $F$ -measure).
$k$ -fold CV: partition $Z$ into $Z_{1}, \dots, Z_{k}$ with (i) $Z_{i} \neq = \emptyset$ , (ii) $Z_{i} \cap Z_{j} = \emptyset$ , (iii) $⋃ Z_{i} = Z$ , (iv) sizes (nearly) equal. For $i = 1.. k$ : train on $P_{i} = Z ∖ Z_{i}$ , predict $Z_{i}$ . Score on the pooled predictions.

Mathematical formulation

Number of trainings: LOO $= n$ ; $k$ -fold $= k$ . $k = n$ ⇒ $k$ -fold = LOO.
Each training set has $n - n / k = n (1 - 1/ k)$ samples.
LOO is deterministic (exhaustive); $k$ -fold with random folds varies between runs. Repeating (e.g. 50 times) and averaging makes the estimate independent of the particular partition.

Key insights

Bias–variance of the estimate: LOO training sets are nearly the full data (low bias) but cost $n$ trainings; $k$ -fold is cheaper.
The folds must be disjoint and cover everything.

Worked examples

Example 1 — Very Easy · Fold sizes

Problem: $n = 10$ points, 3-fold CV. Possible fold sizes? How many models are trained?
Asked: Sizes and count.
Concept: Near-equal folds.
Approach: $10 = 4 + 3 + 3$ .
Solution: Sizes $4, 3, 3$ (any order). One model per fold → 3 models.
Final answer: $4, 3, 3$ ; 3 models.

Key lesson

Folds differ in size by at most one element.

Example 2 — Easy · Cost of LOO

Problem: $n = 100$ . For 5-fold CV and for LOO give training-set size and number of trainings.
Asked: Four numbers.
Concept: Counts.
Approach: Formulas.
Solution: 5-fold: test $= 20$ , train $= 80$ , 5 trainings. LOO: train $= 99$ , 100 trainings.
Final answer: $(80, 5)$ and $(99, 100)$ .

Key lesson

LOO is $n / k$ times more expensive than $k$ -fold.

Example 3 — Moderate · Cats and dogs by weight, LOO with 1-NN

Problem: Animals weighed in kg: $x = 1, 2, 3, 10, 11$ with labels cat, cat, cat, dog, dog. Run LOO with 1-NN and report accuracy. (If two neighbours tie, both are cats.)
Asked: LOO accuracy of 1-NN.
Concept: LOO procedure.
Approach: Remove each point, classify by nearest remaining.
Solution: $x = 1$ : nearest is 2 (cat) ✓. $x = 2$ : neighbours 1 and 3 equidistant, both cats ✓. $x = 3$ : nearest 2 (cat) ✓. $x = 10$ : nearest 11 (dog) ✓. $x = 11$ : nearest 10 (dog) ✓. 5 / 5.
Final answer: $100%$ .

Key lesson

Well-separated clusters give perfect LOO accuracy for 1-NN.

Example 4 — Hard · Choosing $k$ by LOO

Problem: Same data. Use LOO to choose between $k = 1$ and $k = 3$ nearest neighbours (majority vote).
Asked: Best $k$ .
Concept: Parameter tuning by CV.
Approach: Compute LOO accuracy for $k = 3$ .
Solution: $x = 1$ : nearest others 2, 3, 10 → cat, cat, dog → cat ✓. $x = 2$ : 1, 3, 10 → cat ✓. $x = 3$ : 2, 1, 10 → cat ✓. $x = 10$ : 11, 3, 2 → dog, cat, cat → cat ✗. $x = 11$ : 10, 3, 2 → dog, cat, cat → cat ✗. Accuracy $3/5 = 0.6$ . $k = 1$ gave $1.0$ .
Final answer: $k = 1$ (LOO accuracy 1.0 vs 0.6).

Python check

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import LeaveOneOut, cross_val_score
X = np.array([[1], [2], [3], [10], [11]]); y = np.array([0, 0, 0, 1, 1])
for k in (1, 3):
    print(k, cross_val_score(KNeighborsClassifier(k), X, y, cv=LeaveOneOut()).mean())   # 1.0  0.6

Key lesson

A $k$ larger than the size of the smaller class (2 dogs) forces errors. Tune $k$ by CV.

Example 5 — Very Hard · Average of folds vs pooled predictions

Problem: A 2-fold CV gives fold 1: $T P = 2, F P = 0, F N = 2$ ; fold 2: $T P = 8, F P = 4, F N = 0$ . Compute $F$ (a) per fold then averaged; (b) on the pooled predictions as the lecture prescribes (“using the actual labels of $Z$ ”). Explain the difference.
Asked: Two scores.
Concept: Pooling vs averaging.
Approach: $F = \frac{2 T P}{2 T P + F P + F N}$ .
Solution: (a) $F_{1} = \frac{4}{4 + 0 + 2} = 0.6667$ ; $F_{2} = \frac{16}{16 + 4 + 0} = 0.8$ ; mean $= 0.7333$ . (b) Pooled: $T P = 10, F P = 4, F N = 2$ → $F = \frac{20}{20 + 4 + 2} = 0.7692$ .
Final answer: Average-of-folds $0.733$ ; pooled $0.769$ .

Python check

F = lambda tp, fp, fn: 2*tp / (2*tp + fp + fn)
print((F(2,0,2) + F(8,4,0)) / 2, F(10, 4, 2))     # 0.7333  0.7692

Key lesson

$F$ is nonlinear in the counts: fold-averaging weights every fold equally, pooling weights every data point equally. Be explicit about which you use.

Common mistakes

Tuning on the test fold (data leakage) and reporting that score.

Overlapping folds.

Random folds with tiny classes (use stratified folds).

Exam traps

LOO is exhaustive and deterministic; $k$ -fold with a random partition is not.

$k = n$ ⇒ LOO.

Connections: Pruning (C8), $k$ in kNN (C23), $C$ in SVM (C32), $θ, β$ parameters.

Mastery checklist

Write both algorithms.
State the folds’ four properties.
Use CV to pick a hyper-parameter.

Vault

Explorer

03_Evaluation_and_Model_Selection

Module 2 — Evaluating Classifiers and Tuning Parameters

C11. The Confusion Matrix, Accuracy, Precision, Recall, F-measure

Formal definition

Mathematical properties

Derivation of the count form

Key insights

Worked examples

Mastery checklist

C12. Macro/Micro Averaging, Sensitivity & Specificity, Class Imbalance

Formal definitions

Mathematical result (derived)

Derivation

Key insights

Worked examples

Mastery checklist

C13. Cross-Validation: Leave-One-Out and $k$ -Fold

Formal definition

Mathematical formulation

Key insights

Worked examples

Mastery checklist

Graph View

Table of Contents

Vault

Explorer

03_Evaluation_and_Model_Selection

Module 2 — Evaluating Classifiers and Tuning Parameters

C11. The Confusion Matrix, Accuracy, Precision, Recall, F-measure

Formal definition

Mathematical properties

Derivation of the count form

Key insights

Worked examples

Mastery checklist

C12. Macro/Micro Averaging, Sensitivity & Specificity, Class Imbalance

Formal definitions

Mathematical result (derived)

Derivation

Key insights

Worked examples

Mastery checklist

C13. Cross-Validation: Leave-One-Out and k-Fold

Formal definition

Mathematical formulation

Key insights

Worked examples

Mastery checklist

Graph View

Table of Contents

C13. Cross-Validation: Leave-One-Out and $k$ -Fold