Quick summary
Summarize this blog with AI
Introduction
A confusion matrix cross-tabulates predicted classes against known classes on evaluation data. Label its axes and name the positive class before computing metrics. This lesson starts with counts you can check by hand, then evaluates an actual held-out iris classifier.
Tested with R 4.5.1, MASS 7.3-65 and caret 7.0-1. Run the code blocks in order. Install the named packages before starting; inputs are supplied below.
Fix class levels and matrix orientation
Here rows are predictions and columns are reference labels. Positive means “yes.” Fixed factor levels preserve a row or column even when that class is absent. Validate unknown labels before factor conversion, because an unrecognized level would otherwise become NA.
classes <- c("no", "yes")
truth <- factor(c("no", "no", "no", "no", "yes", "yes", "yes", "yes"), levels = classes)
predicted <- factor(c("no", "no", "no", "yes", "yes", "yes", "no", "no"), levels = classes)
cm <- table(Prediction = predicted, Reference = truth)
print(cm)
stopifnot(isTRUE(all.equal(as.vector(cm), c(3L, 1L, 2L, 2L))),
sum(cm) == length(truth))
The cells are TN = 3, FP = 1, FN = 2 and TP = 2. Reversing rows and columns preserves accuracy but changes where false positives and false negatives appear.
Derive metrics and handle zero denominators
Accuracy is (TP + TN)/n, precision is TP/(TP + FP), recall is TP/(TP + FN), and specificity is TN/(TN + FP). F 1 can be calculated directly as 2TP/(2TP + FP + FN). A denominator of zero means the metric is undefined; this helper returns NA rather than silently claiming perfect or zero performance.
divide <- function(a, b) if (b == 0) NA_real_ else a / b
binary_metrics <- function(m) {
if (!is.matrix(m) || !identical(dim(m), c(2L, 2L)) ||
!identical(rownames(m), classes) || !identical(colnames(m), classes) ||
any(!is.finite(m)) || any(m < 0)) stop("Need a nonnegative2x2 table labeled no/yes")
tn <- m["no", "no"]; fp <- m["yes", "no"]
fn <- m["no", "yes"]; tp <- m["yes", "yes"]
c(accuracy = divide(tp + tn, sum(m)), precision = divide(tp, tp + fp),
recall = divide(tp, tp + fn), specificity = divide(tn, tn + fp),
F1 = divide(2 * tp, 2 * tp + fp + fn))
}
metrics <- binary_metrics(cm)
print(round(metrics, 4))
stopifnot(isTRUE(all.equal(unname(metrics), c(5/8, 2/3, 1/2, 3/4, 4/7))))
all_no <- table(factor(rep("no", 8), levels = classes), truth)
stopifnot(is.na(binary_metrics(all_no)["precision"]))
empty <- cm * 0L
stopifnot(all(is.na(binary_metrics(empty))))
Results: accuracy 0.6250, precision 0.6667, recall 0.5000, specificity 0.7500 and F 1 0.5714. Accuracy alone can conceal poor minority-class recall. These are estimates on eight observations, not dependable deployment performance claims.
Extract binary caret metrics with vector indexing
caret::confusionMatrix() takes predictions in data and actual labels in reference. Its binary byClass result is a named vector; select names with one index. Setting positive explicitly avoids depending on factor-level order.
result <- caret::confusionMatrix(data = predicted, reference = truth,
positive = "yes", mode = "everything")
print(result$byClass[c("Precision", "Recall", "F1")])
stopifnot(isTRUE(all.equal(unname(result$byClass[c("Precision", "Recall", "F1")]),
unname(metrics[c("precision", "recall", "F1")]))))
Train LDA and predict held-out iris rows
Use a fixed split with ten held-out rows per species. This transparent split is an instructional fixture; real evaluations need a sampling strategy appropriate for time, people or groups. predict.lda() returns a list. Extract its class component for a confusion matrix; its posterior component contains class probabilities.
test_index <- seq(5, nrow(iris), by = 5)
training <- iris[-test_index, ]
testing <- iris[test_index, ]
stopifnot(length(intersect(rownames(training), rownames(testing))) == 0)
lda_fit <- MASS::lda(Species ~ ., data = training)
prediction <- predict(lda_fit, newdata = testing)
predicted_species <- factor(prediction$class, levels = levels(iris$Species))
multi <- caret::confusionMatrix(predicted_species, testing$Species, mode = "everything")
print(multi$table)
print(round(multi$byClass[, c("Precision", "Recall", "F1")], 4))
stopifnot(sum(multi$table) == 30, is.matrix(multi$byClass),
all(abs(rowSums(prediction$posterior) - 1) < 1e-12))
This fixed split classifies all 30 held-out observations correctly; this small teaching fixture does not establish general performance. For three classes, byClass is a matrix with one-versus-rest rows. Read each species’ metrics separately; a macro average weights classes equally, whereas a support-weighted average weights their evaluation counts. Neither should be reported without naming the convention.
Keep alignment and evaluation decisions explicit
Check lengths, IDs, missing labels and allowed values before tabulation. table() ordinarily drops missing observations, which can hide evaluation failures. Reordering predictions without their corresponding reference rows invalidates the result. Choose thresholds and tune models using training/validation data; repeated decisions on the final test set make it part of training. ROC or precision–recall curves require scores across thresholds, not just a single hard-label matrix.
make_binary_table <- function(pred, actual) {
if (!is.character(pred) || !is.character(actual) || !is.null(dim(pred)) || !is.null(dim(actual)) ||
length(pred) != length(actual) || anyNA(pred) || anyNA(actual) ||
any(!pred %in% classes) || any(!actual %in% classes))
stop("Need aligned character labels drawn from no/yes, with no missing labels")
table(Prediction = factor(pred, levels = classes),
Reference = factor(actual, levels = classes))
}
fails <- function(expr) inherits(tryCatch(expr, error = identity), "error")
stopifnot(identical(make_binary_table(as.character(predicted), as.character(truth)), cm),
fails(make_binary_table("maybe", "yes")), fails(make_binary_table(NA_character_, "yes")),
fails(make_binary_table(c("yes", "no"), "yes")),
fails(make_binary_table(matrix("yes", 1), "yes")),
all(is.na(binary_metrics(make_binary_table(character(), character())))))
