Model Selection & Evaluation
Cross-validation, metrics, and model comparison.
Calibration
When a model says “80% confidence” it should be right 80% of the time – reliability diagrams, Platt scaling, and isotonic regression.
Classification Metrics
Accuracy, precision, recall, F1, AUC-ROC, and AUC-PR – choosing the right metric depends on what errors cost.
Cross-Validation
K-fold, stratified, and leave-one-out validation – maximizing use of limited data for both training and evaluation.
Hyperparameter Tuning
Grid search, random search, and Bayesian optimization – finding optimal settings without overfitting to the validation set.
Learning Curves
Plotting performance vs. training set size or training iterations – diagnosing whether you need more data, more capacity, or more regularization.
Model Comparison
Paired t-tests, McNemar’s test, and Wilcoxon signed-rank – determining if performance differences are real or noise.
Regression Metrics
MSE, RMSE, MAE, MAPE, and R-squared – each captures different aspects of prediction quality.