Core Learning Theory

Bias-variance tradeoff, PAC learning, and generalization.

The fundamental tension between underfitting and overfitting – every model navigates this tradeoff whether you manage it or not.

As dimensions increase, data becomes sparse, distances become meaningless, and exponentially more data is needed.

Minimizing average loss on training data as a proxy for true risk – the theoretical framework underlying most ML algorithms.

The objective being optimized – MSE, cross-entropy, hinge loss, and how the choice shapes what the model learns.

Memorizing training data vs. failing to capture patterns – the two failure modes of every learning algorithm.

Constraining model complexity to improve generalization – L1, L2, dropout, early stopping, and the bias-variance connection.

Supervised, unsupervised, semi-supervised, and self-supervised – a taxonomy based on what labels are available.

Learning patterns from data rather than programming rules explicitly – the three paradigms and when each applies.