Non-classical Bias-Variance Trade-off: Surprising Insights in Deep Learning

1. CMS 165 Lecture 18 Summing up..

2. Most surprising/interesting things learnt in this class • Bias-variance trade-off in non-classical setting • Over-parametrization regime where traditional bias-variance don’t behave the same • Robustness – min-max games (data augmentation + adversarial training) • Decoupling the uncertainties in errors (active learning, fairness) • Symmetries in non-convex optimization can be exploited • Non-isolated optimal points (lives in the manifold) • Generalization can be improved with robust training • Data collection and importance of having good train/test set • Role of local regularization and global regularization • Sample complexity results and how it is not straightforward in deep learning • Stability, effect of single data point in the dataset

3. Missing pieces (not covered in class) • Interpretability • Discussed applications are limited to standard ones, not many deep learning or recent applications of established works • Reinforcement learning/Lifelong learning • Likelihood in test after training • GANs • Negative results and tricks in achieving good results • Explicit discussion of types of learning algorithms/ categorization • Decision trees/classical ML techniques • Recent properties of neural network architectures and computation scales • Sequence models/autoencoders and embeddings • Causal inference(besides Bayesian networks?)

4. Other comments.. • Peer grading (not much learning from peer grading)(maybe on the open ended ones?) • Maybe different application questions • Paper to code kind of questions • More applications/discussions of tensors • Send the good homeworks to students instead of sending randomly • More recitations on both theory and applications