This graduate-level textbook introduces fundamental concepts and methods in machine learning. It describes several important modern algorithms, provides the theoretical underpinnings of these algorithms, and illustrates key aspects for their application. The authors aim to present novel theoretical tools and concepts while giving concise proofs even for relatively advanced topics. "Foundations of Machine Learning" fills the need for a general textbook that also offers theoretical details and an emphasis on proofs. Certain topics that are often treated with insufficient attention are discussed in more detail here; for example, entire chapters are devoted to regression, multi-class classification, and ranking. The first three chapters lay the theoretical foundation for what follows, but each remaining chapter is mostly self-contained. The appendix offers a concise probability review, a short introduction to convex optimization, tools for concentration bounds, and several basic properties of matrices and norms used in the book. The book is intended for graduate students and researchers in machine learning, statistics, and related areas; it can be used either as a textbook or as a reference text for a research seminar.
No image available
· 2019
Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. Ameet Talwalkar (Carnegie Mellon University Determined AI) shares work that aims to help ground the empirical results in this field and proposes new NAS baselines that build off the following observations: NAS is a specialized hyperparameter optimization problem, and random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, Ameet evaluates both random search with early-stopping and a novel random search with a weight-sharing algorithm on two standard NAS benchmarks: PTB and CIFAR-10. Results show that random search with early-stopping is a competitive NAS baseline that performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Ameet concludes by exploring existing reproducibility issues for published NAS results, noting the lack of source material needed to exactly reproduce these results, and discussing the robustness of published results given the various sources of variability in NAS experimental setups. All information (code, random seeds, documentation) needed to exactly reproduce our results will be shared, along with random search with weight-sharing results for each benchmark on two independent experimental runs. This session was recorded at the 2019 O'Reilly Artificial Intelligence Conference in New York.