The tradeoffs of Large-Scale Machine Learning
During the last decade, data sizes have outgrown processor speed. Computing time is then the bottleneck. The first part of the presentation theoretically uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems. The large-scale case involves the computational complexity of the underlying optimization algorithms in non-trivial ways. Unlikely optimization algorithm such as stochastic gradient descent show amazing performance for large-scale machine learning problems. The second part makes a detailed overview of stochastic gradient learning algorithms, with both simple and complex examples.