Check the preview of 2nd version of this platform
being developed by the open MLCommons taskforce on automation and reproducibility
as a free, open-source and technology-agnostic on-prem platform.
MLPerf Training Benchmark
lib:ea2e1cf41b191718 (v1.0.0)
Authors: Peter Mattson,Christine Cheng,Cody Coleman,Greg Diamos,Paulius Micikevicius,David Patterson,Hanlin Tang,Gu-Yeon Wei,Peter Bailis,Victor Bittorf,David Brooks,Dehao Chen,Debojyoti Dutta,Udit Gupta,Kim Hazelwood,Andrew Hock,Xinyuan Huang,Atsushi Ike,Bill Jia,Daniel Kang,David Kanter,Naveen Kumar,Jeffery Liao,Guokai Ma,Deepak Narayanan,Tayo Oguntebi,Gennady Pekhimenko,Lillian Pentecost,Vijay Janapa Reddi,Taylor Robie,Tom St. John,Tsuguchika Tabaru,Carole-Jean Wu,Lingjie Xu,Masafumi Yamazaki,Cliff Young,Matei Zaharia
ArXiv: 1910.01500
Document: PDF DOI
Abstract URL: https://arxiv.org/abs/1910.01500v3
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.
ArXiv: 1910.01500
Document: PDF DOI
Abstract URL: https://arxiv.org/abs/1910.01500v3
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.
Relevant initiatives
Related knowledge about this paper
Reproduced results (crowd-benchmarking and competitions)
Artifact and reproducibility checklists
- ACM/cTuning artifact appendix and reproducibility checklist
- NeurIPS reproducibility checklist
- SIGPLAN's checklist for empirical evaluation
- Collective Knowledge (organizing research projects based on FAIR principles)