Deep Learning for Computer Architects

This is a primer written for computer architects in the new and rapidly evolving field of deep learning. It reviews how machine learning has evolved since its inception in the 1960s and tracks the key developments leading up to the emergence of the powerful deep learning techniques that emerged in the last decade.

Machine learning, and specifically deep learning, has been hugely disruptive in many fields of computer science. The success of deep learning techniques in solving notoriously difficult classification and regression problems has resulted in their rapid adoption in solving real-world problems. The emergence of deep learning is widely attributed to a virtuous cycle whereby fundamental advancements in training deeper models were enabled by the availability of massive datasets and high-performance computer hardware.

It also reviews representative workloads, including the most commonly used datasets and seminal networks across a variety of domains. In addition to discussing the workloads themselves, it also details the most popular deep learning tools and show how aspiring practitioners can use the tools with the workloads to characterize and optimize DNNs.

The remainder of the book is dedicated to the design and optimization of hardware and architectures for machine learning. As high-performance hardware was so instrumental in the success of machine learning becoming a practical solution, this chapter recounts a variety of optimizations proposed recently to further improve future designs. Finally, it presents a review of recent research published in the area as well as a taxonomy to help readers understand how various contributions fall in context.

Brandon Reagen is a Ph.D. candidate at Harvard University. He received his B.S. degree in Computer Systems Engineering and Applied Mathematics from University of Massachusetts, Amherst in 2012 and his M.S. in Computer Science from Harvard in 2014. His research spans the fields of Computer Architecture, VLSI, and Machine Learning with specific interest in designing extremely efficient hardware to enable ubiquitous deployment of Machine Learning models across all compute platforms.

Robert Adolf is a Ph.D. candidate in computer architecture at Harvard University. After earning a B.S. in Computer Science from Northwestern University in 2005, he spent four years doing benchmarking and performance analysis of supercomputers at the Department of Defense. In 2009, he joined Pacific Northwest National Laboratory as a research scientist, where he lead a team building large-scale graph analytics on massively multithreaded architectures. His research interests revolve around modeling, analysis, and optimization techniques for high-performance software, with a current focus on deep learning algorithms. His philosophy is that the combination of statistical methods, code analysis, and domain knowledge leads to better tools for understanding and building fast systems.

Paul Whatmough leads research on computer architecture for Machine Learning at ARM Research, Boston, MA. He is also an Associate in the School of Engineering and Applied Science at Harvard University. Dr. Whatmough received the B.Eng. degree (with first class Honors) from the University of Lancaster, U.K., M.Sc. degree (with distinction) from the University of Bristol, U.K., and Doctorate degree from University College London, U.K. His research interests span algorithms, computer architecture, and circuits. He has previously led various projects on hardware accelerators, Machine Learning, SoC architecture, Digital Signal Processing (DSP), variation tolerance, and supply voltage noise.

Gu-Yeon Wei is Gordon McKay Professor of Electrical Engineering and Computer Science in the School of Engineering and Applied Sciences (SEAS) at Harvard University. He received his B.S., M.S., and Ph.D. degrees in Electrical Engineering from Stanford University in 1994, 1997, and 2001, respectively. His research interests span multiple layers of a computing system: mixed-signal integrated circuits, computer architecture, and design tools for efficient hardware. His research efforts focus on identifying synergistic opportunities across these layers to develop energy-efficient solutions for a broad range of systems from flapping-wing microrobots to machine learning hardware for IoT/edge devices to specialized accelerators for large-scale servers.

David Brooks is the Haley Family Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. Prior to joining Harvard, he was a research staff member at IBM T. J. Watson Research Center. Prof. Brooks received his B.S. in Electrical Engineering at the University of Southern California and M.A. and Ph.D. degrees in Electrical Engineering at Princeton University. His research interests include resilient and power-efficient computer hardware and software design for high-performance and embedded systems. Prof. Brooks is a Fellow of the IEEE and has received several honors and awards including the ACM Maurice Wilkes Award, ISCA Influential Paper Award, NSF CAREER award, IBM Faculty Partnership Award, and DARPA Young Faculty Award.

Margaret Martonosi is the Hugh Trumbull Adams '35 Professor of Computer Science at Princeton University, where she has been on the faculty since 1994. She is also currently serving a four-year term as Director of the Keller Center for Innovation in Engineering Education. Martonosi holds affiliated faculty appointments in Princeton EE, the Center for Information Technology Policy (CITP), the Andlinger Center for Energy and the Environment, and the Princeton Environmental Institute. She also holds an affiliated faculty appointment in Princeton EE. From 2005-2007, she served as Associate Dean for Academic Affairs for the Princeton University School of Engineering and Applied Science. In 2011, she served as Acting Director of Princeton's Center for Information Technology Policy (CITP). From August 2015 through March, 2017, she served as a Jefferson Science Fellow within the U.S. Department of State.

Martonosi's research interests are in computer architecture and mobile computing, with particular focus on power-efficient systems. Her work has included the development of the Wattch power modeling tool and the Princeton ZebraNet mobile sensor network project for the design and real-world deployment of zebra tracking collars in Kenya. Her current research focuses on hardware-software interface approaches to manage heterogeneous parallelism and power-performance tradeoffs in systems ranging from smartphones to chip multiprocessors to large-scale data centers.

Martonosi is a Fellow of both IEEE and ACM. Notable awards include the 2010 Princeton University Graduate Mentoring Award, the 2013 NCWIT Undergraduate Research Mentoring Award, the 2013 Anita Borg Institute Technical Leadership Award, the 2015 Marie Pistilli Women in EDA Achievement Award, the 2015 ISCA Long-Term Influential Paper Award, and the 2017 ACM SIGMOBILE Test-of-Time Award. In addition to many archival publications, Martonosi is an inventor on seven granted US patents, and has co-authored two technical reference books on power-aware computer architecture. She has served on the Board of Directors of the Computing Research Association (CRA), and will co-chair CRA-W from 2017-2020. Martonosi completed her Ph.D. at Stanford University, and also holds a Master's degree from Stanford and a bachelor's degree from Cornell University, all in Electrical Engineering.