Autumn 2020 Calendar and Reading List

Note: Topics and reading are subject to revision, based on evolving discusssions and interest

---

Week 1: Motivation and Introduction

Tuesday, September 29: Introduction to the class

Topics:

• Class goals and structure

• Readings, videos, and discussion sessions

• Assignments

• Project and paper

Things to do:

• Sign up for Google Collaboratory, which we will use to run examples.

• Watch this video and this second video on Google Colab.

Thursday, October 1: Introduction to deep learning

Topics:

• Deep neural networks (DNNs)

• Keras

Before class, watch:

• The four videos on deep learning concepts here

Other reading, for this week and later too:

• Deep Learning book by Goodfellow and friends, available online.

• Deep Learning wth Python book by Francois Chollet, available online (but limited access).

---

Week 2 - Deep Learning Implementation

Tuesday, October 6: How DNNs learn

Topics:

• Gradient descent: how neural networks learn

• Back propagation

• Computational considerations: What DNNs mean for computers

Thursday, October 8: Computational considerations contd

Topics:

• It’s all linear algebra

• CPUs, GPUs, TPUs

• AI accelerators: Cerebras, SambaNova, GraphCore, etc.

Read:

• Chapter 6 of Deep Learning

• The computational limits of deep learning

---

Week 3 - Parallelism

Tuesday, October 13

Read:

• Large Scale Distributed Deep Networks

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

• Beyond Data and Model Parallelism for Deep Neural Networks

Thursday, October 15: Zhao Zhang, TACC

Read:

• On large-batch training for deep learning: Generalization gap and sharp minima

• ImageNet training in minutes

• Convolutional Neural Network Training with Distributed K-FAC

Also interesting:

• Accurate, large minibatch SGD: Training ImageNet in 1 hour

• Large batch training of convolutional networks

• Large batch optimization for deep learning: Training BERT in 76 minutes

Week 4 - Varieties of Neural Network

Tuesday, October 20

Thursday, October 22

Read:

• The neural network zoo

• ImageNet Classification with Deep Convolutional Neural Networks

---

Week 5 - Measuring and analyzing performance

Tuesday, October 27: TBD

Thursday, October 29: TBD

Read:

• DAWNBench: An End-to-End Deep Learning Benchmark and Competition

• DAWNBench web site

• Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

• MLPerf

---

Week 5 - Precision and accuracy

Tuesday, November 3: TBD

Thursday, November 5: TBD

Read:

• Performance-Efficiency Trade-off of Low-Precision Numerical Formats in Deep Neural Networks

---

Week 7 - Nontraditional architectures

Tuesday, November 10: Neuromorphic computing

Thursday, November 12: Optical neural networks

Read:

• Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks

• Neuromorphic computing using non-volatile memory

---

Week 8 - TBD

Tuesday, November 17: TBD

Thursday, November 19: TBD

---

Week 9 - No Class

Thanksgiving week

---

Week 10 - Project presentations

Tuesday, December 1

• Project presentations

Thursday, December 3

• Project presentations