Syllabus

Course description

Deep learning is emerging as a major technique for solving problems in a variety of fields, including computer vision, personalized medicine, autonomous vehicles, and natural language processing. Critical to success in these target domains is the development of learning systems: deep learning frameworks that support the tasks of learning complex models and inferencing with those models, and targeting many devices including CPUs, GPUs, mobile device, edge devices, computer clusters, and scalable parallel systems. The systematic study of how to build and optimize such systems is an active area of research.

This course is aimed as an introduction to this topic. We will cover various aspects of deep learning systems, including: basics of deep learning, programming models for expressing machine learning models, automatic differentiation methods used to compute gradients for training, memory optimization, scheduling, data and model parallel and distributed learning, hardware acceleration, domain specific languages, workflows for large-scale machine learning including hyper parameter optimization and uncertainty quantification, and training data and model serving. Many of these topics intersect with existing research directions in databases, systems and networking, architecture, and programming languages. The goal is to present a comprehensive picture of how current deep learning systems work, discuss and explore research opportunities for extending and building on existing frameworks, and deep dive into the accelerators being developed by numerous startups to address the performance needs of the machine learning community.

We will split out time between concepts and practice, with a typical week having one lecture on a specific aspect of deep learning systems and one lab/discussion session in which technologies such as Keras, Tensorflow, CNTK, Mxnet, and PyTorch are used to address that specific aspect. Some guest lectures may cover emerging computer architectures for next generation deep learning accelerators.

Specific topics to be covered:

  • Introduction to deep learning models

  • Functional content of deep learning frameworks

  • Software architecture and design of frameworks

  • Abstraction layers for deep learning

  • Performance and benchmarking deep learning systems

  • Hardware architectures for accelerating deep learning

  • Parallelism (model, data, ensemble)

  • Portable representations and translations of models

  • Optimization for training, inference

  • Workflows for machine learning and workflow tools

  • Hyper-parameter optimization and ensembles

  • Uncertainty quantification

At the end of the quarter, students will:

  1. Understand the purpose of deep learning systems.

  2. Be able to use common deep learning tools such as Keras, TensorFlow, and PyTorch

  3. Understand major challenges in efficient deep learning and how those challenges are addressed in different systems.

  4. Have basic knowledge of research challenges in deep learning system design and implementation.

Course organization

This course is divided into three components:

  • Lectures: The Tuesday and Thursday lectures will present technical material on deep learning systems.

  • Programming Assignments: Four short programming assignments will be given throughout the quarter.

  • Project and Paper: Students have to define and complete a project that covers some aspect of deep learning systems. Based on their projects, students have to write a final paper evaluating the features and performance of their project. See Project and Paper for more information.

Course Schedule

See Calendar and Reading List.

Grading

The final grade will be divided as follows:

  • 30% for the four programming assignments

  • 40% project

  • 20% final paper

  • 10% class participation

There will be no midterms or final exam.

Policy on academic honesty

The University of Chicago has a formal policy on academic honesty that you are expected to adhere to.

In brief, academic dishonesty (handing in someone else’s work as your own, taking existing code and not citing its origin, etc.) will not be tolerated in this course. Depending on the severity of the offense, you risk getting a hefty point penalty or being dismissed altogether from the course. All occurrences of academic dishonesty will furthermore be referred to the Dean of Students office, which may impose further penalties, including suspension and expulsion.

Even so, discussing the concepts necessary to complete the programming assignments and the project is certainly allowed (and encouraged). Under no circumstances should you show (or email) another student your code or post your solution to a web-page or social media site. If you have discussed parts of an assignment with someone else, then make sure to say so in your submission (e.g., in a README file or as a comment at the top of your source code file). If you consulted other sources, please make sure you cite these sources.

Finally, if you have any questions regarding what would or would not be considered academic dishonesty in this course, please don’t hesitate to ask the instructor.

Asking questions on Piazza

The preferred form of support for this course is through Piazza (http://www.piazza.com/), an on-line discussion service which can be used to ask questions and share useful information with your classmates. Students will be enrolled in Piazza at the start of the quarter.

All questions regarding assignments or material covered in class must be sent to Piazza, and not directly to the instructors, as this allows your classmates to join in the discussion and benefit from the replies to your question. If you send a message directly to the instructor, you will get a gentle reminder that your question should be asked on Piazza.

Piazza has a mechanism that allows you to ask a private question, which will be seen only by the instructors and teaching assistants. This mechanism should be used only for questions that require revealing part of your solution to an assignment.

Piazza also allows students to post anonymously. Anonymous posts will be ignored (you will also get a gentle reminder asking you to not post anonymously). You are expected to feel comfortable sharing your questions and thoughts with your classmates without hiding behind a veil of anonymity.

Additionally, all course announcements will be made through Piazza. It is your responsibility to check Piazza often to see if there are any announcements. Please note that you can configure your Piazza account to send you e-mail notifications every time there is a new post on Piazza. Just go to your Account Settings, then to Class Settings, click on “Edit Notifications” under CMSC 35200. We encourage you to select either the “Real Time” option (get a notification as soon as there are new posts) or the “Smart Digest” option (get a summary of all the posts sent over the last 1-6 hours – you can select the frequency).

The class Slack channel

We have also set up a Slack channel on the UChicago Slack. This is for informal discussions that are easier to handle there than on Piazza. We won’t use Slack for class announcements.