Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Monday Nov 6, 2023, 9:00 AM → 5:00 PM Europe/Berlin

Modern deep learning challenges leverage increasingly larger datasets and more complex models. As a result, significant computational power is required to train models effectively and efficiently. Learning to distribute data across multiple GPUs during deep learning model training makes possible an incredible wealth of new applications utilizing deep learning.

Additionally, the effective use of systems with multiple GPUs reduces training time, allowing for faster application development and much faster iteration cycles. Teams who are able to perform training using multiple GPUs will have an edge, building models trained on more data in shorter periods of time and with greater engineer productivity.

This workshop teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

Learning Objectives

By participating in this workshop, you’ll:

• Understand how data parallel deep learning training is performed using multiple GPUs
• Achieve maximum throughput when training, for the best use of multiple GPUs
• Distribute training to multiple GPUs using Pytorch Distributed Data Parallel
• Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Workshop Details

Duration: 8 hours

Prerequisites: Experience with deep learning training using Python

Technologies: PyTorch, PyTorch Distributed Data Parallel, NCCL

Assessment Type: Skills-based coding assessments evaluate learners' ability to train deep learning models on multiple GPUs.

Certificate: Upon successful completion of the assessment, participants will receive an NVIDIA DLI certificate to recognize their subject matter competency and support professional career growth.

Hardware Requirements:  Desktop or laptop computer capable of running the latest version of Chrome or Firefox. Each participant will be provided with dedicated access to a fully configured, GPU-accelerated workstation in the cloud.

Language: English

Price: No course fees apply.

Registration Deadline: 01. November 2023 at 23:59

dli-datasheet-il-workshop-data-parallelism-multiple-gpus-2527972-r3-web.pdf

1 Welcome and Introduction

• Meet the instructor.
• Create an account at courses.nvidia.com/join

2 Stochastic Gradient Descent and the Effects of Batch Size

Learn the significance of stochastic gradient descent when training on multiple GPUs

• Understand the issues with sequential single-thread data processing and the theory behind speeding up applications with parallel processing.
• Understand loss function, gradient descent, and stochastic gradient descent (SGD).
• Understand the effect of batch size on accuracy and training time with an eye towards its use on multi-GPU systems.

11:15 AM Break

3 Training on Multiple GPUs with PyTorch Distributed Data Parallel (DDP)

Learn to convert single GPU training to multiple GPUs using PyTorch Distributed Data Parallel

• Understand how DDP coordinates training among multiple GPUs.
• Refactor single-GPU training programs to run on multiple GPUs with DDP.

2:15 PM Break

4 Maintaining Model Accuracy when Scaling to Multiple GPUs

Understand and apply key algorithmic considerations to retain accuracy when training on multiple GPUs

• Understand what might cause accuracy to decrease when parallelizing training on multiple GPUs.

• Learn and understand techniques for maintaining accuracy when scaling training to multiple GPUs.

5 Workshop Assessment

Use what you have learned during the workshop: complete the workshop assessment to earn a certificate of competency

6 Final Review

• Review key learnings and wrap up questions.
• Take the workshop survey.