Can Docker Use GPU? A Comprehensive Guide

Estimated reading time: 7 minutes

In the modern computing world, GPUs ( Graphics Processing Units ) have become extremely important for handling data-intensive tasks such as machine learning, AI, and large-scale analytics and business intelligence.

Docker is the most popular platform for running container workloads at scale. Docker is the best choice for running your application and webservers at scale without worrying about capacity.

But the question is can Docker use GPU? How can Docker container can take leverages the power of GPUs? Can we even use a Docker container for the ML workload or data-intensive tasks?

The short answer is yes, they can use it.

In this guide, let’s understand how can Docker use GPUs and learn everything about setup and best practices.

Understanding Docker and GPU Compatibility

What is Docker?

Docker allows the developer to package the entire application into a lightweight, standalone, and executable package that includes app dependencies required for the application called the containers.

As mentioned the container includes everything your application needs to run: code, system libraries, runtime, and dependencies.

If you want to know everything about Docker and provide in-depth explanations, and best practices check out: Mastering Docker: A Comprehensive Guide

The Role of GPUs in Modern Computing

CPUs are powerful and provide all the processing power it needs to keep the computer running. GPUs on the other hand are excellent choices for parallel processing you can think that as more specialized for rendering and graphic-related tasks.

GPUs have special use cases such as deep learning, scientific research, and video rendering, where computational demands are high and need parallel processing.

DevOps and AI: 4 Scary Predictions for 2024

Can Docker Use GPUs?

The answer is YES!

Docker containers can use GPUs but using them inside your container does require the specific setups and configuration in place.

Docker and GPUs can work together with the NVIDIA Container Toolkit. Installing and running the container with this helps to utilize the GPUs for the task requiring performance and data processing.

Let’s understand how to set the docker and the required configuration for Docker to use GPU.

Setting Up Docker Use GPUs

Prerequisites for GPU-Enabled Docker Containers

Before you can run Docker containers with GPU support, you need to check your system meets the following prerequisites:

Installing NVIDIA Container Toolkit

Let’s get started with installing the NVIDIA Container Toolkit. Below is a step-by-step guide for installing it on an Ubuntu system:

1. Install Docker: If Docker isn’t installed, you can install it using:

sudo apt-get update
sudo apt-get install -y docker.io

2. Add the NVIDIA package repositories and the GPG key:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

3. Install the NVIDIA Container Toolkit:

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Configuring Docker to Access GPUs

With the NVIDIA Container Toolkit installed, you can now run Docker containers with GPU support. Before that let’s configure the container runtime by using the nvidia-ctk command:

sudo nvidia-ctk runtime configure --runtime=docker

The above command will modify the /etc/docker/daemon.json file and set the runtime for the NVIDIA.

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

Apply the configuration changes by restarting Docker:

sudo systemctl restart docker

Configuring Rootless Docker to Access GPUs

If you follow the Docker security best practices and run the Docker container in rootless mode, you can follow this guide to configure the Docker to access the GPUs.

The only difference with the running as root and rootless is the Docker daemon file is different and per user basis. Let’s configure the rootless Docker use GPU with the nvidia-ctk command:

nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json

The above command will modify the $HOME/.config/docker/daemon.json file and set the runtime for the NVIDIA.

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

Now let’s restart the Rootless Docker daemon:

systemctl --user restart docker

Configuring containerd for Kubernetes to Access GPUs

NVIDIA provides support for using GPUs with containerd for the Kubernetes using the nvidia-ctk with the below command:

sudo nvidia-ctk runtime configure --runtime=containerd

The above command will modify the /etc/containerd/config.toml and updated so that containerd can use the NVIDIA Container Runtime.

Apply the configuration changes by restarting containerd:

sudo systemctl restart containerd

So far we have learned about rootless Docker, containerd for Kubernetes & Docker use GPUs along with the configuration.

Best Practices for Using GPUs with Docker

1. Ensure Proper GPU Isolation

Use the --gpus flag in Docker to specify the number of GPUs or limit access to specific GPUs. Isolates GPU resources, preventing resources from having exclusive access to the specified GPUs.

docker run --gpus '"device=0"' ubuntu nvidia-smi

2. Use the Latest NVIDIA Docker Toolkit

Keep the NVIDIA Container Toolkit up to date to take advantage of the latest performance improvements, bug fixes, and security patches.

This ensures compatibility with the latest versions of Docker and CUDA.

sudo apt-get update && sudo apt-get upgrade nvidia-container-toolkit

3. Optimizing Docker Use GPU

Use lightweight base images that include only the necessary libraries and dependencies for your applications.

The official NVIDIA CUDA images are optimized for performance and include all necessary drivers and libraries.

FROM nvidia/cuda

4. Implement Security Best Practices

Apply the principle of least privilege by running containers as non-root users whenever possible.

Learn more about Docker Security, with different types of attack surfaces and actions that make your deployments more secure.

Docker Container Security Cheatsheet: Don’t Get Hacked🔐

These best practices help you maximize the performance and reliability of GPU-accelerated workloads where Docker use GPU.

Always refer to the latest official documentation to keep your configurations up to date.

Troubleshooting GPU Issues in Docker

Conclusion

Docker provides the flexibility to utilize GPUs for high-performance computing making it easy for data-intensive applications to run inside the container.

Whether you’re deploying machine learning models or conducting scientific research, Docker with GPU support can significantly enhance performance and efficiency.

Additional Reading

Mastering Docker: A Comprehensive Guide

10 Docker Network Best Practices: For Optimal Container Networking