I’ve been spending some time learning deep learning and tensorflow recently, and as part of that project I wanted to be able to train models using GPUs on EC2. This post contains some notes on what it took to get that working. As many people have commented, the environment setup is often the hardest part of getting a deep learning setup going, so hopefully this will be useful reference to someone.
The end result of my work was a packer image definition that
builds an AMI that boots an Ubuntu on an ec2
g2.2xlarge with all the
appropriate drivers and CUDA and cuDNN installed, and a GPU-enabled
tensorflow preinstalled into a virtualenv.
I’m not going to make my AMIs themselves public, since this is a side
project and I don’t want anyone accidentally depending on my AWS
account, but the packer script should Just Work if you
it with appropriate credentials.
I probably could have used someone else’s AMI and saved myself a lot of pain, but I have a near-obsessive need to understand how things work under the hood and Do It Myself.
The rest of this post is some gotchas I learned, that will hopefully help someone else.
Installing the kernel driver
Installing CUDA requires two components; A kernel driver, and the userspace libraries.
The CUDA installer includes a version of the kernel
driver. However, the GRID card in the
g2.2xlarge instances isn’t
supported by the newest NVIDIA driver, so one has to install the
driver separately. Amazon has
documentation on finding a driver, but the
NVIDIA search page it linked gave me a version that
told me it was too new when I actually tried to install it. It pointed
me at the
367.xx series, which ended up working.
NVIDIA’s installers all seem to support automated install modes, but
they aren’t always easy to discover.
--ui=none was the secret here.
Building any kernel module requires kernel headers, and requires that
you build the module for the kernel you’re going to be running. On
Ubuntu you can install
linux-headers-virtual to ensure you always have the latest
headers. However, if you’re not also running the latest kernel,
you’ll be out of luck.
In general (and this is the technique my packer build uses), I find the best luck by:
- Upgrading the kernel (
- Rebooting the machine (to run the latest kernel)
- Explicitly installing
linux-headers-$(uname -r)to be sure you have the right headers.
- Installing the kernel module
If you ever upgrade kernels again, you’ll likely need to reinstall the driver. On ec2 I solve this by rebuilding the AMI if I need a new kernel.
One other gotcha is that, on ec2, you’ll also need the
linux-image-extra-* package. This contains kernel drivers not
normally needed in a virtualized environment. However, that includes
display drivers and their supporting subsystems, and so your modules
will fail to load without it.
Also, by default, once you’ve got the
extra- package, the system
will try to load the upstream
nouveau driver, which will grab the
GPU before NVIDIA’s driver can. To prevent that, we need to
install a blacklist file
The CUDA installer can be downloaded from NVIDIA’s
website. Like the driver, the installer can be used in a headless
mode, but of course it has a different inscrutable set of options. I
found the combination I wanted was the oxymoronic
--toolkit tells it to install the libraries and
--silent actually means “headless mode”, and
--verbose makes it direct output to stdout, as well as a log file
(so that it shows up in the packer output).
I also pass an explicit path for the toolkit,
--toolkitpath=/usr/local/cuda-8.0. A CUDA upgrade broke me when
NVIDIA changed the default path from
/usr/local/cuda-8.0. I tried
the installer appears to blacklist that. So now I use
explicitly, for future-proofing, and create a symlink.
cuDNN requires an NVIDIA developer login, which is free but requires you answer a lot of silly questions about what you’re going to use CUDA and cuDNN for.
Unfortunately, the download links don’t work unless you’re logged in, which is a problem for automated installation. I mirrored the cuDNN installer into an S3 bucket for automated fetching.
Once you’ve got it, the
cuDNN installer works yet again differently
from the previous installers; It’s a tarball that can be unpacked
/usr/local on top of your cuda installation.
Thanks to recent improvements in Python packaging, and some careful
work on Google’s part, installing tensorflow itself is the easy
part. If you’ve got a
virtualenv with a recent
tensorflow-gpu is good enough!
you can also provide a direct link to a
whl file hosted by Google,
so that you don’t depend on PyPI to find the package. I found this
necessary for my install to work in TravisCI, for whatever reason.
NVIDIA makes available pre-built AMIs that probably would work, and there are various services working on providing easier hosted tensorflow environments. So, if you’re lucky, none of this will ever be relevant to you.
But various people still have need to install these tools themselves, or even onto physical hardware. So hopefully this experience report will help someone. Let me know if any of this advice is useful!