23

I am working on moving the build for a C++ project into a docker image. The image will be built and pushed by a Jenkins job. Prior to docker, I made heavy use of ccache to speed up my builds on Jenkins, especially in the case of builds where very little changed. The trouble with docker is that the build now runs in an isolated environment, so I can no longer benefit from ccache. Is there a way to build inside of an ephemeral container while still taking advantage of ccache?

2 Answers 2

20

You can still use ccache in conjunction with your build.

Create a Data Volume to allow data to persist between compilations/builds using the following command:

$ docker create -v /mnt/ccache:/ccache --name ccache debian

Then create your container that “mounts” the data container created above using the --volumes-from command line option.

$ docker run -e CCACHE_DIR=/ccache --volumes-from ccache -it debian

Now you'll be in the shell of the debian container and can install the required apps and test ccache:

root@15306d02505a:/# apt-get update && apt-get install -y gcc ccache    

Now at this point you can check the cache, and it will be empty as expected:

root@15306d02505a:/# ccache -s
cache directory                     /ccache
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
files in cache                         0
cache size                             0 Kbytes
max cache size                       1.0 Gbytes

The data volume will persist, so even after the container is terminated, the cache is still there. Future builds that mount the volume (and specify the -e ENV variable) will utilize the cache.

Then create a simple app, run it, and check the cache again:

root@15306d02505a:/# cat > foo.c << __EOF__
 int main(int argc, char **argv)
 {
     return 0;
 }
 __EOF__

root@15306d02505a:/# PATH=/usr/lib/ccache:$PATH gcc -o foo.o -c foo.c
root@15306d02505a:/# ccache -s
cache directory                     /ccache
cache hit (direct)                     1
cache hit (preprocessed)               0
cache miss                             1
files in cache                         2
cache size                             8 Kbytes
max cache size                       1.0 Gbytes

You can see the cache is now populated, and further builds will see performance improvements because of it.

The data volume will persist, so even after the container is terminated, the cache is still there. Future builds that mount the volume (and specify the -e ENV variable) will utilize the cache.

This blog post does a good job of explaining it:

Using Ccache with Docker

8
  • 4
    The trouble is (and I might just be misunderstanding) that I am wanting to create an image as a result of my build. This approach seems to let me use ccache in a container but not with a docker build command. Commented Sep 23, 2016 at 4:25
  • 1
    You could add ccache into your Dockerfile using the RUN command, or use Docker Compose to mount & use the volume during docker-compose build. You can also save a container to an image using docker commit.
    – DevOps Dan
    Commented Sep 23, 2016 at 4:33
  • Yeah, I think combining the above with "docker commit" is probably the best route. Thank you. Commented Sep 23, 2016 at 4:59
  • 1
    Will this setup have any issues with multiple containers and builds accessing the same cache?
    – ad22
    Commented Feb 2, 2017 at 4:15
  • 4
    This is not an answer to the question, its a good one for runtime but not build time. If i find a correct answer I will post back
    – krad
    Commented Jul 1, 2019 at 8:30
15

OK as promised.

Prereqs

  1. Be on Docker 18.06 or above

  2. Run in experimental mode, on client and server (at time of writing)

Setup experimental Mode

Server: I created the following systemd override file:

cat /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --experimental -H fd:// 

I then did a systemctl daemon-reload ; systemctl restart docker.service

Client side you need to set and env variable export DOCKER_BUILDKIT=1

Docker file

This is the basis of the docker file. I am building a php image, but you can build whatever you want. The key parts of the 1st line comment (needed to tell docker to parse the file in experimental mode), and the --mount=type=cache,target=/ccache/ option. This pulls in the cache folder for that stage of the build. Make sure you put it on every RUN line where you trigger a compile

# syntax = docker/dockerfile:experimental
FROM php:7.3-fpm-stretch

# Create app directory
WORKDIR /usr/src/app

# setup locales
RUN apt update && \
  apt install -y locales && \
  sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \
  locale-gen

ENV LANGUAGE=en_GB.UTF-8
ENV LANG=en_GB.UTF-8
ENV LC_ALL=en_GB.UTF-8
ENV CCACHE_DIR=/ccache

RUN apt update ; apt install -yq \
        git \
        cloud-guest-utils \
        iproute2 \
        libxml2-dev \
        libxslt1-dev \
        libmemcached-dev \
        libgd-dev \
        libzip-dev \
        libmemcached-dev \
        ccache \
        awscli


# use ccache (make it appear in path earlier then /usr/bin/gcc etc)
RUN for p in gcc g++ cc c++; do ln -vs /usr/bin/ccache /usr/local/bin/$p;  done

# prod format
RUN --mount=type=cache,target=/ccache/ docker-php-source extract && \
    docker-php-ext-install \
    intl \
    bcmath  \
    calendar \
    exif \
    gd \
    gettext \
    mysqli \
    opcache \
    pcntl \
    pdo_mysql \
    shmop \
    soap \
    sockets \
    sysvmsg \
    sysvsem \
    sysvshm \
    xsl \
    zip \
  && \
  docker-php-source delete
RUN --mount=type=cache,target=/ccache/ ccache -s

Sample output

I am running the build with the --progress plain flag as the output in experimental is very different

#25 [16/16] RUN --mount=type=cache,target=/ccache/ ccache -s 
#25 digest: sha256:98c661a0404c71176a4bbf7d02123184524a784fabb2575d5210da088f16ee3a 
#25 name: "[16/16] RUN --mount=type=cache,target=/ccache/ ccache -s" 
#25 started: 2019-07-01 09:35:15.158199623 +0000 UTC 
#25 0.468 cache directory /ccache 
#25 0.468 primary config /ccache/ccache.conf 
#25 0.468 secondary config (readonly) /etc/ccache.conf 
#25 0.468 cache hit (direct) 2450 
#25 0.468 cache hit (preprocessed) 152 
#25 0.468 cache miss 590 
#25 0.468 cache hit rate 81.52 % 
#25 0.468 called for link 163 
#25 0.468 called for preprocessing 1011 
#25 0.468 compile failed 33 
#25 0.468 preprocessor error 3 
#25 0.468 bad compiler arguments 188 
#25 0.468 autoconf compile/link 943 #25 0.468 no input file 554 
#25 0.468 cleanups performed 0 
#25 0.468 files in cache 1288 #25 0.468 cache size 20.6 MB
#25 0.468 max cache size 5.0 GB 
#25 completed: 2019-07-01 09:35:15.732702254 +0000 UTC 
#25 duration: 574.502631ms

More readings here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md https://docs.docker.com/engine/reference/commandline/dockerd/#description

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.