I too had this question recently, and I don't have a complete answer, but here is what I do know, or at least believe:
Setup & Test
The magic to setup - required once per reboot of system, is just this:
# start root's docker (not via any `-rootless` scripts, obviously)
sudo systemctl start docker
# setup QEMU static executables formats
sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
# test
docker run --rm -t arm64v8/ubuntu uname -m
# shoudl expect:
# >> aarch64
# optional: shutdown root's docker
sudo systemctl stop docker
Note that the test example assumes that you are running that your own personal "rootless-"docker
, therefore as yourself, not as root
(nor via sudo
), and it works just dandy.
Gory Details
... which are important if you want to understand how/why this works.
The main sources for this info:
The fundamental trick to making this work is to install new "magic" strings into the kernel process space so that when an (ARM) executable is run inside a docker image, it recognizes the bin-fmt and uses the QEMU interpreter (from the multiarch/*
docker image) to execute it. Before we setup the bin formats, the contents look like this:
root@odysseus # mount | grep binfmt_misc
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=45170)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
root@odysseus # ls /proc/sys/fs/binfmt_misc/
jar llvm-6.0-runtime.binfmt python2.7 python3.6 python3.7 python3.8 register sbcl status
After we start (root's) dockerd
and setup the formats:
root@odysseus # systemctl start docker
root@odysseus # docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
[...]
root@odysseus # ls /proc/sys/fs/binfmt_misc/
jar python3.8 qemu-armeb qemu-microblazeel qemu-mipsn32 qemu-ppc64le qemu-sh4eb qemu-xtensaeb
llvm-6.0-runtime.binfmt qemu-aarch64 qemu-hexagon qemu-mips qemu-mipsn32el qemu-riscv32 qemu-sparc register
python2.7 qemu-aarch64_be qemu-hppa qemu-mips64 qemu-or1k qemu-riscv64 qemu-sparc32plus sbcl
python3.6 qemu-alpha qemu-m68k qemu-mips64el qemu-ppc qemu-s390x qemu-sparc64 status
python3.7 qemu-arm qemu-microblaze qemu-mipsel qemu-ppc64 qemu-sh4 qemu-xtensa
Now we can run an ARM version of ubuntu:
root@odysseus # docker run --rm -t arm64v8/ubuntu uname -m
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested
aarch64
The warning is to be expected since the host CPU is AMD, and can be gotten rid of by specifying the platform to docker:
root@odysseus # docker run --rm --platform linux/arm64 -t arm64v8/ubuntu uname -m
aarch64
How does this really work?
At the base of it is just QEMU's ability to interpose a DBM (dynamic binary modification) interpreter to translate the instruction set of one system to that of the underlying platform.
The only trick we have to do is tell the underlying system where to find those interpreters. Thats what the qemu-user-static
image does in registering the binary format magic strings / interpreters. So, what's in those binfmt
s?
root@odysseus # cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff
Huh - that's interesting, especially because on the host system there is no /usr/bin/qemu-aarch64-static
, and it's not in the target image either, so where does this thing live? It's in the qemu-user-static
image itself, with the appropriate tag of the form: <HOST-ARCH>-<GUEST-ARCH>
, as in multiarch/qemu-user-static:x86_64-aarch64
.
# Not on the local system
odysseus % ls /usr/bin/qemu*
ls: cannot access '/usr/bin/qemu*': No such file or directory
# Not in the target image
odysseus % docker run --rm --platform linux/arm64 -t arm64v8/ubuntu bash -c 'ls /usr/bin/qemu*'
/usr/bin/ls: cannot access '/usr/bin/qemu*': No such file or directory
# where is it?
odysseus % docker run --rm multiarch/qemu-user-static:x86_64-aarch64 sh -c 'ls /usr/bin/qemu*'
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown.
# Hmm, no `sh` in that image - let's try directly...
odysseus % docker run --rm multiarch/qemu-user-static:x86_64-aarch64 /usr/bin/qemu-aarch64-static --version
qemu-aarch64 version 7.0.0 (Debian 1:7.0+dfsg-7)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
# AHA - there it is.
That's the real magic that I don't yet quite understand. Somehow docker
is, I believe, using that image to spin up the QEMU interpreter, and then feeding it the code from the actual image/container you want to run, as in the uname
example from earlier. Some web-searching left me unsatiated as to how this magic is achieved, but I'm guessing if I kept following links from here I might find the true source of that slight-of-hand.