Context
I am running Airflow, and trying to run a proof of concept for a Docker container using Airflow's DockerOperator. I am deploying to Kubernetes (EKS), but not using Kubernetes Executor yet. Given that I am running pods, by using the DockerOperator
I will be running (to my understanding) Docker in Docker.
Whenever I run my task, I am receiving the Error: ERROR - Error while fetching server API version
. The error happens both on docker-compose
as well as EKS (kubernetes).
My current status
Thi is how my airflow Dockerfile looks like:
FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt
This is how the dag I am trying to run looks like:
with DAG(
dag_id='task',
default_args=dict(
start_date=days_ago(0)
),
schedule_interval='@daily'
) as dag:
task_1 = DockerOperator(
dag=dag,
task_id='docker_task',
image='centos:latest',
api_version="auto",
docker_url='unix://var/run/docker.sock',
command='/bin/sleep 30'
)
This is the stack trace of the error I am getting:
[2020-12-29 14:18:52,601] {taskinstance.py:1150} ERROR - Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
return self.request('GET', url, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 260, in execute
self.cli = APIClient(
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
What I have tried
- Mount the socket into docker compose
/var/run/docker.sock:/var/run/docker.sock:ro
First, that gives me a new error to:
ERROR - Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
Second, how will I be able to mount on Kubernetes? I would guess that complicates things
- Install docker within my container and try to give privileges to the airflow user like:
FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Docker
RUN curl -sSL https://get.docker.com/ | sh
# Usermod
RUN usermod -aG docker airflow
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt
But that also did not work.
- Mounting the socket into the DockerOperator task like:
task_1 = DockerOperator(
dag=dag,
task_id='docker_task',
image='centos:latest',
api_version="auto",
docker_url='unix://var/run/docker.sock',
command='/bin/sleep 30',
volumes=['/var/run/docker.sock:/var/run/docker.sock:ro'],
)
But that also has had no effect
kubectl exec
in to check the permission on the docker socket. It's usually something likeroot:docker
and 0660. So you would have to setgid your airflow process to the gid ofdocker
on the underlying host (or run it as uid 0 but please please don't do that).