2

Context

I am running Airflow, and trying to run a proof of concept for a Docker container using Airflow's DockerOperator. I am deploying to Kubernetes (EKS), but not using Kubernetes Executor yet. Given that I am running pods, by using the DockerOperator I will be running (to my understanding) Docker in Docker.

Whenever I run my task, I am receiving the Error: ERROR - Error while fetching server API version. The error happens both on docker-compose as well as EKS (kubernetes).

My current status

Thi is how my airflow Dockerfile looks like:

FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

This is how the dag I am trying to run looks like:

with DAG(
        dag_id='task',
        default_args=dict(
            start_date=days_ago(0)
        ),
        schedule_interval='@daily'
) as dag:

    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30'
    )

This is the stack trace of the error I am getting:

[2020-12-29 14:18:52,601] {taskinstance.py:1150} ERROR - Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 260, in execute
    self.cli = APIClient(
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

What I have tried

  1. Mount the socket into docker compose /var/run/docker.sock:/var/run/docker.sock:ro

First, that gives me a new error to:

ERROR - Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Second, how will I be able to mount on Kubernetes? I would guess that complicates things

  1. Install docker within my container and try to give privileges to the airflow user like:
FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Docker
RUN curl -sSL https://get.docker.com/ | sh  
# Usermod
RUN usermod -aG docker airflow
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

But that also did not work.

  1. Mounting the socket into the DockerOperator task like:
    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30',
        volumes=['/var/run/docker.sock:/var/run/docker.sock:ro'],
    )

But that also has had no effect

4
  • 1
    This is not recommended. You should use the KubernetesPodOperator instead. Mounting in the docker control socket will only work if you carefully align the file permissions and if you are using Docker on the underlying host, which is deprecated and pending removal in a few releases.
    – coderanger
    Commented Dec 30, 2020 at 6:40
  • @coderanger makes sense. The issue I am having with KubernetesPodOperator is that it keeps creating multiple pods even after the tasks succeed. I will probably voer that in a separate question
    – alt-f4
    Commented Dec 30, 2020 at 6:46
  • 1
    You can kubectl exec in to check the permission on the docker socket. It's usually something like root:docker and 0660. So you would have to setgid your airflow process to the gid of docker on the underlying host (or run it as uid 0 but please please don't do that).
    – coderanger
    Commented Dec 30, 2020 at 6:58
  • @coderanger If you want, please group these details of why it is not a good practice in an answer, and I am happy to accept it = )
    – alt-f4
    Commented Dec 30, 2020 at 7:02

1 Answer 1

1

Copying down from comment:

The direct issue is likely that the docker control socket file is owned by something like root:docker on the host, and assuming Airflow isn't running as root (which it shouldn't) then you would need to specifically set it to run with the same gid as the docker group on the host machine (which probably has no matching group inside the container, which could confuse other things).

The real answer is to use the KubernetesPodOperator instead. DockerOperator won't work at all once you stop using Docker on the underlying host which will be happening soon now that dockershim is deprecated.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.