0

I have a query which took ~30minutes to complete, and the output is unload some parquet files into S3, from Redshift.

I'm using the operator RedshiftToS3Operator, and after ~5 minute I receive this error:

struct.error: unpack_from requires a buffer of at least 5 bytes

Trying a PostgresOperator operator, I'm receiving a different error, after the same 5 minutes:

psycopg2.operationalerror: ssl syscall error: eof detected

With some research, I think this error is because the connection crashes after 5 minutes of idleing. I was able to reproduce this code on a Jupyter Notebook and everything went well, which makes me think that docker was the problem.

All the times, even with the Airflow displaying an error, the query was successfully executed in Redshift.

But I also tried to run the Psycopg2 code instead of using some abstractions, and this time I was able to surpass the 5 minutes, but instead of breaking with 5 min, the Running states just don't update even after the process ends on Redshift.

Basically I'm not able to track if the query works or not, only opening the Redshift UI.

2
  • were you able to resolve this? I'm encountering similar issue
    – ZZZZZZZZZ
    Jul 4, 2022 at 14:09
  • I am also running into a similar issue. The results are getting uploaded to s3 but then at some point, my airflow task fails with the same exception.
    – pyhotshot
    Aug 2, 2022 at 13:08

1 Answer 1

0

Adding this to the Docker compose section of my Airflow scheduler fixed it:

  sysctls:
    - net.ipv4.tcp_keepalive_time=200
    - net.ipv4.tcp_keepalive_intvl=200
    - net.ipv4.tcp_keepalive_probes=5

The TCP settings of the Docker container appear to be the culprit.

From https://github.com/docker/for-mac/issues/3679#issuecomment-642885218

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.