I have a query which took ~30minutes to complete, and the output is unload some parquet files into S3, from Redshift.
I'm using the operator RedshiftToS3Operator, and after ~5 minute I receive this error:
struct.error: unpack_from requires a buffer of at least 5 bytes
Trying a PostgresOperator operator, I'm receiving a different error, after the same 5 minutes:
psycopg2.operationalerror: ssl syscall error: eof detected
With some research, I think this error is because the connection crashes after 5 minutes of idleing. I was able to reproduce this code on a Jupyter Notebook and everything went well, which makes me think that docker was the problem.
All the times, even with the Airflow displaying an error, the query was successfully executed in Redshift.
But I also tried to run the Psycopg2 code instead of using some abstractions, and this time I was able to surpass the 5 minutes, but instead of breaking with 5 min, the Running states just don't update even after the process ends on Redshift.
Basically I'm not able to track if the query works or not, only opening the Redshift UI.