0

I would like to read a GZIP compressed Parquet file from GCS into BigQuery using the Python SDK for Apache Beam. However the apache_beam.io.parquetio.ReadFromParquet method doesn't seem to support reading from compressed files. According to the source code source code, the compression type is hardcoded to UNCOMPRESSED.

Is there a trick to reading compressed Parquet files without the need to uncompress the files in GCS beforehand? If that is the only way, is there a way to uncompress the file directly in GCS?

1 Answer 1

1

I was investigating this issue and I found this issue tracker regarding your question. As you can see currently is not possible to uncompress GZIP files directly without moving out of the bucket.

If you want to get updates for this request you can star it to have some updates when it would be updated or when some more information will be posted from the team in charge.

1
  • Thanks @sneacsu. I've indeed stared the issue and am looking forward for an update there
    – 3thanZ
    Nov 26, 2019 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.