1

I have been implementing a suite of RecordBatchReaders for a genomics toolset. The standard unit of work is a RecordBatch. I ended up implementing a lot of my own compression and IO tools instead of using the existing utilities in the arrow cpp platform because I was confused about them. Are there any clear examples of using the existing compression and file IO utilities to simply get a file stream that inflates standard zlib data? Also, an object diagram for the cpp platform would be helpful in ramping up.

1 Answer 1

0

Here is an example program that inflates a compressed zlib file and reads it as CSV.

#include <iostream>

#include <arrow/api.h>
#include <arrow/csv/api.h>
#include <arrow/io/api.h>
#include <arrow/util/compression.h>
#include <arrow/util/logging.h>

arrow::Status RunMain(int argc, char **argv) {

  if (argc < 2) {
    return arrow::Status::Invalid(
        "You must specify a gzipped CSV file to read");
  }

  std::string file_to_read = argv[1];
  ARROW_ASSIGN_OR_RAISE(auto in_file,
                        arrow::io::ReadableFile::Open(file_to_read));
  ARROW_ASSIGN_OR_RAISE(auto codec,
                        arrow::util::Codec::Create(arrow::Compression::GZIP));
  ARROW_ASSIGN_OR_RAISE(
      auto compressed_in,
      arrow::io::CompressedInputStream::Make(codec.get(), in_file));

  auto read_options = arrow::csv::ReadOptions::Defaults();
  auto parse_options = arrow::csv::ParseOptions::Defaults();
  auto convert_options = arrow::csv::ConvertOptions::Defaults();
  ARROW_ASSIGN_OR_RAISE(
      auto table_reader,
      arrow::csv::TableReader::Make(arrow::io::default_io_context(),
                                    std::move(compressed_in), read_options,
                                    parse_options, convert_options));

  ARROW_ASSIGN_OR_RAISE(auto table, table_reader->Read());
  std::cout << "The table had " << table->num_rows() << " rows and "
            << table->num_columns() << " columns." << std::endl;

  return arrow::Status::OK();
}

int main(int argc, char **argv) {
  arrow::Status st = RunMain(argc, argv);
  if (!st.ok()) {
    std::cerr << st << std::endl;
    return 1;
  }
  return 0;
}

Compression is handled in different ways in different parts of Arrow. The file readers typically accept an arrow::io::InputStream. You should be able to use arrow::io::CompressedInputStream to wrap an arrow::io::InputStream with decompression. This gives you whole-file compression. This is fine for something like CSV.

For Parquet, this approach does not work (ParquetFileReader::Open expects arrow::io::RandomAccessFile). For IPC, this approach is inefficient (unless you are reading the entire file). Effective reading of these formats involves seekable reads which is not possible with whole-file compression. Both formats support their own format-specific compression options. You only need to specify these options on write. On read the compression will be detected from the metadata (the metadata is stored uncompressed) of the file itself. If you are writing data you can find the information in parquet::ArrowWriterProperties and arrow::ipc::WriteOptions.

Since whole-file compression is still a thing for CSV the datasets API has recently (as of 4.0.0) added support for detecting compression from file extensions for CSV datasets. More details can be found here.

As for documentation and an object diagram, those are excellent topics for the user mailing list, or you are welcome to provide a pull request.

Not the answer you're looking for? Browse other questions tagged or ask your own question.