Questions tagged [parquet]

Apache Parquet is a columnar storage format for Hadoop.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
23 views

Reading file from parquet

I encountered a challenge while attempting to read a parquet file. Initially, I suspected that the file might be corrupted. However, after altering the method of reading, the file was successfully ...
sea cucumber's user avatar
0 votes
0 answers
42 views

Databricks parquet write jobs resulting in empty files

We have several Databricks PySpark jobs that write parquet files into the same container in Azure Blob Storage. Recently we sometimes get an output file with 0 bytes content even though the source ...
Veronika Vrana's user avatar
0 votes
0 answers
11 views

Azure Event Hubs: Avro v Parquet v Deltalake (preview) - most optimal output format? [closed]

I wanted to get an opinion on a design decision. I am ingesting data using Azure Event Hubs, and this initially ingest goes into a raw data layer. This is then transformed downstream. For this ingest ...
Itachi07's user avatar
1 vote
1 answer
50 views

Reading a column of strings from a parquet file into C++ string variables

I have a parquet file which I have created using the python polars package. It has a single column of variable length strings that looks like: ┌──────────┐ │ str_list │ │ --- │ │ str │ ╞═════...
vinnewbie's user avatar
3 votes
1 answer
95 views

Why are the offsets produced by arrow::ListArray::offsets() interspersed with zeros?

I am trying to read the data in a parquet file containing lists of floats using C++. I produced a simple parquet file using the following python code: import polars as pl import struct import random ...
vinnewbie's user avatar
0 votes
0 answers
36 views

Is it possible to pass the schema as a side input to beam.io.parquetio.WriteToParquet?

When attempting to provide the schema as a side input, beam errors out: Query result is a {"schema": arrow_table.schema, "data": arrow_table.to_pydict()} ... with beam.Pipeline(...
maininformer's user avatar
  • 1,017
0 votes
1 answer
17 views

how to get redshift to recognize the correct format of timestamp in external table of parquet files

I am converting dates in a parquet file like this: timestamp_column = next((col for col in df.columns if 'timestamp' in col), None) if timestamp_column: # Convert epoch to ...
Caullyn's user avatar
  • 156
0 votes
0 answers
24 views

Reading individual elements from a arrow::Array column using GetScalar() not compiling

I am trying to read a parquet file using the Apache Arrow library. The following code opens a parquet file and attempts to read its rows using a arrow::RecordBatchReader. But it fails to build with ...
vinnewbie's user avatar
-2 votes
1 answer
30 views

Put data from Parquet files into DynamoDB with NiFi

I want to integrate data into DynamoDB from Parquet files using NiFi (which I run in a Docker container). I fetch my files from AWS S3 using the ListS3 and FetchS3Object processors and then, as I ...
yan439's user avatar
  • 1
0 votes
0 answers
23 views

python pandas writing parquet file I just read: Conversion failed for column

import pandas as pd file = pd.read_parquet("/Users/myuser/Downloads/input.parquet") file.to_parquet("/Users/hazout/mysuer/output.parquet") I am trying to read and write a parquet ...
Nathan H's user avatar
  • 48.2k
0 votes
0 answers
22 views

Parquet. Compressing column with large plain text values

I have a table in parquet, where, amongst other columns there is a column that contains large plain text values. For example, there is a column that contains uncompressed html webpage content. Values ...
Aleksei Malkov's user avatar
0 votes
0 answers
26 views

How to Read parquet file using apache arrow in cpp from cloud storage S3

the aws path where parquet file is stored will look something like: /aws/default/... I have tried using #include <arrow/filesystem/s3fs.h> but couldnt figure out the exact methods of this ...
vikramjit dutta's user avatar
0 votes
0 answers
31 views

how to save/load pandas dataframe that contain non-ordered dictonary as field without knowing possible key name and value type to/from parquet?

In our application, we need to append segments of dataframe to parquet storage. However the following code will fail at the load time, due to unmatched type import pyarrow as pa import pyarrow.parquet ...
Wang's user avatar
  • 7,461
1 vote
0 answers
30 views

Apache Parquet Writer throws ArrayIndexOutOfBoundsException when writing Avro

I am trying to generate some parquet files from my POJO classes. I am mostly successful in that but AvroWriter is giving me a failure when I try to save a list of custom POJO object. try { ...
Anil's user avatar
  • 23
0 votes
0 answers
21 views

How can I export a 3d xarray as a parquet file in python

I have multiple 3D xarrays (and previous numpy arrays) and I was looking to export them as a parquet file to be able to quickly import them at a later date. Is it even possible to do this as by the ...
oblongcalculator's user avatar
1 vote
0 answers
88 views

Python: write dataset as hive-partitioned and clustered parquet files (no JVM)

I would like write a table stored in a dataframe-like object (e.g. pandas dataframe, duckdb table, pyarrow table) in the parquet format that is both hive partitioned and clustered. Here's what I mean ...
conradlee's user avatar
  • 13.1k
0 votes
0 answers
43 views

Read parquet file in Jupyter Notebook - Python3

I am trying to read a .parquet file in Jupyter Notebook using Python3 but ii get errors. I tired all the solutions that you can normally find in the web, but the only ones that is actually returning a ...
Mining's user avatar
  • 115
0 votes
0 answers
49 views

Unable to read Parquet file with PyArrow: Malformed levels

Assume that I am unable to change how the Parquet file is written, i.e. it is immutable and so we must find a way of reading it given the following complexities... In: import pandas as pd pd....
Tom Bomer's user avatar
0 votes
1 answer
19 views

Save dataframe to parquet with encrypted columns

I would like to save a dataframe to parquet file. However, while writing the data to parquet file, I get the following error df.to_parquet(path_to_data + 'data_temp.gzip', compression='GZIP') ...
bravopapa's user avatar
  • 349
-1 votes
0 answers
38 views

Is there a support to read Parquet files in Gathr?

I'm exploring Gathr for creating a data pipeline, and I'm looking to emit Parquet file data to an S3 location. Can Gathr read data from Parquet files, and which sources within Gathr support Parquet ...
agaonsindhe's user avatar
0 votes
0 answers
41 views

PySpark: MutableLong cannot be cast to MutableInt

I am trying to read multiple parquet files from AWS Glue Job using PySpark and Glue API in the following: snapshot_as_dynamic_frame = glueContext.create_dynamic_frame.from_options( format_options={...
anonymus1205's user avatar
0 votes
1 answer
23 views

Error creating an external table in BQ off partitioned parquet files

I'm trying to create an external table in BigQuery off partitioned parquet data found in GCS. I'm doing this by running the following SQL statement: CREATE OR REPLACE EXTERNAL TABLE `project-id.poc....
Frank Pinto's user avatar
0 votes
1 answer
42 views

Unable to copy parquet data from S3 to redshift

I want to be able to copy parquet data from S3 to Redshift. I have tried the code below: -- Part 1: Unload SET search_path TO my_schema_from; DROP TABLE IF EXISTS my_table; CREATE TABLE my_table ( ...
Sam Gilbert's user avatar
  • 1,672
0 votes
1 answer
62 views

Unexpectedly big parquet file size in Pyspark after sorting

I have a 600Mb .CSV file with 6,5M rows and 10 columns. The columns are mostly ids and sums, but one column "type" has only 2 unique values: "online" and "offline". When ...
sevatster's user avatar
  • 133
0 votes
1 answer
24 views

How to Handle Growing _metadata File Size and Avoid Corruption in Amazon Redshift Spectrum Parquet Append

Context: Our web application generates a lot of log files that arrive in an S3 bucket. The files in the bucket contain JSON strings and have a .txt file format. We process these files in chunks of 200 ...
Aakash's user avatar
  • 39
-1 votes
1 answer
42 views

Load parquet file to S3 from Airflow without using pandas [duplicate]

I was wondering if there was a direct way of uploading a parquet file to S3 without using pandas. I'm using pyarrow and Airflow's S3Hook class. I have a pyarrow.Table object but I cannot find any ...
imatiasmb's user avatar
  • 111
0 votes
0 answers
9 views

parquet.net ParquetOptions.UseDeltaBinaryPackedEncoding how to disable

I am using Parquet.Net library to generate parquet files from C# code. 4.15.0 version works fine, but after switching to latest 4.16.4 the files generated are not readable using ParquetViewer. It ...
MKG's user avatar
  • 57
0 votes
1 answer
59 views

binary format that allows to store multiple pandas dataframes with different columns, width, rows

I have like 200 pandas dataframe, and every dataframe has some unique column, or maybe completely different columns. example: df1 = pd.DataFrame({ 'Product': ['Apple', 'Banana', 'Orange', 'Mango'],...
Abdulrahman Sheikho's user avatar
0 votes
0 answers
22 views

The most efficient way of downloading a parquet file and saving it to local disk for use with pyarrow.memory_map

I have a large input dataset that needs some processing that can be parallelized. I've been able to scale to a fairly large number of parallel processes by first storing a huge input pyarrow table to ...
Sergey's user avatar
  • 36
0 votes
0 answers
35 views

Can't make unit tests for reading parquet files created from protobuf messages

I wrote a parquet file from a protobuf message as such ProtobufData model = ProtobufData.get(); Schema schema = model.getSchema(ExampleMessage.class); Configuration conf = new Configuration(); ...
ANIMO's user avatar
  • 15
0 votes
0 answers
71 views

Extracting lots of data from SQL Server into parquet file efficiently?

I have data in SQL Server and I need to save it in parquet format. Right now I'm using pyodbc and pyarrow libraries for this. As there are serveral million rows, I'm doing it by using queries within a ...
imatiasmb's user avatar
  • 111
0 votes
1 answer
53 views

What is the "Parameters" field of an AWS Glue Data Catalog table column?

The AWS documentation for Glue Data Catalog tables notes that a column has four fields: Name Type Comment Parameters The first three are easy enough to understand; here is an excerpt from my ...
Andrew Parsons's user avatar
0 votes
1 answer
108 views

Reading parquet files in Databricks from Azure Data Lake results in java.lang.OutOfMemoryError: Java heap space

We have an Azure Databricks Notebook that is reading parquet data files from our "RAW" folder that has been streamed from Azure Eventhub into Azure Data Lake in Parquet format and ...
JonJagd's user avatar
  • 65
0 votes
1 answer
32 views

pyarrow write dataset drops partition columns

I am trying to use pyarrow to partition and write parquet files !pip install pyarrow==13.0.0 import pyarrow as pa table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021], 'n_legs'...
Swavig's user avatar
  • 37
0 votes
1 answer
23 views

retro-actively add partitions to parquet files?

i have a spark job that uses Apache Hudi to write parquet into our AwS S3 data lake. I have a pretty decent sized dataset (about ~20M rows and growing) that i would like to add a new partition to. Is ...
lollerskates's user avatar
  • 1,004
0 votes
1 answer
41 views

Applying custom functions to parquet files loaded with open_dataset

Apologies if this has been applied somewhere else, but I haven't been able to find a solution that works. I have some data, spread across 4232 parquet files, that aggregate into roughly 6745697 rows ...
TDeramus's user avatar
  • 117
0 votes
1 answer
62 views

Dask dataframe -containing columns that are numpy arrays- conversion to parquet error

I have a dask dataframe which I am trying to convert to parquet files. This dataframe has columns which are numpy arrays, but column type is "object". When I try to do this: name_function = ...
Ahmed Gamal's user avatar
0 votes
0 answers
24 views

Error and Partition Adjustment Issue when Saving Parquet Files in PySpark

I'm encountering an issue when attempting to save parquet files using PySpark. It appears that I'm running into an error, and I suspect it may be related to the number of partitions I'm working with. ...
alldaycoding21's user avatar
0 votes
1 answer
46 views

Are parquet files written incrementally via pyarrow's ParquetWriter visible to others before the writer has been closed?

I am trying to figure out whether a parquet file being written to incrementally by pyarrow's ParquetWriter would be visible to others before the close() method is called. I've tried running some ...
Gabriele Giuseppini's user avatar
0 votes
1 answer
61 views

Emoticons or Emoji issue in Azure synapse for Parquet data

I have a parquet data there are some emoticons when I opened this in any online parquet viewer it showing data with emoticons same data if query in synapse instead of emoticons it shows (?? or \uD83E\...
Rishi's user avatar
  • 3
0 votes
1 answer
53 views

What is causing malformed level error on parquet file generated via stream analytics?

I have a service bus topic subscription and a logic app that sends the data to event hub, this data is read by stream analytics and a parquet file is generated and stored in data lake. I am then ...
data_and_coffee's user avatar
1 vote
0 answers
24 views

How to handle parquet file (read&wite) in Android Studio?

I'm develloping an app with Android Studio and I need to send a parquet file to a server after gathering enough info (mostly FLOAT32 and INT4). Does anyone know a way to write a parquet file in ...
jeremy chretien's user avatar
0 votes
1 answer
60 views

Remove metadata properties from JSON ADF

I have json file in ADLS and wanted to remove metadata properties from json file and convert it to parquet using ADF pipelines or using pyspark dataframe. An example of this data is the following [ ...
SHIVAM YADAV's user avatar
1 vote
1 answer
87 views

Microsoft Fabric-Delta parquet format?

I started learning Microsoft fabric. As per Microsoft document data's are stored in Delta-Parquet format. I searched over internet I found both are different file fomats (delta format /parquet format)....
Shivanesh Raghuraman's user avatar
0 votes
1 answer
249 views

DuckDb - Reading parquet files from S3 - Handling Segmentation Fault Core Dump error

I receive 1000s of parquet files with same schema every day into a S3 bucket. I am using duckdb with python3 extension to read all the parquet files to subset data from them. The following is the ...
Apricot's user avatar
  • 2,969
0 votes
1 answer
75 views

How to create Athena table through parquet from pandas?

I start with a pandas Dataframe and saved to parquet format as such: import pandas as pd df = pd.DataFrame([ {'abc': 1.231, 'xyz':2, 'jkl': False, 'idx': 'first row', 'date': '2023-12-01'}, {'...
alvas's user avatar
  • 116k
0 votes
1 answer
88 views

Permission error, having trouble loading parquet files from AWS S3 to SQL Server 2022

Basically every time I try to connect to AWS bucket to grab a parquet file in SQL Server 2022, I am getting a permission error Cannot find the CREDENTIAL 's3_dc', because it does not exist or you do ...
Hazem Mohamed's user avatar
0 votes
0 answers
29 views

Decimal conversion using AvroSchema/Apache Beam for Parquet file is failing

I am trying to convert a Parquet file, after reading using it, to Beam Rows using AvroUtils and PCollection. It is failing with ClassCastException to convert a Dcimal (38, 0) column. org.apache.beam....
Damacharla Ashok Kumar's user avatar
0 votes
1 answer
63 views

Reading parquet file as is using spark

I want to read parquet file as is using spark and process each file's content one by one. I was trying to achieve it using following approach spark.read .option("wholetext", "...
Akhtem's user avatar
  • 1
0 votes
1 answer
54 views

Direct Parquet source (Spark Engine) vs "SQL-Select external table (pointing to that Parquet) on Serverless SQL" source (SQL Engine)

In a pipeline, in a Data Flow, I can use as a source: the parquet file or a sql query to an external table on the serverless sql, that points to the parquet file Assuming that the sink is another ...
Hernán's user avatar
  • 45

1
2 3 4 5
79