Questions tagged [parquet]
Apache Parquet is a columnar storage format for Hadoop.
3,944
questions
-1
votes
0
answers
23
views
Reading file from parquet
I encountered a challenge while attempting to read a parquet file. Initially, I suspected that the file might be corrupted. However, after altering the method of reading, the file was successfully ...
0
votes
0
answers
42
views
Databricks parquet write jobs resulting in empty files
We have several Databricks PySpark jobs that write parquet files into the same container in Azure Blob Storage. Recently we sometimes get an output file with 0 bytes content even though the source ...
0
votes
0
answers
11
views
Azure Event Hubs: Avro v Parquet v Deltalake (preview) - most optimal output format? [closed]
I wanted to get an opinion on a design decision. I am ingesting data using Azure Event Hubs, and this initially ingest goes into a raw data layer. This is then transformed downstream.
For this ingest ...
1
vote
1
answer
50
views
Reading a column of strings from a parquet file into C++ string variables
I have a parquet file which I have created using the python polars package. It has a single column of variable length strings that looks like:
┌──────────┐
│ str_list │
│ --- │
│ str │
╞═════...
3
votes
1
answer
95
views
Why are the offsets produced by arrow::ListArray::offsets() interspersed with zeros?
I am trying to read the data in a parquet file containing lists of floats using C++.
I produced a simple parquet file using the following python code:
import polars as pl
import struct
import random
...
0
votes
0
answers
36
views
Is it possible to pass the schema as a side input to beam.io.parquetio.WriteToParquet?
When attempting to provide the schema as a side input, beam errors out:
Query result is a {"schema": arrow_table.schema, "data": arrow_table.to_pydict()}
...
with beam.Pipeline(...
0
votes
1
answer
17
views
how to get redshift to recognize the correct format of timestamp in external table of parquet files
I am converting dates in a parquet file like this:
timestamp_column = next((col for col in df.columns if 'timestamp' in col), None)
if timestamp_column:
# Convert epoch to ...
0
votes
0
answers
24
views
Reading individual elements from a arrow::Array column using GetScalar() not compiling
I am trying to read a parquet file using the Apache Arrow library. The following code opens a parquet file and attempts to read its rows using a arrow::RecordBatchReader. But it fails to build with ...
-2
votes
1
answer
30
views
Put data from Parquet files into DynamoDB with NiFi
I want to integrate data into DynamoDB from Parquet files using NiFi (which I run in a Docker container). I fetch my files from AWS S3 using the ListS3 and FetchS3Object processors and then, as I ...
0
votes
0
answers
23
views
python pandas writing parquet file I just read: Conversion failed for column
import pandas as pd
file = pd.read_parquet("/Users/myuser/Downloads/input.parquet")
file.to_parquet("/Users/hazout/mysuer/output.parquet")
I am trying to read and write a parquet ...
0
votes
0
answers
22
views
Parquet. Compressing column with large plain text values
I have a table in parquet, where, amongst other columns there is a column that contains large plain text values.
For example, there is a column that contains uncompressed html webpage content. Values ...
0
votes
0
answers
26
views
How to Read parquet file using apache arrow in cpp from cloud storage S3
the aws path where parquet file is stored will look something like:
/aws/default/...
I have tried using #include <arrow/filesystem/s3fs.h> but couldnt figure out the exact methods of this ...
0
votes
0
answers
31
views
how to save/load pandas dataframe that contain non-ordered dictonary as field without knowing possible key name and value type to/from parquet?
In our application, we need to append segments of dataframe to parquet storage.
However the following code will fail at the load time, due to unmatched type
import pyarrow as pa
import pyarrow.parquet ...
1
vote
0
answers
30
views
Apache Parquet Writer throws ArrayIndexOutOfBoundsException when writing Avro
I am trying to generate some parquet files from my POJO classes. I am mostly successful in that but AvroWriter is giving me a failure when I try to save a list of custom POJO object.
try {
...
0
votes
0
answers
21
views
How can I export a 3d xarray as a parquet file in python
I have multiple 3D xarrays (and previous numpy arrays) and I was looking to export them as a parquet file to be able to quickly import them at a later date. Is it even possible to do this as by the ...
1
vote
0
answers
88
views
Python: write dataset as hive-partitioned and clustered parquet files (no JVM)
I would like write a table stored in a dataframe-like object (e.g. pandas dataframe, duckdb table, pyarrow table) in the parquet format that is both hive partitioned and clustered. Here's what I mean
...
0
votes
0
answers
43
views
Read parquet file in Jupyter Notebook - Python3
I am trying to read a .parquet file in Jupyter Notebook using Python3 but ii get errors. I tired all the solutions that you can normally find in the web, but the only ones that is actually returning a ...
0
votes
0
answers
49
views
Unable to read Parquet file with PyArrow: Malformed levels
Assume that I am unable to change how the Parquet file is written, i.e. it is immutable and so we must find a way of reading it given the following complexities...
In:
import pandas as pd
pd....
0
votes
1
answer
19
views
Save dataframe to parquet with encrypted columns
I would like to save a dataframe to parquet file. However, while writing the data to parquet file, I get the following error
df.to_parquet(path_to_data + 'data_temp.gzip', compression='GZIP')
...
-1
votes
0
answers
38
views
Is there a support to read Parquet files in Gathr?
I'm exploring Gathr for creating a data pipeline, and I'm looking to emit Parquet file data to an S3 location. Can Gathr read data from Parquet files, and which sources within Gathr support Parquet ...
0
votes
0
answers
41
views
PySpark: MutableLong cannot be cast to MutableInt
I am trying to read multiple parquet files from AWS Glue Job using PySpark and Glue API in the following:
snapshot_as_dynamic_frame = glueContext.create_dynamic_frame.from_options(
format_options={...
0
votes
1
answer
23
views
Error creating an external table in BQ off partitioned parquet files
I'm trying to create an external table in BigQuery off partitioned parquet data found in GCS. I'm doing this by running the following SQL statement:
CREATE OR REPLACE EXTERNAL TABLE `project-id.poc....
0
votes
1
answer
42
views
Unable to copy parquet data from S3 to redshift
I want to be able to copy parquet data from S3 to Redshift.
I have tried the code below:
-- Part 1: Unload
SET search_path TO my_schema_from;
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(
...
0
votes
1
answer
62
views
Unexpectedly big parquet file size in Pyspark after sorting
I have a 600Mb .CSV file with 6,5M rows and 10 columns. The columns are mostly ids and sums, but one column "type" has only 2 unique values: "online" and "offline".
When ...
0
votes
1
answer
24
views
How to Handle Growing _metadata File Size and Avoid Corruption in Amazon Redshift Spectrum Parquet Append
Context:
Our web application generates a lot of log files that arrive in an S3 bucket.
The files in the bucket contain JSON strings and have a .txt file format. We process these files in chunks of 200 ...
-1
votes
1
answer
42
views
Load parquet file to S3 from Airflow without using pandas [duplicate]
I was wondering if there was a direct way of uploading a parquet file to S3 without using pandas.
I'm using pyarrow and Airflow's S3Hook class. I have a pyarrow.Table object but I cannot find any ...
0
votes
0
answers
9
views
parquet.net ParquetOptions.UseDeltaBinaryPackedEncoding how to disable
I am using Parquet.Net library to generate parquet files from C# code. 4.15.0 version works fine, but after switching to latest 4.16.4 the files generated are not readable using ParquetViewer. It ...
0
votes
1
answer
59
views
binary format that allows to store multiple pandas dataframes with different columns, width, rows
I have like 200 pandas dataframe, and every dataframe has some unique column, or maybe completely different columns. example:
df1 = pd.DataFrame({
'Product': ['Apple', 'Banana', 'Orange', 'Mango'],...
0
votes
0
answers
22
views
The most efficient way of downloading a parquet file and saving it to local disk for use with pyarrow.memory_map
I have a large input dataset that needs some processing that can be parallelized. I've been able to scale to a fairly large number of parallel processes by first storing a huge input pyarrow table to ...
0
votes
0
answers
35
views
Can't make unit tests for reading parquet files created from protobuf messages
I wrote a parquet file from a protobuf message as such
ProtobufData model = ProtobufData.get();
Schema schema = model.getSchema(ExampleMessage.class);
Configuration conf = new Configuration();
...
0
votes
0
answers
71
views
Extracting lots of data from SQL Server into parquet file efficiently?
I have data in SQL Server and I need to save it in parquet format. Right now I'm using pyodbc and pyarrow libraries for this.
As there are serveral million rows, I'm doing it by using queries within a ...
0
votes
1
answer
53
views
What is the "Parameters" field of an AWS Glue Data Catalog table column?
The AWS documentation for Glue Data Catalog tables notes that a column has four fields:
Name
Type
Comment
Parameters
The first three are easy enough to understand; here is an excerpt from my ...
0
votes
1
answer
108
views
Reading parquet files in Databricks from Azure Data Lake results in java.lang.OutOfMemoryError: Java heap space
We have an Azure Databricks Notebook that is reading parquet data files from our "RAW" folder that has been streamed from Azure Eventhub into Azure Data Lake in Parquet format and ...
0
votes
1
answer
32
views
pyarrow write dataset drops partition columns
I am trying to use pyarrow to partition and write parquet files
!pip install pyarrow==13.0.0
import pyarrow as pa
table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
'n_legs'...
0
votes
1
answer
23
views
retro-actively add partitions to parquet files?
i have a spark job that uses Apache Hudi to write parquet into our AwS S3 data lake. I have a pretty decent sized dataset (about ~20M rows and growing) that i would like to add a new partition to. Is ...
0
votes
1
answer
41
views
Applying custom functions to parquet files loaded with open_dataset
Apologies if this has been applied somewhere else, but I haven't been able to find a solution that works.
I have some data, spread across 4232 parquet files, that aggregate into roughly 6745697 rows ...
0
votes
1
answer
62
views
Dask dataframe -containing columns that are numpy arrays- conversion to parquet error
I have a dask dataframe which I am trying to convert to parquet files. This dataframe has columns which are numpy arrays, but column type is "object".
When I try to do this:
name_function = ...
0
votes
0
answers
24
views
Error and Partition Adjustment Issue when Saving Parquet Files in PySpark
I'm encountering an issue when attempting to save parquet files using PySpark. It appears that I'm running into an error, and I suspect it may be related to the number of partitions I'm working with.
...
0
votes
1
answer
46
views
Are parquet files written incrementally via pyarrow's ParquetWriter visible to others before the writer has been closed?
I am trying to figure out whether a parquet file being written to incrementally by pyarrow's ParquetWriter would be visible to others before the close() method is called.
I've tried running some ...
0
votes
1
answer
61
views
Emoticons or Emoji issue in Azure synapse for Parquet data
I have a parquet data there are some emoticons when I opened this in any online parquet viewer it showing data with emoticons same data if query in synapse instead of emoticons it shows (?? or \uD83E\...
0
votes
1
answer
53
views
What is causing malformed level error on parquet file generated via stream analytics?
I have a service bus topic subscription and a logic app that sends the data to event hub, this data is read by stream analytics and a parquet file is generated and stored in data lake. I am then ...
1
vote
0
answers
24
views
How to handle parquet file (read&wite) in Android Studio?
I'm develloping an app with Android Studio and I need to send a parquet file to a server after gathering enough info (mostly FLOAT32 and INT4).
Does anyone know a way to write a parquet file in ...
0
votes
1
answer
60
views
Remove metadata properties from JSON ADF
I have json file in ADLS and wanted to remove metadata properties from json file and convert it to parquet using ADF pipelines or using pyspark dataframe. An example of this data is the following
[
...
1
vote
1
answer
87
views
Microsoft Fabric-Delta parquet format?
I started learning Microsoft fabric. As per Microsoft document data's are stored in Delta-Parquet format. I searched over internet I found both are different file fomats (delta format /parquet format)....
0
votes
1
answer
249
views
DuckDb - Reading parquet files from S3 - Handling Segmentation Fault Core Dump error
I receive 1000s of parquet files with same schema every day into a S3 bucket. I am using duckdb with python3 extension to read all the parquet files to subset data from them.
The following is the ...
0
votes
1
answer
75
views
How to create Athena table through parquet from pandas?
I start with a pandas Dataframe and saved to parquet format as such:
import pandas as pd
df = pd.DataFrame([
{'abc': 1.231, 'xyz':2, 'jkl': False, 'idx': 'first row', 'date': '2023-12-01'},
{'...
0
votes
1
answer
88
views
Permission error, having trouble loading parquet files from AWS S3 to SQL Server 2022
Basically every time I try to connect to AWS bucket to grab a parquet file in SQL Server 2022, I am getting a permission error
Cannot find the CREDENTIAL 's3_dc', because it does not exist or you do ...
0
votes
0
answers
29
views
Decimal conversion using AvroSchema/Apache Beam for Parquet file is failing
I am trying to convert a Parquet file, after reading using it, to Beam Rows using AvroUtils and PCollection. It is failing with ClassCastException to convert a Dcimal (38, 0) column.
org.apache.beam....
0
votes
1
answer
63
views
Reading parquet file as is using spark
I want to read parquet file as is using spark and process each file's content one by one.
I was trying to achieve it using following approach
spark.read
.option("wholetext", "...
0
votes
1
answer
54
views
Direct Parquet source (Spark Engine) vs "SQL-Select external table (pointing to that Parquet) on Serverless SQL" source (SQL Engine)
In a pipeline, in a Data Flow, I can use as a source:
the parquet file
or a sql query to an external table on the serverless sql, that points to the parquet file
Assuming that the sink is another ...