1,223
questions
0
votes
1
answer
21
views
pyarrow chunkedarray get items at given indices
Say I have
In [3]: import pyarrow as pa
In [4]: ca = pa.chunked_array([[1,2,3], [4,5,6]])
I'd like to extract elements [1, 4, 2] and end up with
<pyarrow.lib.Int64Array object at 0x7f6eb43c2d40&...
0
votes
2
answers
30
views
pyarrow chunkedarray set at indices
Say I have
In [1]: import pyarrow as ap
In [2]: import pyarrow as pa
In [3]: ca = pa.chunked_array([[1,2,3], [4,5,6]])
In [4]: ca
Out[4]:
<pyarrow.lib.ChunkedArray object at 0x7f7afaaa4a90>
[
...
-1
votes
1
answer
39
views
Optimal PyArrow/mmap pipeline for large txt to parquet
I have a large text fie (100gb) which I need to process. The file, after the first 109 rows, has a repeating pattern: a timestamp, 16 rows of data, a blank line. The timestamp refers to the next 16 ...
0
votes
1
answer
41
views
How to resolve OOM when .map concatenate the sharded parts?
When using the Dataset.map function:
dataset.map(myfunc, num_proc=16,
keep_in_memory=False,
cache_file_name='parts.arrow',
batch_size=16, writer_batch_size=16
)
Due to the size of my ...
0
votes
0
answers
33
views
How to set max_memory pool of pyarrow to just use max available on the instance?
I've a machine with 80GB RAM but whenever it does the pa.concat_tables operation, it goes out of memory.
I've tried doing the following to set the default memory pool but it's showing the ...
1
vote
2
answers
57
views
Reading / Fixing a corrupt parquet file
I have a parquet file, that was being written on by a continuous loop as follows:
def process_data(self):
#... other code ...
with pq.ParquetWriter(self.destination_file, schema) ...
0
votes
1
answer
45
views
How to create a PARTITIONED table in Python using PyIceberg with pyarrow
business_time
value
2024-06-29 05:00:03.287073252+00:00
1.3
2024-06-29 11:00:03.504073740+00:00
1.4
I am trying to create a partitioned table using the python code from the given DataFrame (see ...
0
votes
1
answer
30
views
how to effeciently write chunks to partitioned dataset?
I have multiple level of index in my data, for example
schema = pa.schema(
[
('level1', pa.dictionary(pa.int64(), pa.utf8())),
('level2', pa.binary(16)),
('level3', pa....
1
vote
2
answers
67
views
create polars array series sharing data with numpy array, with unpredictable offsets
I'm working with "records" that are represented by ~1,000 sample arrays. I have millions of these records, so they take up a fairly significant amount of memory, often exceeding what is ...
1
vote
1
answer
26
views
How to zip together two PyArrow arrays?
In Polars, I can use zip_width in order to take values from s1 or s2 according to a mask:
In [1]: import polars as pl
In [2]: import pyarrow as pa
In [3]: import pyarrow as pc
In [4]: s1 = pl....
0
votes
0
answers
48
views
pandas_udf causing Python worker to crash in PySpark on macOS with M3 chip
I'm working with PySpark and trying to use a pandas_udf on my macOS system with an M3 chip. My environment is Python 3.10 running from a virtual environment. The code runs fine until I introduce the ...
1
vote
0
answers
51
views
PyArrow Table manipulation: Unnest float-array column to individual columns
I have nested data stored in parquet files.
Polars was my main entrypoint for fast data formatting of this nested data, but for performance reasons, I'd like to use native arrow, using the PyArrow ...
2
votes
1
answer
41
views
Force PyArrow table write to ignore NULL type and use original schema type for a column
I have this piece of code that appends two parts of the same data to a PyArrow table. The second write fails because the column gets assigned null type. I understand why it is doing that. Is there a ...
0
votes
1
answer
30
views
Is there a way to validate Python objects against a pyarrow schema?
I have a pyarrow Schema defined and a list of native Python dictionaries. I can use
pyarrow.Table.from_pylist(list_of_python_objects, schema=SCHEMA)
and that will create a table matching the schema. ...
1
vote
0
answers
64
views
Non-nullable field in schema does not prevent null values in corresponding table column
I have read the documentation of the field() method, and as far as I understand I can say that a field does not allow NULL values by specifying nullable=False. So, I have tried this example:
schema = ...
0
votes
0
answers
103
views
Pandas parquet file pyarrow.lib.ArrowMemoryError: malloc of size 106255424 failed
I am trying to run a python script in cPanel terminal. I am getting an error as the script tries to open the parquet file that is a 46.65 MB size. This worked on my home computer.
df = pd.read_parquet(...
0
votes
0
answers
19
views
Inferring schema from a pyarrow dataframe created with parquet file is giving wrong data type for a column
I have a parquet file which is loaded into pandas dataframe as below.
df = pq.read_table("full_file.parquet")
and If I check the column types in the schema, as below, for some columns it is ...
0
votes
0
answers
28
views
How does pyarrow handles date partitions?
I store files on s3 in the following format:
../country/state/city/date=2024-08-02/12-00-57/time_series.parquet
and the table contains multiple columns on of which is named date of type pa.date64(). ...
1
vote
0
answers
66
views
Python arrow_odbc issue with mysql odbc drivers libmyodbc9
I am losing my mind around this issue here. The problem started while I was using Polars to connect to both SqlServer and MySql databases through odbc drivers, which Polars handles making use of the ...
1
vote
1
answer
1k
views
AttributeError: module 'pyarrow.lib' has no attribute 'ListViewType'
------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[67], line 2
1 import pandas as pd
----> 2 ...
0
votes
1
answer
37
views
How to automatically get table create statement for Redshift serverless from Pandas dataframe
I have an S3 bucket which contains parquet files.
I need to analyse that parquet file and create the required table in Redshift serverless.
import pyarrow.parquet as pq
df = pq.read_table(f"s3://{...
0
votes
1
answer
38
views
Convert PyArrow table to pandas, but transform string to `string[pyarrow]` in pandas directly
Say I have
import pyarrow as pa
tbl = pa.table({'a': ['foo', 'bar']})
If I do tbl.to_pandas(), then the dtypes of the result are object:
In [14]: tbl.to_pandas().dtypes
Out[14]:
a object
dtype: ...
0
votes
1
answer
35
views
Shift a chunkedarray
If I start with
import pyarrow as pa
ca = pa.chunked_array([[1,3, 2], [5, 2, 1]])
I'd like to end up with a pyarrow array with elements
[None, 1, 3, 2, 5, 2]
I don't really mind if the output is an ...
0
votes
1
answer
35
views
Open with Python an R data.table saved as metadata in a Parquet file
With R, I created a Parquet file containing a data.table as main data, and another data.table as metadata.
library(data.table)
library(arrow)
dt = data.table(x = c(1, 2, 3), y = c("a", "...
0
votes
0
answers
64
views
Pyarrow error with ray & lightning on databricks
I am trying to train a neural net with pytorch lightning on ray on a databricks cluster. As a start, I copied the example from https://docs.ray.io/en/latest/train/getting-started-pytorch-lightning....
2
votes
1
answer
75
views
pyarrow ipc stream - how to communicate using it?
After reading pyarrow IPC docs, I had an impression that RecordBatchStreamReader will read stream until it is done (0-s in the end, that are written by closing stream writer).
But I see that reader ...
0
votes
2
answers
33
views
Convert hexadecimal StringArray to FixedSizeBinaryArray
I have an array of ethereum addresses in hexadecimal format (20 bytes), encoded as an hexadecimal string. How can I transform that into a FixedSizeBinaryArray?
I tried using cast:
>>> import ...
1
vote
1
answer
31
views
Read CSV with epoch timestamp column as timestamp
I'm using pyarrow.csv to read and convert a CSV file to parquet. This CSV file has a timestamp column with an int representing Unix time.
Nevertheless, it reads it as an int64, and if I try to use ...
1
vote
3
answers
93
views
Dummy-encoded PyArrow table from PyArrow ChunkedArray, without going through pandas?
Say I have
import pyarrow as pa
ca = pa.chunked_array([['a', 'b', 'b', 'c']])
print(ca)
<pyarrow.lib.ChunkedArray object at 0x7fc938bcea70>
[
[
"a",
"b",
&...
0
votes
2
answers
65
views
How to append string to each element of chunked array?
Say I have
In [22]: import pyarrow as pa
In [23]: t = pa.table({'a': ['one', 'two', 'three']})
and I'd like to append '_frobenius' to each element of 'a'
Expected output:
pyarrow.Table
a: string
----...
0
votes
0
answers
37
views
How to append time-series data with PyArrow Datasets?
Problem
I'm looking to store time-series data that's being aggregated live to Parquet Datasets via PyArrow. I receive live batched data, for example, video view count each hour for the last 24 hours. ...
0
votes
1
answer
67
views
Size of pyarrow Table in bytes
I have a basic pyarrow.Table. What's the best way to get it's size in bytes?
Here is an example table:
import pyarrow as pa
n_legs = pa.array([2, 4, 5, 100])
animals = pa.array(["Flamingo", ...
0
votes
0
answers
363
views
ArrowInvalid: offset overflow while concatenating arrays when subsetting a Pandas Dataframe
When reading parquet files into a Pandas Dataframe, I am doing so using the following
df = pd.read_parquet(PATH_TO_FILE, dtype_backend='pyarrow')
In this case, my dataframe is quite large, 52 million ...
0
votes
1
answer
31
views
pyarrow: find diff for chunkedarray
If I have a chunkedarray, how do I find its diff (similar to pandas.Series.diff or polars.Series.diff)?
e.g. if I start with
import pyarrow as pa
ca = pa.chunked_array([[1,3, 2], [5, 2, 1]])
I'd like ...
0
votes
1
answer
89
views
Read parquet file using pandas and pyarrow fails for time values larger than 24 hours
I have exported a parquet file using parquet.net which includes a duration column that contains values that are greater than 24 hours. I've opened the tool using the floor tool that's included with ...
0
votes
0
answers
39
views
What determines that a given Python type is coercible into a given pyarrow datatype?
For example:
I know a dictionary can be coerced into a struct; what else can be coerced into a struct? What determines whether a Python type is coercible into a pyarrow datatype? Convention? ...
-1
votes
1
answer
41
views
Ways of creating a `pyarrow.StructScalar` directly?
I know I can create a pa.StructScalar by casting:
import pyarrow as pa
import pyarrow.compute as pac
struct_scalar = pac.cast(
{"hello": "greetings", "world": 5},
...
0
votes
0
answers
36
views
Infer `pyarrow.DataType` from Python type?
Consider the following example:
from typing import Any, get_type_hints
import pyarrow as pa
import pyarrow.compute as pac
ArrowTypes = {
int: pa.int64(),
str: pa.string(),
}
class StructLike:...
0
votes
1
answer
40
views
Get categories of arrow chunkedarray
If I have a PyArrow chunkedarray and want to know all its categories, I can go through each individual array, get the categories from there, and find the union:
import pyarrow as pa
import functools
#...
0
votes
1
answer
75
views
Efficiently storing data that is not yet purely-columnar into the Arrow format
I previously asked a question similar to this one (in that it uses the same example to illustrate the problem): Efficiently storing data that is not-quite-columnar-yet into a DuckDB database.
However, ...
1
vote
0
answers
70
views
how can I create a arrow flight server that receives more than one dataset with different schemas?
I want to send a geopandas.GeoDataFrame (a map) and a pandas.DataFrame (a coordinates list) over the network using arrow flight in python. So I would transform them to arrow data objects and send them ...
0
votes
1
answer
60
views
reading csv file with header and tail into apache arrow
I have a issue to read a csv flat file into apache arrow. The file has a header and a tail. The header take just one line, so it is easy dealt with, just use the skip or skip_rows arguments of the ...
0
votes
1
answer
260
views
Estimating the size of data when loaded from parquet file into an arrow table
I have a pyarrow table with a large amount of columns (>2000). For 1000 rows, it takes about 20M RAM. For many columns, there’s a single value over all the rows. When I save it to a parquet file, ...
1
vote
1
answer
130
views
How to get the values of a dictionary type from a parquet file using pyarrow?
I have a parquet file which I am reading with pyarrow.
In [83]: pq.read_schema('dummy_file.parquet').field('dummy_column').type
Out[83]: DictionaryType(dictionary<values=string, indices=int32, ...
0
votes
1
answer
124
views
Using Pyarrow, how do you group by a value, then select the top K largest values in that group?
I'm looking group by a PyArrow column, then within each of those groups select the top K values without using Pandas. For example, given this dataset:
id
score
1
0
1
1
1
2
1
3
1
4
2
5
2
6
2
7
...
0
votes
1
answer
29
views
how can I merge multiple columns into array with datasets.Dataset.from_csv()?
I have a CSV file, it has N columns, if I do datasets.Dataset.from_csv(path), it will be N feature each of them is a int value.
However, I want to say: column-0 to column-4 is feature1, the rest is ...
0
votes
0
answers
145
views
FastApi Streaming Response with Partitioned Parquet File
I am trying to create a FastAPI GET endpoint that serves up a parquet octet stream to the client (assumed to be a python client library that turns it into a pandas dataframe).
The parquet file is ...
1
vote
1
answer
215
views
Poetry failing to install Datasets and Transformers in Docker
I am trying to install Datasets and Transformers libraries to my project which is using Poetry environment. Everything works fine if I try it locally but if I try it run in docker it starts failing. ...
0
votes
1
answer
85
views
How can I extract data from parquet files using pyarrow?
I'm trying to perform data analysis on a large number of quite large parquet files. The analysis itself is relatively simple, but using e.g. pandas requires nested for loops to slice the data into ...
1
vote
0
answers
77
views
PyInstaller Issues with PyArrow
TL;DR:
My single file Windows app dependent on PyArrow and built with PyInstaller fails to run with a DLL load failure while attempting to import pyarrow.lib.
Version/System Info:
Python 3.11.9
...