Skip to main content
Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

pyarrow chunkedarray get items at given indices

Say I have In [3]: import pyarrow as pa In [4]: ca = pa.chunked_array([[1,2,3], [4,5,6]]) I'd like to extract elements [1, 4, 2] and end up with <pyarrow.lib.Int64Array object at 0x7f6eb43c2d40&...
ignoring_gravity's user avatar
0 votes
2 answers
30 views

pyarrow chunkedarray set at indices

Say I have In [1]: import pyarrow as ap In [2]: import pyarrow as pa In [3]: ca = pa.chunked_array([[1,2,3], [4,5,6]]) In [4]: ca Out[4]: <pyarrow.lib.ChunkedArray object at 0x7f7afaaa4a90> [ ...
ignoring_gravity's user avatar
-1 votes
1 answer
39 views

Optimal PyArrow/mmap pipeline for large txt to parquet

I have a large text fie (100gb) which I need to process. The file, after the first 109 rows, has a repeating pattern: a timestamp, 16 rows of data, a blank line. The timestamp refers to the next 16 ...
tiarap00's user avatar
0 votes
1 answer
41 views

How to resolve OOM when .map concatenate the sharded parts?

When using the Dataset.map function: dataset.map(myfunc, num_proc=16, keep_in_memory=False, cache_file_name='parts.arrow', batch_size=16, writer_batch_size=16 ) Due to the size of my ...
alvas's user avatar
  • 121k
0 votes
0 answers
33 views

How to set max_memory pool of pyarrow to just use max available on the instance?

I've a machine with 80GB RAM but whenever it does the pa.concat_tables operation, it goes out of memory. I've tried doing the following to set the default memory pool but it's showing the ...
alvas's user avatar
  • 121k
1 vote
2 answers
57 views

Reading / Fixing a corrupt parquet file

I have a parquet file, that was being written on by a continuous loop as follows: def process_data(self): #... other code ... with pq.ParquetWriter(self.destination_file, schema) ...
dev-mirzabicer's user avatar
0 votes
1 answer
45 views

How to create a PARTITIONED table in Python using PyIceberg with pyarrow

business_time value 2024-06-29 05:00:03.287073252+00:00 1.3 2024-06-29 11:00:03.504073740+00:00 1.4 I am trying to create a partitioned table using the python code from the given DataFrame (see ...
user23873134's user avatar
0 votes
1 answer
30 views

how to effeciently write chunks to partitioned dataset?

I have multiple level of index in my data, for example schema = pa.schema( [ ('level1', pa.dictionary(pa.int64(), pa.utf8())), ('level2', pa.binary(16)), ('level3', pa....
Wang's user avatar
  • 8,034
1 vote
2 answers
67 views

create polars array series sharing data with numpy array, with unpredictable offsets

I'm working with "records" that are represented by ~1,000 sample arrays. I have millions of these records, so they take up a fairly significant amount of memory, often exceeding what is ...
gggg's user avatar
  • 2,376
1 vote
1 answer
26 views

How to zip together two PyArrow arrays?

In Polars, I can use zip_width in order to take values from s1 or s2 according to a mask: In [1]: import polars as pl In [2]: import pyarrow as pa In [3]: import pyarrow as pc In [4]: s1 = pl....
ignoring_gravity's user avatar
0 votes
0 answers
48 views

pandas_udf causing Python worker to crash in PySpark on macOS with M3 chip

I'm working with PySpark and trying to use a pandas_udf on my macOS system with an M3 chip. My environment is Python 3.10 running from a virtual environment. The code runs fine until I introduce the ...
Md. Moniruzzaman's user avatar
1 vote
0 answers
51 views

PyArrow Table manipulation: Unnest float-array column to individual columns

I have nested data stored in parquet files. Polars was my main entrypoint for fast data formatting of this nested data, but for performance reasons, I'd like to use native arrow, using the PyArrow ...
Durand's user avatar
  • 89
2 votes
1 answer
41 views

Force PyArrow table write to ignore NULL type and use original schema type for a column

I have this piece of code that appends two parts of the same data to a PyArrow table. The second write fails because the column gets assigned null type. I understand why it is doing that. Is there a ...
Gopala's user avatar
  • 10.4k
0 votes
1 answer
30 views

Is there a way to validate Python objects against a pyarrow schema?

I have a pyarrow Schema defined and a list of native Python dictionaries. I can use pyarrow.Table.from_pylist(list_of_python_objects, schema=SCHEMA) and that will create a table matching the schema. ...
ldrg's user avatar
  • 4,412
1 vote
0 answers
64 views

Non-nullable field in schema does not prevent null values in corresponding table column

I have read the documentation of the field() method, and as far as I understand I can say that a field does not allow NULL values by specifying nullable=False. So, I have tried this example: schema = ...
MarcoS's user avatar
  • 13.5k
0 votes
0 answers
103 views

Pandas parquet file pyarrow.lib.ArrowMemoryError: malloc of size 106255424 failed

I am trying to run a python script in cPanel terminal. I am getting an error as the script tries to open the parquet file that is a 46.65 MB size. This worked on my home computer. df = pd.read_parquet(...
Shane S's user avatar
  • 2,171
0 votes
0 answers
19 views

Inferring schema from a pyarrow dataframe created with parquet file is giving wrong data type for a column

I have a parquet file which is loaded into pandas dataframe as below. df = pq.read_table("full_file.parquet") and If I check the column types in the schema, as below, for some columns it is ...
Poreddy Siva Sukumar Reddy US's user avatar
0 votes
0 answers
28 views

How does pyarrow handles date partitions?

I store files on s3 in the following format: ../country/state/city/date=2024-08-02/12-00-57/time_series.parquet and the table contains multiple columns on of which is named date of type pa.date64(). ...
zipp's user avatar
  • 23
1 vote
0 answers
66 views

Python arrow_odbc issue with mysql odbc drivers libmyodbc9

I am losing my mind around this issue here. The problem started while I was using Polars to connect to both SqlServer and MySql databases through odbc drivers, which Polars handles making use of the ...
Lorenzo Sciarra's user avatar
1 vote
1 answer
1k views

AttributeError: module 'pyarrow.lib' has no attribute 'ListViewType'

------------------------------------------------------------ AttributeError Traceback (most recent call last) Cell In[67], line 2 1 import pandas as pd ----> 2 ...
Percival Shimange's user avatar
0 votes
1 answer
37 views

How to automatically get table create statement for Redshift serverless from Pandas dataframe

I have an S3 bucket which contains parquet files. I need to analyse that parquet file and create the required table in Redshift serverless. import pyarrow.parquet as pq df = pq.read_table(f"s3://{...
Poreddy Siva Sukumar Reddy US's user avatar
0 votes
1 answer
38 views

Convert PyArrow table to pandas, but transform string to `string[pyarrow]` in pandas directly

Say I have import pyarrow as pa tbl = pa.table({'a': ['foo', 'bar']}) If I do tbl.to_pandas(), then the dtypes of the result are object: In [14]: tbl.to_pandas().dtypes Out[14]: a object dtype: ...
ignoring_gravity's user avatar
0 votes
1 answer
35 views

Shift a chunkedarray

If I start with import pyarrow as pa ca = pa.chunked_array([[1,3, 2], [5, 2, 1]]) I'd like to end up with a pyarrow array with elements [None, 1, 3, 2, 5, 2] I don't really mind if the output is an ...
ignoring_gravity's user avatar
0 votes
1 answer
35 views

Open with Python an R data.table saved as metadata in a Parquet file

With R, I created a Parquet file containing a data.table as main data, and another data.table as metadata. library(data.table) library(arrow) dt = data.table(x = c(1, 2, 3), y = c("a", "...
julien.leroux5's user avatar
0 votes
0 answers
64 views

Pyarrow error with ray & lightning on databricks

I am trying to train a neural net with pytorch lightning on ray on a databricks cluster. As a start, I copied the example from https://docs.ray.io/en/latest/train/getting-started-pytorch-lightning....
DataDiver's user avatar
2 votes
1 answer
75 views

pyarrow ipc stream - how to communicate using it?

After reading pyarrow IPC docs, I had an impression that RecordBatchStreamReader will read stream until it is done (0-s in the end, that are written by closing stream writer). But I see that reader ...
OldAG's user avatar
  • 65
0 votes
2 answers
33 views

Convert hexadecimal StringArray to FixedSizeBinaryArray

I have an array of ethereum addresses in hexadecimal format (20 bytes), encoded as an hexadecimal string. How can I transform that into a FixedSizeBinaryArray? I tried using cast: >>> import ...
David Davó's user avatar
1 vote
1 answer
31 views

Read CSV with epoch timestamp column as timestamp

I'm using pyarrow.csv to read and convert a CSV file to parquet. This CSV file has a timestamp column with an int representing Unix time. Nevertheless, it reads it as an int64, and if I try to use ...
David Davó's user avatar
1 vote
3 answers
93 views

Dummy-encoded PyArrow table from PyArrow ChunkedArray, without going through pandas?

Say I have import pyarrow as pa ca = pa.chunked_array([['a', 'b', 'b', 'c']]) print(ca) <pyarrow.lib.ChunkedArray object at 0x7fc938bcea70> [ [ "a", "b", &...
ignoring_gravity's user avatar
0 votes
2 answers
65 views

How to append string to each element of chunked array?

Say I have In [22]: import pyarrow as pa In [23]: t = pa.table({'a': ['one', 'two', 'three']}) and I'd like to append '_frobenius' to each element of 'a' Expected output: pyarrow.Table a: string ----...
ignoring_gravity's user avatar
0 votes
0 answers
37 views

How to append time-series data with PyArrow Datasets?

Problem I'm looking to store time-series data that's being aggregated live to Parquet Datasets via PyArrow. I receive live batched data, for example, video view count each hour for the last 24 hours. ...
humanlikely's user avatar
0 votes
1 answer
67 views

Size of pyarrow Table in bytes

I have a basic pyarrow.Table. What's the best way to get it's size in bytes? Here is an example table: import pyarrow as pa n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", ...
jjbskir's user avatar
  • 9,885
0 votes
0 answers
363 views

ArrowInvalid: offset overflow while concatenating arrays when subsetting a Pandas Dataframe

When reading parquet files into a Pandas Dataframe, I am doing so using the following df = pd.read_parquet(PATH_TO_FILE, dtype_backend='pyarrow') In this case, my dataframe is quite large, 52 million ...
Vincent's user avatar
  • 8,586
0 votes
1 answer
31 views

pyarrow: find diff for chunkedarray

If I have a chunkedarray, how do I find its diff (similar to pandas.Series.diff or polars.Series.diff)? e.g. if I start with import pyarrow as pa ca = pa.chunked_array([[1,3, 2], [5, 2, 1]]) I'd like ...
ignoring_gravity's user avatar
0 votes
1 answer
89 views

Read parquet file using pandas and pyarrow fails for time values larger than 24 hours

I have exported a parquet file using parquet.net which includes a duration column that contains values that are greater than 24 hours. I've opened the tool using the floor tool that's included with ...
Wouter's user avatar
  • 2,262
0 votes
0 answers
39 views

What determines that a given Python type is coercible into a given pyarrow datatype?

For example: I know a dictionary can be coerced into a struct; what else can be coerced into a struct? What determines whether a Python type is coercible into a pyarrow datatype? Convention? ...
bzm3r's user avatar
  • 4,228
-1 votes
1 answer
41 views

Ways of creating a `pyarrow.StructScalar` directly?

I know I can create a pa.StructScalar by casting: import pyarrow as pa import pyarrow.compute as pac struct_scalar = pac.cast( {"hello": "greetings", "world": 5}, ...
bzm3r's user avatar
  • 4,228
0 votes
0 answers
36 views

Infer `pyarrow.DataType` from Python type?

Consider the following example: from typing import Any, get_type_hints import pyarrow as pa import pyarrow.compute as pac ArrowTypes = { int: pa.int64(), str: pa.string(), } class StructLike:...
bzm3r's user avatar
  • 4,228
0 votes
1 answer
40 views

Get categories of arrow chunkedarray

If I have a PyArrow chunkedarray and want to know all its categories, I can go through each individual array, get the categories from there, and find the union: import pyarrow as pa import functools #...
ignoring_gravity's user avatar
0 votes
1 answer
75 views

Efficiently storing data that is not yet purely-columnar into the Arrow format

I previously asked a question similar to this one (in that it uses the same example to illustrate the problem): Efficiently storing data that is not-quite-columnar-yet into a DuckDB database. However, ...
bzm3r's user avatar
  • 4,228
1 vote
0 answers
70 views

how can I create a arrow flight server that receives more than one dataset with different schemas?

I want to send a geopandas.GeoDataFrame (a map) and a pandas.DataFrame (a coordinates list) over the network using arrow flight in python. So I would transform them to arrow data objects and send them ...
Daí's user avatar
  • 64
0 votes
1 answer
60 views

reading csv file with header and tail into apache arrow

I have a issue to read a csv flat file into apache arrow. The file has a header and a tail. The header take just one line, so it is easy dealt with, just use the skip or skip_rows arguments of the ...
Leonardo Miceli's user avatar
0 votes
1 answer
260 views

Estimating the size of data when loaded from parquet file into an arrow table

I have a pyarrow table with a large amount of columns (>2000). For 1000 rows, it takes about 20M RAM. For many columns, there’s a single value over all the rows. When I save it to a parquet file, ...
urim's user avatar
  • 591
1 vote
1 answer
130 views

How to get the values of a dictionary type from a parquet file using pyarrow?

I have a parquet file which I am reading with pyarrow. In [83]: pq.read_schema('dummy_file.parquet').field('dummy_column').type Out[83]: DictionaryType(dictionary<values=string, indices=int32, ...
In78's user avatar
  • 462
0 votes
1 answer
124 views

Using Pyarrow, how do you group by a value, then select the top K largest values in that group?

I'm looking group by a PyArrow column, then within each of those groups select the top K values without using Pandas. For example, given this dataset: id score 1 0 1 1 1 2 1 3 1 4 2 5 2 6 2 7 ...
Mason Kirchner's user avatar
0 votes
1 answer
29 views

how can I merge multiple columns into array with datasets.Dataset.from_csv()?

I have a CSV file, it has N columns, if I do datasets.Dataset.from_csv(path), it will be N feature each of them is a int value. However, I want to say: column-0 to column-4 is feature1, the rest is ...
Wang's user avatar
  • 8,034
0 votes
0 answers
145 views

FastApi Streaming Response with Partitioned Parquet File

I am trying to create a FastAPI GET endpoint that serves up a parquet octet stream to the client (assumed to be a python client library that turns it into a pandas dataframe). The parquet file is ...
Sebastian Janisch's user avatar
1 vote
1 answer
215 views

Poetry failing to install Datasets and Transformers in Docker

I am trying to install Datasets and Transformers libraries to my project which is using Poetry environment. Everything works fine if I try it locally but if I try it run in docker it starts failing. ...
user22386756's user avatar
0 votes
1 answer
85 views

How can I extract data from parquet files using pyarrow?

I'm trying to perform data analysis on a large number of quite large parquet files. The analysis itself is relatively simple, but using e.g. pandas requires nested for loops to slice the data into ...
Sal's user avatar
  • 37
1 vote
0 answers
77 views

PyInstaller Issues with PyArrow

TL;DR: My single file Windows app dependent on PyArrow and built with PyInstaller fails to run with a DLL load failure while attempting to import pyarrow.lib. Version/System Info: Python 3.11.9 ...
nguzman's user avatar
  • 183

1
2 3 4 5
25