Questions tagged [pyarrow]
pyarrow is a Python interface for Apache Arrow
1,088
questions
0
votes
0
answers
12
views
PyArrow - Working with larger than memory Arrow IPC (feather) files?
I'm trying to work Arrow IPC (feather) files which are larger than memory. PyArrow's documentation notes that "Arrow IPC files can be memory-mapped locally, which allow you to work with data ...
1
vote
1
answer
44
views
Writing DataFrames as partitioned parquet object in Polars with PyArrow
I have 12 parquet files in a directory with matching columns I am trying to write to a partitioned object with Polars and PyArrow. I am iterating through each file in the directory and reading it in ...
1
vote
1
answer
26
views
Using pyarrow read_json() to parse a list of structs into a StructArray
I can successfully use pyarrow read_json to read some JSON financial trading data into a table. The JSON contains two lists of structs, asks and bids.
schema = pa.schema([('time', pa.timestamp(unit='...
0
votes
1
answer
37
views
How to iterate and read one row at at time from multiple parquet files?
I know we can use pyarrow to read batches of rows from a given parquet file
import pyarrow.parquet as pq
parquet_file = pq.ParquetFile('/mybigparquet.pq')
for i in parquet_file.iter_batches(...
0
votes
1
answer
28
views
use gensim with a pyarrow iterable
Consider this code
import pyarrow.parquet as pq
from gensim.models import Word2Vec
parquet_file = pq.ParquetFile('/mybigparquet.pq')
for i in parquet_file.iter_batches(batch_size=100):
print(&...
2
votes
0
answers
41
views
pyarrow breaks pyodbc MySQL?
I have a Docker container with MySQL ODBC driver, unixODBC, and a bunch of Python stuff installed. My MySQL driver works through isql, and it works when connecting from Python with pyodbc, if I do so ...
-1
votes
0
answers
38
views
PyArrow Table to Spark DataFrame
import pyarrow.parquet as pq
table = pq.read_table(parquet_blob_data)
Now how to create a spark dataframe from table without using pandas library or to_pandas() in-built method?
I found only ...
0
votes
1
answer
27
views
Error ModuleNotFoundError: No module named 'cmake' when installing pyarrow [duplicate]
I'm trying to install streamlit, which requires pyarrow. But when I install pyarrow, I get the errow "ModuleNotFoundError: No module named 'cmake'". I have already installed cmake (version 3....
2
votes
1
answer
76
views
Python Polars: Low memory read, process, writing of parquet to/from Hadoop
I would like to be able to process very large files in Polars without running out of memory. In the documentation they suggest using scanning, lazyframes and sinks, but it is hard to find proper ...
0
votes
0
answers
27
views
Replace all inf, -inf values with NaN in a pandas dataframe if dtype backend is arrow instead of numpy
I was wondering if with the changes of pandas going forward from 2.0 to 3.0, the solution of using replace still applies if we don't know how +-Inf values are stored, i.e. their types. What is a ...
1
vote
3
answers
55
views
How to write and read dataframe to parquet where column contains list of dicts
I have a column that contain a list of dictionaries and I'm trying to write it to disk using parquet and reading it back into the same original object. However I'm not able to get the same exact ...
0
votes
1
answer
31
views
How to perform a conditional sum in pyarrow
I'm trying to map a SQL query over to pyarrow and it's my first time tinkering with pyarrow. I can't find a compute function that seems to fit the bill.
select
group_col_1,
group_col_2,
sum(
...
0
votes
0
answers
18
views
Unable to set up HDFS interface with PyArrow
I am trying to access the HDFS directory using pyarrow as follows.
Install Hadoop and Spark
Locate libhdfs.so
/usr/local/hadoop-3.3.5/lib/native/libhdfs.so
Create Home Directories
import os
os....
1
vote
0
answers
26
views
Specifying logical types (in particular, UUID) when writing parquet files from pyarrow
The pyarrow documentation builds a custom UUID type many times like this:
import pyarrow as pa
class UuidType(pa.PyExtensionType):
def __init__(self):
pa.PyExtensionType.__init__(self, pa....
0
votes
0
answers
58
views
pandas.convert_dtype(dtype_backend="pyarrow") raises error
I'm starting to use pandas with Pyarrow and realized there's no easy way to declare a df directly as with Pyarrow engine, so I ended up in this post. The thing is that I'm having the same isssue as ...
0
votes
1
answer
32
views
Why does the sum function not work with pyarrow fields?
Consider the following list of fields.
import pyarrow.dataset as ds
fields = [ds.field('a'), ds.field('b'), ds.field('c')]
sum(fields) produces the following error:
TypeError: Argument 'self' has ...
0
votes
1
answer
117
views
Construct filtering expression based on lexicographical ordering
I have a dataset with integer columns for 'year', 'month' and 'day'.
For a given start_date and end_date (pandas Timestamps), I want to build an expression to filter for all the rows between these ...
0
votes
0
answers
17
views
How to convert R Arrow expression to pyarrow equivalent. Larger than memory dataset
I am trying to run a simple query on full NYC taxi dataset (my local copy on HDD), which counts number of rows per group, i.e group by X then count (*)
In R-arrow, this can be done using
nyc_taxi = ...
2
votes
1
answer
72
views
How to apply ip lookup using polars?
Given two tables I'd like to conduct a lookup over all ips and find the network it belongs to:
I have two large tables:
and the following networks:
Regarding the ClientIP (First table) I thought of ...
1
vote
0
answers
25
views
Loading int16 data with NULLs from snowflake into a pandas DataFrame
I am looking for an automated way to fetch data that doesn't fit into numpy datatypes (e.g. nullable ints) from a snowflake db (using the snowflake connector).
E.g. I have a query that loads a column ...
0
votes
1
answer
28
views
Create a dask array with pyarrow dtype
In pandas, I can create a series with a pyarrow dtype the following way:
>>> import pandas as pd
>>> s = pd.Series([1,2,3]).astype("int64[pyarrow]")
>>> s.dtype
...
0
votes
1
answer
53
views
Unable to install pyarrow - ERROR: Failed building wheel for pyarrow
I am using python v 3.8 , numpy-1.24.4 pandas-2.0.3, cmake 3.27.3 on Windows 10 - 64 bit
However ,when I try to install pyarrow using
pip install pyarrow
I get the error below File "<string&...
0
votes
0
answers
71
views
Disable ssl validation while connecting to s3 using pyarrow fs library/ s3fs library
I am using pyarrow fs.S3FileSystem library to write a csv to s3 bucket. Although this code runs fine in my local when I deploy to VM (linux) it throws error:
OSError: When listing objects under key xx ...
0
votes
1
answer
37
views
How does Copy-on-Write (CoW) work when using the new pandas pyarrow backend?
I think the question says it all — the pandas docs have good information on both the new CoW behavior and the pyarrow backend, but was wondering if CoW works the same with pyarrow as it does with a ...
1
vote
1
answer
45
views
To_parquet giving error: __arrow_array__() got an unexpected keyword argument 'type'
I'm reading a root file using uproot and converting parts of it into a DataFrame using the arrays method.
This works fine, until I try to save to parquet using the to_parquet method on the dataframe. ...
0
votes
0
answers
193
views
pyarrow.lib.ArrowInvalid: ("Could not convert 'col_x' with type str: tried to convert to double", 'Conversion failed for column with type object')
While enforcing pyarrrow schema on a pandas dataframe (to create a parquet file of the same), there is a number(24,3) datatype column coming from Oracle which is not being typecasted and it is ...
2
votes
1
answer
48
views
Read/write partitioned parquet from/to SFTP server with pyarrow
recently I got myself into data analysis with some friends and to improve our data exchange we got a linux server which we use as a SFTP server. Following this we no longer want to write outputs to ...
0
votes
0
answers
98
views
CMake and PyArrow Errors While Installing Streamlit
During the installation of Streamlit using the 'pip install streamlit' command, I encountered errors related to both CMake and PyArrow. The process failed with the following error messages:
-- ...
3
votes
0
answers
88
views
pyarrow memory consumption difference between Dataset.to_batches and ParquetFile.iter_batches
I am using pyarrow and am struggling to understand the big difference in memory usage between the Dataset.to_batches method compared to ParquetFile.iter_batches.
Using pyarrow.dataset
>>> ...
0
votes
1
answer
73
views
Apache Arrow IPC streams: SPMC concurrency
Is it expected that a multiple IPC stream readers can concurrently tail a single stream writer publishing from another process? The descriptions of things like "IPC streams" leads me to ...
0
votes
0
answers
37
views
Is there a way to apply custom filters when reading a row group from a parquet file with pyarrow?
I'm trying to figure out how Modin could implement the ability to read Parquet files using filters, with the same behavior as pandas.read_parquet('my_parquet_file.parquet', filters=[('col', '==', 'val'...
0
votes
1
answer
36
views
read_parquet and merge schema
I have a lot of parquet files which are partitioned by YEAR/MONTH/DAY, which I read in with read_parquet(PATH, engine="pyarrow").
All parquets have the columns "column_1", column_2&...
0
votes
0
answers
159
views
FIFO Queue with Polars DataFrame
I'm currently working on a use case that involves the usage of sensors and IoT data in the form of a (multivariate) time series. The final goal is to produce an ML model for fault/anomaly detection.
...
1
vote
0
answers
24
views
Error while loading parquet format on the basis of date or size or required size format from s3
I am trying to load parquet file format data from s3 in a filtered fashion, by loading few MB or load it on the basis of date
code:
obj = s3_client.get_object(Bucket=bucket_name, Key=file_path1,
...
1
vote
1
answer
540
views
What is actually meant when referring to parquet row-group size?
I am starting to work with the parquet file format.
The official Apache site recommends large row groups of 512MB to 1GB (here).
Several online source (e.g. this one) suggest that the default row ...
0
votes
0
answers
213
views
Using scan_parquet and scan_pyarrow_dataset on S3 loads entire dataset into memory
I have a large, partitioned, parquet dataset that does not fit into memory and is stored in an S3 location. A piece of of processing code is meant to process a subset of this data by performing a join ...
0
votes
2
answers
158
views
How to send arrow data from fastapi to the js apache arrow package without copying?
Let's say generically my setup is like this:
from fastapi import FastAPI, Response
import pyarrow as pa
import pyarrow.ipc as ipc
app = FastAPI()
@app.get("/api/getdata")
async def getdata()...
0
votes
0
answers
41
views
Snowflake cursor fetch_arrow_all returns 1816 date
I have a use case where I need to fetch snowflake response as pyarrow table using fetch_arrow_all. The date which are like 9999-10-10 getting defaulted to 1815/1816.
cursor.execute("select ...
0
votes
0
answers
103
views
Pyarrow writing parquet dataset behavior after updating pyarrow
I'm working on a codebase that processes daily data. We have a Pyarrow Table with 3 columns, one of which is a date. The data goes back to 2007, so ~ 5800 unique dates. The table has 43 million rows, ...
0
votes
0
answers
138
views
Memory consumption in PyArrow iter_batches
Let's suppose I have n parquet files that represent the columns of my final parquet file. The columns have 5 milion rows and I want to read my entire file in batches and each batch is given by the ...
2
votes
0
answers
165
views
Cannot install pyarrow
I am having issues with installing pyarrow and any package that relies on it (e.g., pandas-gbq). I tried various combinations of modules (including different cmake versions), but always get an error ...
0
votes
1
answer
28
views
got an length eroor when using read_parquet function
When i use read_parquet method to read parquet file, it occurs Column 8 named hostIp expected length 548 but got length 549 error, hostIP is one column in REQUIRED_COLUMNS.
import pandas as pd
...
0
votes
1
answer
34
views
Parquet / pyarrow: Malformed levels
I use Azure Stream Analytics which converts some json documents in parquet files.
For most of them, I can read them after but for some of them I get the following error:
Traceback (most recent call ...
-1
votes
1
answer
247
views
Having issue installing 'Streamlit' with pip, I believe the failure is linked to Pyarrow and Cmake. I'm running MacOS High Sierra 10.13
I'm trying to install Streamlit with pip. I've tried using an virtual environment with Python version 3.9 and 3.10. Not sure if reinstalling Python will fix it...? Any help would be much appreciated, ...
0
votes
3
answers
178
views
Split a parquet file by groups
I have a large-ish dataframe in a Parquet file and I want to split it into multiple files to leverage Hive partitioning with pyarrow.
Preferably without loading all data into memory.
(This question ...
1
vote
0
answers
137
views
ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found
I'm starting to experiment with pyarrow, and I'm hitting a strange error when writing a CSV file. Say I have this CSV input as dates.csv:
dates
2022-10-04T15:52:25.000Z
2022-03-29T08:08:13.000Z
2023-...
0
votes
0
answers
81
views
Why pandas.read_parquet only use one CPU?
I'm using pandas.read_parquet to read about 66GB from s3 into memory on a SageMaker instance ml.g5.24xlarge. I use engine='pyarrow', and use_threads=True. It took about 8 mins to finish. When I ...
0
votes
0
answers
83
views
How to specify the file encoding for pyarrow.csv.write_csv?
When reading a CSV file with pyarrow, you can specify the encoding with a pyarrow.csv.ReadOptions constructor.
However, I find no equivalent on pyarrow.csv.write_csv:
The pyarrow.csv.WriteOptions ...
0
votes
0
answers
82
views
Pyarrow issue with hdfs connectivity
Earlier we were using pyarrow 0.13.0 version with python 3.7.16 version and through hdfs.connect we were able to establish connection with hdfs. As python version got upgraded to 3.9.10 we have to ...
0
votes
0
answers
51
views
How to iterate through a folder of zipped .csv files to create & save multiple dataframes based on the unique values of a dataframe column?
I'm setting up a data pipeline and need to start by preprocessing a set of zipped .csv files. These are in a source folder and number ~3K. Each zipped file represents a full day's worth of S&P ...