Questions tagged [pyarrow]

pyarrow is a Python interface for Apache Arrow

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

PyArrow - Working with larger than memory Arrow IPC (feather) files?

I'm trying to work Arrow IPC (feather) files which are larger than memory. PyArrow's documentation notes that "Arrow IPC files can be memory-mapped locally, which allow you to work with data ...
Matt Robin's user avatar
1 vote
1 answer
44 views

Writing DataFrames as partitioned parquet object in Polars with PyArrow

I have 12 parquet files in a directory with matching columns I am trying to write to a partitioned object with Polars and PyArrow. I am iterating through each file in the directory and reading it in ...
Stephen Nicholson's user avatar
1 vote
1 answer
26 views

Using pyarrow read_json() to parse a list of structs into a StructArray

I can successfully use pyarrow read_json to read some JSON financial trading data into a table. The JSON contains two lists of structs, asks and bids. schema = pa.schema([('time', pa.timestamp(unit='...
John Duffy's user avatar
0 votes
1 answer
37 views

How to iterate and read one row at at time from multiple parquet files?

I know we can use pyarrow to read batches of rows from a given parquet file import pyarrow.parquet as pq parquet_file = pq.ParquetFile('/mybigparquet.pq') for i in parquet_file.iter_batches(...
ℕʘʘḆḽḘ's user avatar
0 votes
1 answer
28 views

use gensim with a pyarrow iterable

Consider this code import pyarrow.parquet as pq from gensim.models import Word2Vec parquet_file = pq.ParquetFile('/mybigparquet.pq') for i in parquet_file.iter_batches(batch_size=100): print(&...
ℕʘʘḆḽḘ's user avatar
2 votes
0 answers
41 views

pyarrow breaks pyodbc MySQL?

I have a Docker container with MySQL ODBC driver, unixODBC, and a bunch of Python stuff installed. My MySQL driver works through isql, and it works when connecting from Python with pyodbc, if I do so ...
Aron's user avatar
  • 1,572
-1 votes
0 answers
38 views

PyArrow Table to Spark DataFrame

import pyarrow.parquet as pq table = pq.read_table(parquet_blob_data) Now how to create a spark dataframe from table without using pandas library or to_pandas() in-built method? I found only ...
Uday kumar's user avatar
0 votes
1 answer
27 views

Error ModuleNotFoundError: No module named 'cmake' when installing pyarrow [duplicate]

I'm trying to install streamlit, which requires pyarrow. But when I install pyarrow, I get the errow "ModuleNotFoundError: No module named 'cmake'". I have already installed cmake (version 3....
Helen's user avatar
  • 1
2 votes
1 answer
76 views

Python Polars: Low memory read, process, writing of parquet to/from Hadoop

I would like to be able to process very large files in Polars without running out of memory. In the documentation they suggest using scanning, lazyframes and sinks, but it is hard to find proper ...
Esben Eickhardt's user avatar
0 votes
0 answers
27 views

Replace all inf, -inf values with NaN in a pandas dataframe if dtype backend is arrow instead of numpy

I was wondering if with the changes of pandas going forward from 2.0 to 3.0, the solution of using replace still applies if we don't know how +-Inf values are stored, i.e. their types. What is a ...
LucaM's user avatar
  • 49
1 vote
3 answers
55 views

How to write and read dataframe to parquet where column contains list of dicts

I have a column that contain a list of dictionaries and I'm trying to write it to disk using parquet and reading it back into the same original object. However I'm not able to get the same exact ...
Miguel's user avatar
  • 13
0 votes
1 answer
31 views

How to perform a conditional sum in pyarrow

I'm trying to map a SQL query over to pyarrow and it's my first time tinkering with pyarrow. I can't find a compute function that seems to fit the bill. select group_col_1, group_col_2, sum( ...
Stephen Lloyd's user avatar
0 votes
0 answers
18 views

Unable to set up HDFS interface with PyArrow

I am trying to access the HDFS directory using pyarrow as follows. Install Hadoop and Spark Locate libhdfs.so /usr/local/hadoop-3.3.5/lib/native/libhdfs.so Create Home Directories import os os....
Calcutta's user avatar
  • 1,009
1 vote
0 answers
26 views

Specifying logical types (in particular, UUID) when writing parquet files from pyarrow

The pyarrow documentation builds a custom UUID type many times like this: import pyarrow as pa class UuidType(pa.PyExtensionType): def __init__(self): pa.PyExtensionType.__init__(self, pa....
NoName's user avatar
  • 111
0 votes
0 answers
58 views

pandas.convert_dtype(dtype_backend="pyarrow") raises error

I'm starting to use pandas with Pyarrow and realized there's no easy way to declare a df directly as with Pyarrow engine, so I ended up in this post. The thing is that I'm having the same isssue as ...
juanmac's user avatar
  • 121
0 votes
1 answer
32 views

Why does the sum function not work with pyarrow fields?

Consider the following list of fields. import pyarrow.dataset as ds fields = [ds.field('a'), ds.field('b'), ds.field('c')] sum(fields) produces the following error: TypeError: Argument 'self' has ...
actual_panda's user avatar
  • 1,128
0 votes
1 answer
117 views

Construct filtering expression based on lexicographical ordering

I have a dataset with integer columns for 'year', 'month' and 'day'. For a given start_date and end_date (pandas Timestamps), I want to build an expression to filter for all the rows between these ...
actual_panda's user avatar
  • 1,128
0 votes
0 answers
17 views

How to convert R Arrow expression to pyarrow equivalent. Larger than memory dataset

I am trying to run a simple query on full NYC taxi dataset (my local copy on HDD), which counts number of rows per group, i.e group by X then count (*) In R-arrow, this can be done using nyc_taxi = ...
Pratham's user avatar
  • 159
2 votes
1 answer
72 views

How to apply ip lookup using polars?

Given two tables I'd like to conduct a lookup over all ips and find the network it belongs to: I have two large tables: and the following networks: Regarding the ClientIP (First table) I thought of ...
JammingThebBits's user avatar
1 vote
0 answers
25 views

Loading int16 data with NULLs from snowflake into a pandas DataFrame

I am looking for an automated way to fetch data that doesn't fit into numpy datatypes (e.g. nullable ints) from a snowflake db (using the snowflake connector). E.g. I have a query that loads a column ...
Hans's user avatar
  • 1,741
0 votes
1 answer
28 views

Create a dask array with pyarrow dtype

In pandas, I can create a series with a pyarrow dtype the following way: >>> import pandas as pd >>> s = pd.Series([1,2,3]).astype("int64[pyarrow]") >>> s.dtype ...
Adrien Pacifico's user avatar
0 votes
1 answer
53 views

Unable to install pyarrow - ERROR: Failed building wheel for pyarrow

I am using python v 3.8 , numpy-1.24.4 pandas-2.0.3, cmake 3.27.3 on Windows 10 - 64 bit However ,when I try to install pyarrow using pip install pyarrow I get the error below File "<string&...
stackoverflow rohit's user avatar
0 votes
0 answers
71 views

Disable ssl validation while connecting to s3 using pyarrow fs library/ s3fs library

I am using pyarrow fs.S3FileSystem library to write a csv to s3 bucket. Although this code runs fine in my local when I deploy to VM (linux) it throws error: OSError: When listing objects under key xx ...
prashant's user avatar
0 votes
1 answer
37 views

How does Copy-on-Write (CoW) work when using the new pandas pyarrow backend?

I think the question says it all — the pandas docs have good information on both the new CoW behavior and the pyarrow backend, but was wondering if CoW works the same with pyarrow as it does with a ...
nick_eu's user avatar
  • 3,573
1 vote
1 answer
45 views

To_parquet giving error: __arrow_array__() got an unexpected keyword argument 'type'

I'm reading a root file using uproot and converting parts of it into a DataFrame using the arrays method. This works fine, until I try to save to parquet using the to_parquet method on the dataframe. ...
Gozmit97's user avatar
0 votes
0 answers
193 views

pyarrow.lib.ArrowInvalid: ("Could not convert 'col_x' with type str: tried to convert to double", 'Conversion failed for column with type object')

While enforcing pyarrrow schema on a pandas dataframe (to create a parquet file of the same), there is a number(24,3) datatype column coming from Oracle which is not being typecasted and it is ...
Ashwini's user avatar
2 votes
1 answer
48 views

Read/write partitioned parquet from/to SFTP server with pyarrow

recently I got myself into data analysis with some friends and to improve our data exchange we got a linux server which we use as a SFTP server. Following this we no longer want to write outputs to ...
Drax's user avatar
  • 23
0 votes
0 answers
98 views

CMake and PyArrow Errors While Installing Streamlit

During the installation of Streamlit using the 'pip install streamlit' command, I encountered errors related to both CMake and PyArrow. The process failed with the following error messages: -- ...
Zoro's user avatar
  • 1
3 votes
0 answers
88 views

pyarrow memory consumption difference between Dataset.to_batches and ParquetFile.iter_batches

I am using pyarrow and am struggling to understand the big difference in memory usage between the Dataset.to_batches method compared to ParquetFile.iter_batches. Using pyarrow.dataset >>> ...
teejay's user avatar
  • 103
0 votes
1 answer
73 views

Apache Arrow IPC streams: SPMC concurrency

Is it expected that a multiple IPC stream readers can concurrently tail a single stream writer publishing from another process? The descriptions of things like "IPC streams" leads me to ...
kdkavanagh's user avatar
0 votes
0 answers
37 views

Is there a way to apply custom filters when reading a row group from a parquet file with pyarrow?

I'm trying to figure out how Modin could implement the ability to read Parquet files using filters, with the same behavior as pandas.read_parquet('my_parquet_file.parquet', filters=[('col', '==', 'val'...
zmbc's user avatar
  • 58
0 votes
1 answer
36 views

read_parquet and merge schema

I have a lot of parquet files which are partitioned by YEAR/MONTH/DAY, which I read in with read_parquet(PATH, engine="pyarrow"). All parquets have the columns "column_1", column_2&...
Max86's user avatar
  • 65
0 votes
0 answers
159 views

FIFO Queue with Polars DataFrame

I'm currently working on a use case that involves the usage of sensors and IoT data in the form of a (multivariate) time series. The final goal is to produce an ML model for fault/anomaly detection. ...
Marco Bignotti's user avatar
1 vote
0 answers
24 views

Error while loading parquet format on the basis of date or size or required size format from s3

I am trying to load parquet file format data from s3 in a filtered fashion, by loading few MB or load it on the basis of date code: obj = s3_client.get_object(Bucket=bucket_name, Key=file_path1, ...
amm's user avatar
  • 11
1 vote
1 answer
540 views

What is actually meant when referring to parquet row-group size?

I am starting to work with the parquet file format. The official Apache site recommends large row groups of 512MB to 1GB (here). Several online source (e.g. this one) suggest that the default row ...
teejay's user avatar
  • 103
0 votes
0 answers
213 views

Using scan_parquet and scan_pyarrow_dataset on S3 loads entire dataset into memory

I have a large, partitioned, parquet dataset that does not fit into memory and is stored in an S3 location. A piece of of processing code is meant to process a subset of this data by performing a join ...
bvdw's user avatar
  • 1
0 votes
2 answers
158 views

How to send arrow data from fastapi to the js apache arrow package without copying?

Let's say generically my setup is like this: from fastapi import FastAPI, Response import pyarrow as pa import pyarrow.ipc as ipc app = FastAPI() @app.get("/api/getdata") async def getdata()...
Dean MacGregor's user avatar
0 votes
0 answers
41 views

Snowflake cursor fetch_arrow_all returns 1816 date

I have a use case where I need to fetch snowflake response as pyarrow table using fetch_arrow_all. The date which are like 9999-10-10 getting defaulted to 1815/1816. cursor.execute("select ...
Mammu yedukondalu's user avatar
0 votes
0 answers
103 views

Pyarrow writing parquet dataset behavior after updating pyarrow

I'm working on a codebase that processes daily data. We have a Pyarrow Table with 3 columns, one of which is a date. The data goes back to 2007, so ~ 5800 unique dates. The table has 43 million rows, ...
CliveCoYo's user avatar
0 votes
0 answers
138 views

Memory consumption in PyArrow iter_batches

Let's suppose I have n parquet files that represent the columns of my final parquet file. The columns have 5 milion rows and I want to read my entire file in batches and each batch is given by the ...
marco's user avatar
  • 525
2 votes
0 answers
165 views

Cannot install pyarrow

I am having issues with installing pyarrow and any package that relies on it (e.g., pandas-gbq). I tried various combinations of modules (including different cmake versions), but always get an error ...
Marius Zoican's user avatar
0 votes
1 answer
28 views

got an length eroor when using read_parquet function

When i use read_parquet method to read parquet file, it occurs Column 8 named hostIp expected length 548 but got length 549 error, hostIP is one column in REQUIRED_COLUMNS. import pandas as pd ...
chunhuagod's user avatar
0 votes
1 answer
34 views

Parquet / pyarrow: Malformed levels

I use Azure Stream Analytics which converts some json documents in parquet files. For most of them, I can read them after but for some of them I get the following error: Traceback (most recent call ...
Kiva's user avatar
  • 9,213
-1 votes
1 answer
247 views

Having issue installing 'Streamlit' with pip, I believe the failure is linked to Pyarrow and Cmake. I'm running MacOS High Sierra 10.13

I'm trying to install Streamlit with pip. I've tried using an virtual environment with Python version 3.9 and 3.10. Not sure if reinstalling Python will fix it...? Any help would be much appreciated, ...
jgroshak's user avatar
0 votes
3 answers
178 views

Split a parquet file by groups

I have a large-ish dataframe in a Parquet file and I want to split it into multiple files to leverage Hive partitioning with pyarrow. Preferably without loading all data into memory. (This question ...
robertdj's user avatar
  • 909
1 vote
0 answers
137 views

ArrowInvalid: Cannot locate timezone 'UTC': Timezone database not found

I'm starting to experiment with pyarrow, and I'm hitting a strange error when writing a CSV file. Say I have this CSV input as dates.csv: dates 2022-10-04T15:52:25.000Z 2022-03-29T08:08:13.000Z 2023-...
mrgou's user avatar
  • 1,598
0 votes
0 answers
81 views

Why pandas.read_parquet only use one CPU?

I'm using pandas.read_parquet to read about 66GB from s3 into memory on a SageMaker instance ml.g5.24xlarge. I use engine='pyarrow', and use_threads=True. It took about 8 mins to finish. When I ...
Bewang's user avatar
  • 453
0 votes
0 answers
83 views

How to specify the file encoding for pyarrow.csv.write_csv?

When reading a CSV file with pyarrow, you can specify the encoding with a pyarrow.csv.ReadOptions constructor. However, I find no equivalent on pyarrow.csv.write_csv: The pyarrow.csv.WriteOptions ...
mrgou's user avatar
  • 1,598
0 votes
0 answers
82 views

Pyarrow issue with hdfs connectivity

Earlier we were using pyarrow 0.13.0 version with python 3.7.16 version and through hdfs.connect we were able to establish connection with hdfs. As python version got upgraded to 3.9.10 we have to ...
Abhay's user avatar
  • 63
0 votes
0 answers
51 views

How to iterate through a folder of zipped .csv files to create & save multiple dataframes based on the unique values of a dataframe column?

I'm setting up a data pipeline and need to start by preprocessing a set of zipped .csv files. These are in a source folder and number ~3K. Each zipped file represents a full day's worth of S&P ...
cel's user avatar
  • 1

1
2 3 4 5
22