Questions tagged [apache-arrow]
Apache Arrow™ enables execution engines to take advantage of the latest SIM D (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing.
629
questions
0
votes
1
answer
25
views
Frequency of A list in a Pyarrow Table
Find myself in an interesting conundrum, in that I need to compute the frequency of a series of lists by participant ID. Data looks not unlike the following:
test_table_orig = pa.table([
pa.array([...
1
vote
1
answer
53
views
Reading a column of strings from a parquet file into C++ string variables
I have a parquet file which I have created using the python polars package. It has a single column of variable length strings that looks like:
┌──────────┐
│ str_list │
│ --- │
│ str │
╞═════...
0
votes
0
answers
43
views
Remove rows out-of-memory in arrow
Suppose I have a huge dataset that does not fit into memory.
How can I delete rows in an arrow without loading the dataset into memory, and update the already changed dataset on disk.
library(arrow)
...
3
votes
1
answer
97
views
Why are the offsets produced by arrow::ListArray::offsets() interspersed with zeros?
I am trying to read the data in a parquet file containing lists of floats using C++.
I produced a simple parquet file using the following python code:
import polars as pl
import struct
import random
...
2
votes
2
answers
36
views
How to convert int to double when using arrow to read in multiple CSVs with open_dataset in R?
I'm trying to convert a large folder of CSVs to parquet. The dataset it too large to fit into memory, so I'm using arrow.
For some columns, some files have only whole numbers, and others have decimal/...
0
votes
1
answer
70
views
How to efficiently extract rows arrow package
Please advise. I have about 30 csv files, each 1-3 gigabytes. The files have the same columns, I open them as one large file through arrow. I need to quickly extract sub-samples of rows from the ...
0
votes
0
answers
43
views
How to read arrow ipc data in Rust
I'm trying to translate a working Python code into Rust. This code receives a JSON payload (the event object in the following snippet), the entries are the following:
input_schema: a string ...
0
votes
1
answer
28
views
Getting the Most Frequent Value by An Aggregated Group in a Pyarrow Table
I have a pyarrow table that looks something like the following:
import pandas as pd
import pyarrow as pa
import pyarrow.compute as pc
test_table = pa.table([
pa.array(["a", "a"...
0
votes
0
answers
26
views
Reading individual elements from a arrow::Array column using GetScalar() not compiling
I am trying to read a parquet file using the Apache Arrow library. The following code opens a parquet file and attempts to read its rows using a arrow::RecordBatchReader. But it fails to build with ...
0
votes
0
answers
28
views
How to Read parquet file using apache arrow in cpp from cloud storage S3
the aws path where parquet file is stored will look something like:
/aws/default/...
I have tried using #include <arrow/filesystem/s3fs.h> but couldnt figure out the exact methods of this ...
-1
votes
2
answers
76
views
How to connect to a remote csv file with duckdb or arrow in R?
Goal
Connect to a large remote csv file to query a subset of the data.
This page has a button to download a csv file. If I copy the link and run the following, the data is loaded into memory:
foo <-...
1
vote
0
answers
67
views
Data formats for very large data while preserving data.table functionality
I have a script which produces several intermediate data files which would significantly exceed the maximum number of rows in R (2^31-1). My system is large enough to store the data (e.g. I can store ...
2
votes
1
answer
28
views
dplyr syntax for arrow to sum columns specified in a variable
Working in R, I would like Arrow to sum a set of columns specified in a variable.
library(arrow)
library(dplyr)
example_data = InMemoryDataset$create(data.frame(a1 = c(1,2,3), b2=c(4,5,6), c3=c(7,8,...
0
votes
0
answers
25
views
The most efficient way of downloading a parquet file and saving it to local disk for use with pyarrow.memory_map
I have a large input dataset that needs some processing that can be parallelized. I've been able to scale to a fairly large number of parallel processes by first storing a huge input pyarrow table to ...
1
vote
0
answers
61
views
Apache Arrow R package || Error in dplyr verbs
I am trying to run Apache Arrow using Dplyr verbs -
In Sample code, I am getting following error, Anyone tried Apache Arrow R package with Dplyr verbs????
arrow_table(starwars) %>%
...
-2
votes
0
answers
24
views
Why am I getting link error undefined reference to `arrow::fs::GcsOptions::Anonymous()
I am encounter a link problem ".text.startup+0x229): undefined reference to `arrow::fs::GcsOptions::Anonymous()'namespace gcs = ::google::cloud::storage;
gcs::ClientOptions cc =gcs::...
0
votes
1
answer
33
views
pyarrow write dataset drops partition columns
I am trying to use pyarrow to partition and write parquet files
!pip install pyarrow==13.0.0
import pyarrow as pa
table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
'n_legs'...
0
votes
1
answer
41
views
Applying custom functions to parquet files loaded with open_dataset
Apologies if this has been applied somewhere else, but I haven't been able to find a solution that works.
I have some data, spread across 4232 parquet files, that aggregate into roughly 6745697 rows ...
0
votes
0
answers
36
views
Data Manipulation in PyArrow for Large Datasets
I am working on a project where I need to process a large dataset (50GB) with limited available memory (12GB). I have a function written in Pandas that performs grouping and calculates various ...
1
vote
2
answers
53
views
R Arrow finds no compress even when it is installed
Currently I am trying to generate zstd compressed Apache Parquet files in R in an docker container.
Even when I install all dependencies and arrow itself works fine it does not find the zstd (or ...
1
vote
1
answer
99
views
What are the aims for the Apache Arrow Tensor Extension types?
Apache Arrow (Columnar Format) is a "standard and efficient in-memory representation of various datatypes".
DLPack is an "open in-memory tensor structure for sharing tensors among ...
0
votes
1
answer
41
views
How to get an Arrow's Array's TimeUnit C++
It's a very simple example. I am trying to extract the TimeUnit::type from the DataType.
array = arrow::Array *array
auto a = static_cast<arrow::Date64Array*>(array);
arrow::...
0
votes
0
answers
57
views
How to iterate through all partitions in a PyArrow FileSystemDataset for aggregations
I have a parquet dataset partitioned by year, and I am trying to iterate through each year partition to do aggregations on each one, instead of loading the entire dataset into memory at once.
The ...
0
votes
0
answers
28
views
How to process the results of a compute operation? (arrow::compute::ValueCounts)
I'm trying to count the number of occurrences of different words in a column (arrow::ChunkedArray) of arrow::Type::String.
I know that compute::ValueCounts returns a dictionary {"values":,&...
0
votes
0
answers
109
views
How to convert a RecordBatch (Arrow type) into a Polars Dataframe in Rust
I have a record batch data in the following format.
Defining a schema
let schema = Arc::new(Schema::new(vec![
Field::new("store_id", DataType::Int64, false),
Field::new("week_id&...
0
votes
1
answer
52
views
R+arrow: reading double with comma decimal separator
Please have a look at the code at the end of the post.
You can download the input tsv file (nothing malicious!) from
https://e.pcloud.link/publink/show?code=XZ5eCWZdrwFuo5POSVzi7ywCmteHfE4rdmV
I am ...
0
votes
0
answers
45
views
How to split apache arrow flight data into chunks in go
The arrow flight data is columnar data and I want to implement DoGet function in such a way that it firstly split arrow.record in to specific size of data (let say 4 MB), then send it to the client ...
0
votes
1
answer
100
views
Dynamically constructing a query with the `arrow` package in R
I'm using the arrow package to interface with a large (30GB) dataset of parquet files in R. This works great when I have a single clear query but when I try to construct queries dynamically I'm unsure ...
1
vote
0
answers
45
views
What is the difference between pd.ArrowDtype(pa.string()) and pd.StringDtype("pyarrow")?
From here we can read: https://pandas.pydata.org/docs/user_guide/pyarrow.html
The string alias "string[pyarrow]" maps to pd.StringDtype("pyarrow") which is not equivalent to ...
0
votes
1
answer
80
views
R: how to save multidimensional array to disk with arrow/parquet?
in R, I would like to save a multidimensional array (say 10 dimensions or more, each dimension with possibly different lengths) to disk in a cross-platform format. For this reason I'm excluding .rds ...
0
votes
0
answers
134
views
PyArrow - Working with larger than memory Arrow IPC (feather) files?
I'm trying to work Arrow IPC (feather) files which are larger than memory. PyArrow's documentation notes that "Arrow IPC files can be memory-mapped locally, which allow you to work with data ...
0
votes
0
answers
26
views
Different options for package installations on Linux / Windows
I have develope R package findata which contains arrow R package:
https://github.com/MislavSag/findata/blob/main/DESCRIPTION
I can install R package on Windows, but on linux it the package never ...
-1
votes
1
answer
68
views
Is it possible to convert a column contain numeric and character into numeric in R arrow?
For example,
if I run the following code:
library(arrow)
library(tidyverse)
mtcars$cyl[2:5] = 'a'
mtcars %>%
arrow_table() %>%
mutate(cyl = cast(cyl, numeric())) %>%
collect()
An ...
0
votes
1
answer
43
views
Apache Arrow Datafusion RecordBatch with timestamp
I’m trying to clone an existing delta table locally using delta-rs.
I’m using DeltaTable.get_schema().unwrap().clone(); to get the schema and I create a new table with
let mut config = DeltaOps(...
0
votes
0
answers
56
views
Running datafusion query-aws-s3 example results to an error
I am specifically running this example from datafusion. I get this error
Error: ObjectStore(Generic { store: "S3", source: Error { retries: 0, message: "No Body", source: Some(...
0
votes
0
answers
47
views
How to load parquet file using the $read_dictionary method?
I need to read a 7.5 mb parquet file containing 5 million rows and 11 columns, but this uses up more than 14 gb of memory on my system, which is unacceptable. Some research told me that I can tell ...
1
vote
0
answers
37
views
Facing error in local environment setup for arrow with c++
Facing undefined reference error while reading an apache arrow file in c++ using apache arrow libraries. Basically I'm trying to read an arrow file generated by java in the apache arrow format.
I'm ...
1
vote
0
answers
106
views
Cannot read parquet file metadata for each field with the arrow package for R
I am trying to read metadata for each field of a parquet file hosted in a S3 bucket using arrow for R. I am able to load the parquet file into R without issue, but I only get NULL as response when ...
3
votes
0
answers
89
views
Specifying logical types (in particular, UUID) when writing parquet files from pyarrow
The pyarrow documentation builds a custom UUID type many times like this:
import pyarrow as pa
class UuidType(pa.PyExtensionType):
def __init__(self):
pa.PyExtensionType.__init__(self, pa....
1
vote
0
answers
67
views
Debug apache arrow using R
I am rather new to apache arrow hence this may be an easy question. I've tried to find information online without success. I have a algorithm I'm using to simulate data as well as perform calculations....
0
votes
0
answers
51
views
arrow/python/platform.h: No such file or directory
Iam using apache arrow with C++ and Iam using cython to make an interface between some of the C++ function and python
when I try to build the setup file I get this error message :
arrow/python/...
1
vote
1
answer
48
views
How to setup and use Apache arrow from within Xcode C++ project
I need to read and write some parquet.gzip files within my C++ project in Xcode. From what I understand I should use the library Arrow from Apache. However I am struggling to use it within my Xcode ...
0
votes
1
answer
60
views
how to get real row indices R arrow package
let's imagine that I have a huge file divided into many pieces that I can read with an arrow
X <- iris
library(arrow)
# bigX a large file that is stored in chunks on disk out of memory
bigX <- ...
5
votes
1
answer
181
views
Deterministic random number generation in duckdb with dplyr syntax
How can I use duckdb's setseed() function (see reference doc) with dplyr syntax to make sure the analysis below is reproducible?
# dplyr version 1.1.1
# arrow version 11.0.0.3
# duckdb 0.7.1.1
out_dir ...
0
votes
0
answers
21
views
How to convert R Arrow expression to pyarrow equivalent. Larger than memory dataset
I am trying to run a simple query on full NYC taxi dataset (my local copy on HDD), which counts number of rows per group, i.e group by X then count (*)
In R-arrow, this can be done using
nyc_taxi = ...
0
votes
1
answer
47
views
Spark to pandas: np.nan is converted differently if Arrow is enabled
I noticed that NULL values in pandas (np.nan) are converted differently depending on whether or not Arrow is enabled. I was wondering whether this behaviour is intentional? If so, shouldn't this be ...
1
vote
1
answer
209
views
Filter based on a list column using arrow and duckdb
I'm using the R arrow package to interact with a duckdb table that contains a list column. My goal is to filter on the list column before collecting the results into memory. Can this be accomplished ...
1
vote
1
answer
53
views
arrow::to_duckdb coerces int64 columns to doubles
arrow::to_duckdb() converts int64 columns to a double in the duckdb table. This happens if the .data being converted is an R data frame or a parquet file. How can I maintain the int64 data type?
...
0
votes
1
answer
86
views
Create a dask array with pyarrow dtype
In pandas, I can create a series with a pyarrow dtype the following way:
>>> import pandas as pd
>>> s = pd.Series([1,2,3]).astype("int64[pyarrow]")
>>> s.dtype
...
-1
votes
1
answer
123
views
How to convert a buffer of data into a arrow::Table without intermediate file creation in C++?
I have a pipeline of processes that do different stuff. One of the pipes reads a file and de-compresses it into a buffer. The buffer in question contains an arrow Table. There is a component that ...