Questions tagged [apache-arrow]

Apache Arrow™ enables execution engines to take advantage of the latest SIM D (Single input multiple data) operations included in modern processors, for native vectorized optimization of analytical data processing.

Filter by
Sorted by
Tagged with
0 votes
1 answer
25 views

Frequency of A list in a Pyarrow Table

Find myself in an interesting conundrum, in that I need to compute the frequency of a series of lists by participant ID. Data looks not unlike the following: test_table_orig = pa.table([ pa.array([...
TDeramus's user avatar
  • 117
1 vote
1 answer
53 views

Reading a column of strings from a parquet file into C++ string variables

I have a parquet file which I have created using the python polars package. It has a single column of variable length strings that looks like: ┌──────────┐ │ str_list │ │ --- │ │ str │ ╞═════...
vinnewbie's user avatar
0 votes
0 answers
43 views

Remove rows out-of-memory in arrow

Suppose I have a huge dataset that does not fit into memory. How can I delete rows in an arrow without loading the dataset into memory, and update the already changed dataset on disk. library(arrow) ...
mr.T's user avatar
  • 275
3 votes
1 answer
97 views

Why are the offsets produced by arrow::ListArray::offsets() interspersed with zeros?

I am trying to read the data in a parquet file containing lists of floats using C++. I produced a simple parquet file using the following python code: import polars as pl import struct import random ...
vinnewbie's user avatar
2 votes
2 answers
36 views

How to convert int to double when using arrow to read in multiple CSVs with open_dataset in R?

I'm trying to convert a large folder of CSVs to parquet. The dataset it too large to fit into memory, so I'm using arrow. For some columns, some files have only whole numbers, and others have decimal/...
falsePockets's user avatar
  • 3,925
0 votes
1 answer
70 views

How to efficiently extract rows arrow package

Please advise. I have about 30 csv files, each 1-3 gigabytes. The files have the same columns, I open them as one large file through arrow. I need to quickly extract sub-samples of rows from the ...
mr.T's user avatar
  • 275
0 votes
0 answers
43 views

How to read arrow ipc data in Rust

I'm trying to translate a working Python code into Rust. This code receives a JSON payload (the event object in the following snippet), the entries are the following: input_schema: a string ...
André Claudino's user avatar
0 votes
1 answer
28 views

Getting the Most Frequent Value by An Aggregated Group in a Pyarrow Table

I have a pyarrow table that looks something like the following: import pandas as pd import pyarrow as pa import pyarrow.compute as pc test_table = pa.table([ pa.array(["a", "a"...
TDeramus's user avatar
  • 117
0 votes
0 answers
26 views

Reading individual elements from a arrow::Array column using GetScalar() not compiling

I am trying to read a parquet file using the Apache Arrow library. The following code opens a parquet file and attempts to read its rows using a arrow::RecordBatchReader. But it fails to build with ...
vinnewbie's user avatar
0 votes
0 answers
28 views

How to Read parquet file using apache arrow in cpp from cloud storage S3

the aws path where parquet file is stored will look something like: /aws/default/... I have tried using #include <arrow/filesystem/s3fs.h> but couldnt figure out the exact methods of this ...
vikramjit dutta's user avatar
-1 votes
2 answers
76 views

How to connect to a remote csv file with duckdb or arrow in R?

Goal Connect to a large remote csv file to query a subset of the data. This page has a button to download a csv file. If I copy the link and run the following, the data is loaded into memory: foo <-...
umair durrani's user avatar
1 vote
0 answers
67 views

Data formats for very large data while preserving data.table functionality

I have a script which produces several intermediate data files which would significantly exceed the maximum number of rows in R (2^31-1). My system is large enough to store the data (e.g. I can store ...
Nils R's user avatar
  • 105
2 votes
1 answer
28 views

dplyr syntax for arrow to sum columns specified in a variable

Working in R, I would like Arrow to sum a set of columns specified in a variable. library(arrow) library(dplyr) example_data = InMemoryDataset$create(data.frame(a1 = c(1,2,3), b2=c(4,5,6), c3=c(7,8,...
Maciek's user avatar
  • 35
0 votes
0 answers
25 views

The most efficient way of downloading a parquet file and saving it to local disk for use with pyarrow.memory_map

I have a large input dataset that needs some processing that can be parallelized. I've been able to scale to a fairly large number of parallel processes by first storing a huge input pyarrow table to ...
Sergey's user avatar
  • 36
1 vote
0 answers
61 views

Apache Arrow R package || Error in dplyr verbs

I am trying to run Apache Arrow using Dplyr verbs - In Sample code, I am getting following error, Anyone tried Apache Arrow R package with Dplyr verbs???? arrow_table(starwars) %>% ...
Harshal Khanolkar's user avatar
-2 votes
0 answers
24 views

Why am I getting link error undefined reference to `arrow::fs::GcsOptions::Anonymous()

I am encounter a link problem ".text.startup+0x229): undefined reference to `arrow::fs::GcsOptions::Anonymous()'namespace gcs = ::google::cloud::storage;  gcs::ClientOptions cc =gcs::...
Razi's user avatar
  • 1
0 votes
1 answer
33 views

pyarrow write dataset drops partition columns

I am trying to use pyarrow to partition and write parquet files !pip install pyarrow==13.0.0 import pyarrow as pa table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021], 'n_legs'...
Swavig's user avatar
  • 37
0 votes
1 answer
41 views

Applying custom functions to parquet files loaded with open_dataset

Apologies if this has been applied somewhere else, but I haven't been able to find a solution that works. I have some data, spread across 4232 parquet files, that aggregate into roughly 6745697 rows ...
TDeramus's user avatar
  • 117
0 votes
0 answers
36 views

Data Manipulation in PyArrow for Large Datasets

I am working on a project where I need to process a large dataset (50GB) with limited available memory (12GB). I have a function written in Pandas that performs grouping and calculates various ...
kappa101's user avatar
1 vote
2 answers
53 views

R Arrow finds no compress even when it is installed

Currently I am trying to generate zstd compressed Apache Parquet files in R in an docker container. Even when I install all dependencies and arrow itself works fine it does not find the zstd (or ...
Someone2's user avatar
  • 461
1 vote
1 answer
99 views

What are the aims for the Apache Arrow Tensor Extension types?

Apache Arrow (Columnar Format) is a "standard and efficient in-memory representation of various datatypes". DLPack is an "open in-memory tensor structure for sharing tensors among ...
nenb's user avatar
  • 75
0 votes
1 answer
41 views

How to get an Arrow's Array's TimeUnit C++

It's a very simple example. I am trying to extract the TimeUnit::type from the DataType. array = arrow::Array *array auto a = static_cast<arrow::Date64Array*>(array); arrow::...
Ochetski's user avatar
  • 115
0 votes
0 answers
57 views

How to iterate through all partitions in a PyArrow FileSystemDataset for aggregations

I have a parquet dataset partitioned by year, and I am trying to iterate through each year partition to do aggregations on each one, instead of loading the entire dataset into memory at once. The ...
Stephen Nicholson's user avatar
0 votes
0 answers
28 views

How to process the results of a compute operation? (arrow::compute::ValueCounts)

I'm trying to count the number of occurrences of different words in a column (arrow::ChunkedArray) of arrow::Type::String. I know that compute::ValueCounts returns a dictionary {"values":,&...
Nicolas Loira's user avatar
0 votes
0 answers
109 views

How to convert a RecordBatch (Arrow type) into a Polars Dataframe in Rust

I have a record batch data in the following format. Defining a schema let schema = Arc::new(Schema::new(vec![ Field::new("store_id", DataType::Int64, false), Field::new("week_id&...
amit r's user avatar
  • 1
0 votes
1 answer
52 views

R+arrow: reading double with comma decimal separator

Please have a look at the code at the end of the post. You can download the input tsv file (nothing malicious!) from https://e.pcloud.link/publink/show?code=XZ5eCWZdrwFuo5POSVzi7ywCmteHfE4rdmV I am ...
larry77's user avatar
  • 1,327
0 votes
0 answers
45 views

How to split apache arrow flight data into chunks in go

The arrow flight data is columnar data and I want to implement DoGet function in such a way that it firstly split arrow.record in to specific size of data (let say 4 MB), then send it to the client ...
liana's user avatar
  • 37
0 votes
1 answer
100 views

Dynamically constructing a query with the `arrow` package in R

I'm using the arrow package to interface with a large (30GB) dataset of parquet files in R. This works great when I have a single clear query but when I try to construct queries dynamically I'm unsure ...
Dubukay's user avatar
  • 1,776
1 vote
0 answers
45 views

What is the difference between pd.ArrowDtype(pa.string()) and pd.StringDtype("pyarrow")?

From here we can read: https://pandas.pydata.org/docs/user_guide/pyarrow.html The string alias "string[pyarrow]" maps to pd.StringDtype("pyarrow") which is not equivalent to ...
Ziur Olpa's user avatar
  • 1,861
0 votes
1 answer
80 views

R: how to save multidimensional array to disk with arrow/parquet?

in R, I would like to save a multidimensional array (say 10 dimensions or more, each dimension with possibly different lengths) to disk in a cross-platform format. For this reason I'm excluding .rds ...
pglpm's user avatar
  • 568
0 votes
0 answers
134 views

PyArrow - Working with larger than memory Arrow IPC (feather) files?

I'm trying to work Arrow IPC (feather) files which are larger than memory. PyArrow's documentation notes that "Arrow IPC files can be memory-mapped locally, which allow you to work with data ...
Matt Robin's user avatar
0 votes
0 answers
26 views

Different options for package installations on Linux / Windows

I have develope R package findata which contains arrow R package: https://github.com/MislavSag/findata/blob/main/DESCRIPTION I can install R package on Windows, but on linux it the package never ...
Mislav Sagovac's user avatar
-1 votes
1 answer
68 views

Is it possible to convert a column contain numeric and character into numeric in R arrow?

For example, if I run the following code: library(arrow) library(tidyverse) mtcars$cyl[2:5] = 'a' mtcars %>% arrow_table() %>% mutate(cyl = cast(cyl, numeric())) %>% collect() An ...
doraemon's user avatar
  • 675
0 votes
1 answer
43 views

Apache Arrow Datafusion RecordBatch with timestamp

I’m trying to clone an existing delta table locally using delta-rs. I’m using DeltaTable.get_schema().unwrap().clone(); to get the schema and I create a new table with let mut config = DeltaOps(...
user1920407's user avatar
0 votes
0 answers
56 views

Running datafusion query-aws-s3 example results to an error

I am specifically running this example from datafusion. I get this error Error: ObjectStore(Generic { store: "S3", source: Error { retries: 0, message: "No Body", source: Some(...
Neil Martin's user avatar
0 votes
0 answers
47 views

How to load parquet file using the $read_dictionary method?

I need to read a 7.5 mb parquet file containing 5 million rows and 11 columns, but this uses up more than 14 gb of memory on my system, which is unacceptable. Some research told me that I can tell ...
Bastián Olea Herrera's user avatar
1 vote
0 answers
37 views

Facing error in local environment setup for arrow with c++

Facing undefined reference error while reading an apache arrow file in c++ using apache arrow libraries. Basically I'm trying to read an arrow file generated by java in the apache arrow format. I'm ...
Aswath's user avatar
  • 871
1 vote
0 answers
106 views

Cannot read parquet file metadata for each field with the arrow package for R

I am trying to read metadata for each field of a parquet file hosted in a S3 bucket using arrow for R. I am able to load the parquet file into R without issue, but I only get NULL as response when ...
lidefi87's user avatar
3 votes
0 answers
89 views

Specifying logical types (in particular, UUID) when writing parquet files from pyarrow

The pyarrow documentation builds a custom UUID type many times like this: import pyarrow as pa class UuidType(pa.PyExtensionType): def __init__(self): pa.PyExtensionType.__init__(self, pa....
NoName's user avatar
  • 131
1 vote
0 answers
67 views

Debug apache arrow using R

I am rather new to apache arrow hence this may be an easy question. I've tried to find information online without success. I have a algorithm I'm using to simulate data as well as perform calculations....
stat_student's user avatar
0 votes
0 answers
51 views

arrow/python/platform.h: No such file or directory

Iam using apache arrow with C++ and Iam using cython to make an interface between some of the C++ function and python when I try to build the setup file I get this error message : arrow/python/...
Mostafa's user avatar
1 vote
1 answer
48 views

How to setup and use Apache arrow from within Xcode C++ project

I need to read and write some parquet.gzip files within my C++ project in Xcode. From what I understand I should use the library Arrow from Apache. However I am struggling to use it within my Xcode ...
marco's user avatar
  • 177
0 votes
1 answer
60 views

how to get real row indices R arrow package

let's imagine that I have a huge file divided into many pieces that I can read with an arrow X <- iris library(arrow) # bigX a large file that is stored in chunks on disk out of memory bigX <- ...
mr.T's user avatar
  • 275
5 votes
1 answer
181 views

Deterministic random number generation in duckdb with dplyr syntax

How can I use duckdb's setseed() function (see reference doc) with dplyr syntax to make sure the analysis below is reproducible? # dplyr version 1.1.1 # arrow version 11.0.0.3 # duckdb 0.7.1.1 out_dir ...
Ashirwad's user avatar
  • 1,920
0 votes
0 answers
21 views

How to convert R Arrow expression to pyarrow equivalent. Larger than memory dataset

I am trying to run a simple query on full NYC taxi dataset (my local copy on HDD), which counts number of rows per group, i.e group by X then count (*) In R-arrow, this can be done using nyc_taxi = ...
Pratham's user avatar
  • 169
0 votes
1 answer
47 views

Spark to pandas: np.nan is converted differently if Arrow is enabled

I noticed that NULL values in pandas (np.nan) are converted differently depending on whether or not Arrow is enabled. I was wondering whether this behaviour is intentional? If so, shouldn't this be ...
Matthias Roels's user avatar
1 vote
1 answer
209 views

Filter based on a list column using arrow and duckdb

I'm using the R arrow package to interact with a duckdb table that contains a list column. My goal is to filter on the list column before collecting the results into memory. Can this be accomplished ...
davechilders's user avatar
  • 8,793
1 vote
1 answer
53 views

arrow::to_duckdb coerces int64 columns to doubles

arrow::to_duckdb() converts int64 columns to a double in the duckdb table. This happens if the .data being converted is an R data frame or a parquet file. How can I maintain the int64 data type? ...
davechilders's user avatar
  • 8,793
0 votes
1 answer
86 views

Create a dask array with pyarrow dtype

In pandas, I can create a series with a pyarrow dtype the following way: >>> import pandas as pd >>> s = pd.Series([1,2,3]).astype("int64[pyarrow]") >>> s.dtype ...
Adrien Pacifico's user avatar
-1 votes
1 answer
123 views

How to convert a buffer of data into a arrow::Table without intermediate file creation in C++?

I have a pipeline of processes that do different stuff. One of the pipes reads a file and de-compresses it into a buffer. The buffer in question contains an arrow Table. There is a component that ...
mohabouje's user avatar
  • 3,962

1
2 3 4 5
13