Questions tagged [zstd]
Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.
95
questions
-1
votes
0
answers
11
views
how to decompress zstd files with unknown original file size with zstd-jni
We have a use case where we need to retrieve multiple zstd files and decompress them and we don't know the size of the original files.
Zstd-jni requires us to provide the size of the decompressed file ...
0
votes
0
answers
13
views
Build android shared object from makefile
I want to include Facebook's zstd into my Unity project targetting Android.
The contributors made it easily to build .dll for .Net platform. But I can't make it work on the Android build, it only work ...
1
vote
1
answer
149
views
How to decode and read a zstd file in Rust?
I'm looking for some advice on how to decode and read a zstd file and I'm feeling a bit lost since it is my first big project since I started learning Rust.
I am using Rust for this project since it ...
0
votes
0
answers
43
views
arch linux install error - zstd: symbol lookup error: zstd: undefined symbol: POOL_joinJobs
After update to latest version gnome 45 and latest kernel, I'm encountering an error when trying to install applications on my Arch Linux system ( using pacman, yay, paru or makepkg ). The error ...
0
votes
1
answer
180
views
How to set "zstd" compression level in AWS Glue job?
Background
"zstd" compression codec has 22 compression levels. I read this Uber blog. Regarding compressing time and file size, I verified using df.to_parquet with our data and got same ...
0
votes
1
answer
70
views
I'm trying to open a zstd file in Python
I'm trying to open a zstd file in Python. I downloaded the file from https://the-eye.eu/redarcs/ archive. This archive provides Reddit posts and comments. When I download data from this site and open ...
2
votes
3
answers
295
views
Pandas DataFrame.write_parquet() and setting the Zstd compression level
I am writing out a compressed Parquet file from DataFrame as following:
result_df.to_parquet("my-data.parquet", compression="zstd")
How can I instruct Pandas on the compression ...
0
votes
0
answers
36
views
trino gives zstd invalid magic prefix?
I ve kerberized hive and hdfs
Trino version 422
Hadoop 3.3.4
I ve zst files on hdfs and when i try to query that table include zst files it gives this error
Trino gives error invalid magic prefix ...
0
votes
2
answers
55
views
How to decompress the compressed response In Jmeter Performance Testing tool
In JMeter Performance Testing tool I get the response in a compressed format so I need to uncompress the entire response using the zStd in the postprocessor.
I have download the jar file zstd-jni-1.5....
1
vote
0
answers
789
views
What exactly does each ZSTD compression level do?
The ZSTD compression library by Facebook offers over 19 compression levels, but sometimes some seem to have more effect on overall compression efficiency and compression time than others.
What exactly ...
0
votes
1
answer
182
views
Read Zstandard compressed Parquet in Spark 2.4.7 Hadoop 2.7.7
We have a limitation on a platform we are using which has Spark 2.4.7 and underneath Hadoop 2.7.7 libraries present.
We have some data present on s3 which is in zstandard parquet format.
Is there a ...
2
votes
0
answers
123
views
zstd: can't stat web.log.* : No such file or directory -- ignored
I can't seem to find help on using zstdgrep correctly. I thought it works like ordinary grep, but I am not getting what I want. I have several subfolders that have files in the format of webserver.log....
2
votes
1
answer
286
views
What is the most efficient way to unpack a .tar.zstd download in Python?
I am using the zstandard library to pack and unpack. I am essentially trying to recreate the following bash in Python:
curl -O - http://download.com/file.tar.zst | zstd -d - | tar xf -
I found this ...
1
vote
0
answers
263
views
"Decompressing large Zstandard file in Rust results in 'Frame requires too much memory for decoding' error"
I'm using the zstd-rs crate to decompress a large Zstandard file (~30GB) in Rust. My code opens the file, reads it into a buffer, and then uses the copy_decode function from the zstd::stream module to ...
1
vote
1
answer
755
views
ZSTD compression with tar and --ultra 22 on Mac OSX and homebrew (with progress bar)
I want to be able to compress a file or folder, and see the progress of said input in the terminal. This is my current progress so far, but I have two issues with it:
This should be possible without ...
0
votes
0
answers
124
views
How can I calculate ZSTD_compressed file size in C?
void compresszstd(const char *input,const char *output) {
//
size_t const fBuff_size = (size_t)fileSize; //input file size
void *const fBuff = malloc(fBuff_size); //input file buff
size_t ...
1
vote
0
answers
119
views
Unknown frame descriptor while reading zst file on hive?
I have a file a.csv.zst and i ve created textfile table.
When i select on hive i see first 100 rows without an error but when i select count(*) it gives an error unknown frame descriptor
I can ...
0
votes
2
answers
471
views
How to read_csv a zstd-compressed file using python-polars
In contrast to pandas, polars doesn't natively support reading zstd compressed csv files.
How can I get polars to read a csv compressed file, for example using xopen?
I've tried this:
from xopen ...
1
vote
1
answer
102
views
"No enum constant org.apache.orc.CompressionKind.ZSTD" When Insert Data to ORC Compress ZSTD Table
I have created a table in hive 3.1.3 as below;
Create external table test_tez_orc_zstd
(
Id bigint
)stored as orc
Tblproperties(orc.compress=zstd)
Location '...'
It is created, and then I wanted to ...
0
votes
1
answer
74
views
how can I read csv.zstd files on hive 3 on hadoop 3.x.y?
I have files .csv.zstd
it is like this :
col1|col2
1,2
so i create table on hive :
create external table test_zstd (
col1 bigint,
col2 bigint
) row format delimited fields terminated by '|'
stored ...
0
votes
0
answers
32
views
Zstd decompress with Armv7 32bit performance is too poor
I am using ZStd with armv7 32bit platform to decompress, but 128M data would require about 2 minutes to finish. But on armv8 64bit, it is only require about 2seconds, so what result armv7 32bit ...
0
votes
0
answers
128
views
WinUI 3, Desktop application error during "Package and Publish"
I'm working on a WinUI 3 desktop application. During the option of "Package and Publish", I get an error of which I've attached the image (as the error details are too long, I thought of ...
0
votes
0
answers
300
views
Issue reinstalling package Phyloseq after updating R to v4.3.0: "fatal error: zstd.h: No such file or directory"
I just installed new Ubuntu updates, and apparently it contained a new version of R (4.3.0), which I didn't realize. I ended up having to reinstall some packages, but there is one that I am having ...
0
votes
0
answers
148
views
How do I decompress a zstd compressed payload in scala
Hi I have a payload that i need to decompress in scala
val uncompressed = "" //payload to be decompressed
val uncomprBytes = uncompressed.getBytes()
val size = uncomprBytes.length
val ...
0
votes
0
answers
253
views
ROS topic communication Buffer overrun
I intended to use the zstd compression api to compress the data sent from the client side of ROS topic communication to the server side .
When I wrote the code and ran the code after compiling, the ...
0
votes
1
answer
192
views
Error in AWS Lambda when using pyzstd module
I'm trying to use module pyzstd on Lambda service, but I get the following error:
pyzstd module: Neither C implementation nor CFFI implementation can be imported. If pyzstd module is dynamically ...
0
votes
1
answer
270
views
duckdb getting 'Error: IO Error: Frame requires too much memory for decoding' error when reading very large files
I'm trying to read a month's worth of Reddit data from Pushshift. These files are around 30gigs, compressed (zst format) of json data. I am trying to convert these files to parquet.
Here is the code:
...
-1
votes
1
answer
147
views
ZSTD throws an exception if src is less than or equal to 4 bytes
can you help me to solve a problem with ZSTD library compression.
The thing is, when I tried to compress less than or equal to 4 bytes, the ZSTD_compress function returned me an exception
but if i ...
1
vote
1
answer
287
views
How to add zstd and lz4 algorithm to tar on MacOS?
So my tar version on MacOS doesn't seem to support the zstd or lz4 algorithms, and I am unsure how to update tar or add support for it. My Ubuntu's tar version seems to have support for zstd.
0
votes
1
answer
564
views
How to compress and decompress byte array with zstd algorithm in java
I'm trying to use ZSTD algorithm to compress and decompress byte array.
I read the ZSTD document that will give more details about implementation, but I can't still implement sufficient method to ...
4
votes
1
answer
508
views
Stream a .zst compressed file line by line
I am trying to sift through a big database that is compressed in a .zst. I am aware that I can simply just decompress it and then work on the resulting file, but that uses up a lot of space on my ssd ...
2
votes
0
answers
152
views
Only one package in the dependency graph may specify the same links value
I have two dependencies, actix-web and sled. Both rely on zstd, however they rely on zstd-sys v2.0.4 and zstd-sys 1.6.1 respectively. Is there a way I can "link" actix-web to one version of ...
2
votes
0
answers
104
views
Getting message "Compression negotiation not requested by client" in db connection logs
I am trying to use network compression feature of mongoDb using golang,
I am trying to use Zstd compression, and I am using the following method at the time of db connection,
opts := options.Client()....
0
votes
1
answer
653
views
How do I transfer small files quickly over the network with zstd?
As the question states, I want to backup many small files and send them via ssh to a destination. Does rsync speed things up significantly vs tar?
0
votes
1
answer
157
views
Zstandart(zstd) - how use fucntion ZSTD_CCtx_setParameter
Tried to figure out how to restore values when using the ZSTD_CCtx setParameter function.
I understood only on the example of setting the compression level, like this for example:
cctx = ...
1
vote
1
answer
108
views
Zstandart(zstd) - checking return
Can you please explain how the "ZSTD_isError" function works?
I check the return code from ZSTD_isError to be zero:
ZSTD_cParameter cParam;
ZSTD_bounds ZSTD_bounds_ = ZSTD_cParam_getBounds(...
0
votes
1
answer
251
views
Zstandart - compress with function reuse context
I'm trying to figure out when to use the Zstandart function, which, as written, reuses the context.
Please explain what you mean in this case:
`ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void);
...
0
votes
1
answer
549
views
How to correctly compress a vector using ZSTD simple API?
I'm new to C++ and I wanted to compress a vector via ZSTD compression library. I used ZSTD simple API ZSTD_compress and ZSTD_decompress in the same way as the example. But I found a wired issue that ...
3
votes
1
answer
957
views
How does zstandard compression behave when passed a size hint, instead of compressing a stream?
The zstd compressor can operate in streaming mode, or the total size to be compressed can be given in advance (for example, with the size parameter in this Python binding
How does the library behave ...
0
votes
0
answers
72
views
Presto hive connector reads .zst file
for the following query, both Hive and Spark-SQL work fine,
but the result returned by Presto (hive connector ) has wrong encoding/decoding.
Wonders how should I set the hive connector, or presto ...
0
votes
0
answers
333
views
Spark driver fails when running zstd compression on multiple threads in databricks single node cluster
I'm using a single node databricks cluster with the following configuration:
I need to write partitions from different dataframes in zstd compressed .csv files and upload them to DBFS. What I would ...
0
votes
2
answers
354
views
zstandard library linker error using conan
I think I'm missing something very simple but this is the setup I have:
I'm using conan to install zstandard:
[requires]
...
zstd/1.5.1
with following config:
[settings]
os=Linux
arch=x86_64
...
-1
votes
1
answer
89
views
Error when creating zstd file with ProcessBuilder on Java
I'm trying to use ProcessBuilder in order to create a tar.zst file that compresses a folder. However, I'm getting the following error after running this code:
try {
ProcessBuilder pb = new ...
1
vote
1
answer
118
views
ZSTD equivalent of ZLIB deflateInit2 method
In a c++ project I have a function using deflateInit2 method of ZLIB.
ZEXTERN int ZEXPORT deflateInit2 OF((z_streamp strm,
int level,
...
1
vote
1
answer
905
views
cannot enable redis-zstd in php:8.0.2-fpm-buster
In a container based on "php:8.0.2-fpm-buster" image, I run
pecl install zstd
echo "extension=zstd.so" > /path/to/php.ini
then run
pecl install redis
enable zstd compression ...
3
votes
1
answer
1k
views
How to change zstd level for qcow2 image?
There is an option compression_type=zstd to enable zstd compression for qcow2 according to wiki.
But it always uses zstd level 3, how could I do to compress it with level 19? This image is read-only ...
0
votes
0
answers
491
views
Streaming decompression of data in a vector
I need to do the following procedure.
Compress an input text into an array.
Split the compressed output into multiple pieces with approx. the same length and store them in a vector.
Decompress using ...
1
vote
0
answers
386
views
Compression without dictionary
I have been testing the various compression algorithms with parquet files, and have settled on Zstd.
Now as far as I understand Zstd uses adaptive dictionary unless one is explicitly specified, thus ...
2
votes
0
answers
364
views
Visual Studio lnk2001 error for boost iostreams::zstd
I am trying to build OpenMVS with Visual Studio 2017. I have installed boost 1.74, which is found as well. But when I try to build, I get the following error:
Error LNK2001 unresolved external ...
3
votes
1
answer
3k
views
How do I create a dictionary object for passing to Zstd.compress?
I am using Zstd compression in Java for compressing a large JSON payload. I am using methods from the zstd-jni library for Java. I create a byte array out of the JSON string and use this method.
...