Questions tagged [zstd]

Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
11 views

how to decompress zstd files with unknown original file size with zstd-jni

We have a use case where we need to retrieve multiple zstd files and decompress them and we don't know the size of the original files. Zstd-jni requires us to provide the size of the decompressed file ...
munna's user avatar
  • 25
0 votes
0 answers
13 views

Build android shared object from makefile

I want to include Facebook's zstd into my Unity project targetting Android. The contributors made it easily to build .dll for .Net platform. But I can't make it work on the Android build, it only work ...
Khang Dinh Hoang's user avatar
1 vote
1 answer
149 views

How to decode and read a zstd file in Rust?

I'm looking for some advice on how to decode and read a zstd file and I'm feeling a bit lost since it is my first big project since I started learning Rust. I am using Rust for this project since it ...
TrkDgnr's user avatar
  • 127
0 votes
0 answers
43 views

arch linux install error - zstd: symbol lookup error: zstd: undefined symbol: POOL_joinJobs

After update to latest version gnome 45 and latest kernel, I'm encountering an error when trying to install applications on my Arch Linux system ( using pacman, yay, paru or makepkg ). The error ...
Pamungkas Jayuda's user avatar
0 votes
1 answer
180 views

How to set "zstd" compression level in AWS Glue job?

Background "zstd" compression codec has 22 compression levels. I read this Uber blog. Regarding compressing time and file size, I verified using df.to_parquet with our data and got same ...
Hongbo Miao's user avatar
  • 46.6k
0 votes
1 answer
70 views

I'm trying to open a zstd file in Python

I'm trying to open a zstd file in Python. I downloaded the file from https://the-eye.eu/redarcs/ archive. This archive provides Reddit posts and comments. When I download data from this site and open ...
dhkim's user avatar
  • 1
2 votes
3 answers
295 views

Pandas DataFrame.write_parquet() and setting the Zstd compression level

I am writing out a compressed Parquet file from DataFrame as following: result_df.to_parquet("my-data.parquet", compression="zstd") How can I instruct Pandas on the compression ...
Mikko Ohtamaa's user avatar
0 votes
0 answers
36 views

trino gives zstd invalid magic prefix?

I ve kerberized hive and hdfs Trino version 422 Hadoop 3.3.4 I ve zst files on hdfs and when i try to query that table include zst files it gives this error Trino gives error invalid magic prefix ...
CompEng's user avatar
  • 7,201
0 votes
2 answers
55 views

How to decompress the compressed response In Jmeter Performance Testing tool

In JMeter Performance Testing tool I get the response in a compressed format so I need to uncompress the entire response using the zStd in the postprocessor. I have download the jar file zstd-jni-1.5....
Nivethitha SK's user avatar
1 vote
0 answers
789 views

What exactly does each ZSTD compression level do?

The ZSTD compression library by Facebook offers over 19 compression levels, but sometimes some seem to have more effect on overall compression efficiency and compression time than others. What exactly ...
Wallace Long's user avatar
0 votes
1 answer
182 views

Read Zstandard compressed Parquet in Spark 2.4.7 Hadoop 2.7.7

We have a limitation on a platform we are using which has Spark 2.4.7 and underneath Hadoop 2.7.7 libraries present. We have some data present on s3 which is in zstandard parquet format. Is there a ...
Nikhil Pandit's user avatar
2 votes
0 answers
123 views

zstd: can't stat web.log.* : No such file or directory -- ignored

I can't seem to find help on using zstdgrep correctly. I thought it works like ordinary grep, but I am not getting what I want. I have several subfolders that have files in the format of webserver.log....
Tomáš Zato's user avatar
  • 50.8k
2 votes
1 answer
286 views

What is the most efficient way to unpack a .tar.zstd download in Python?

I am using the zstandard library to pack and unpack. I am essentially trying to recreate the following bash in Python: curl -O - http://download.com/file.tar.zst | zstd -d - | tar xf - I found this ...
drowsily.throbbing872's user avatar
1 vote
0 answers
263 views

"Decompressing large Zstandard file in Rust results in 'Frame requires too much memory for decoding' error"

I'm using the zstd-rs crate to decompress a large Zstandard file (~30GB) in Rust. My code opens the file, reads it into a buffer, and then uses the copy_decode function from the zstd::stream module to ...
weiserhase's user avatar
1 vote
1 answer
755 views

ZSTD compression with tar and --ultra 22 on Mac OSX and homebrew (with progress bar)

I want to be able to compress a file or folder, and see the progress of said input in the terminal. This is my current progress so far, but I have two issues with it: This should be possible without ...
OpenGears's user avatar
  • 112
0 votes
0 answers
124 views

How can I calculate ZSTD_compressed file size in C?

void compresszstd(const char *input,const char *output) { // size_t const fBuff_size = (size_t)fileSize; //input file size void *const fBuff = malloc(fBuff_size); //input file buff size_t ...
Jonah KIM's user avatar
1 vote
0 answers
119 views

Unknown frame descriptor while reading zst file on hive?

I have a file a.csv.zst and i ve created textfile table. When i select on hive i see first 100 rows without an error but when i select count(*) it gives an error unknown frame descriptor I can ...
CompEng's user avatar
  • 7,201
0 votes
2 answers
471 views

How to read_csv a zstd-compressed file using python-polars

In contrast to pandas, polars doesn't natively support reading zstd compressed csv files. How can I get polars to read a csv compressed file, for example using xopen? I've tried this: from xopen ...
Cornelius Roemer's user avatar
1 vote
1 answer
102 views

"No enum constant org.apache.orc.CompressionKind.ZSTD" When Insert Data to ORC Compress ZSTD Table

I have created a table in hive 3.1.3 as below; Create external table test_tez_orc_zstd ( Id bigint )stored as orc Tblproperties(orc.compress=zstd) Location '...' It is created, and then I wanted to ...
CompEng's user avatar
  • 7,201
0 votes
1 answer
74 views

how can I read csv.zstd files on hive 3 on hadoop 3.x.y?

I have files .csv.zstd it is like this : col1|col2 1,2 so i create table on hive : create external table test_zstd ( col1 bigint, col2 bigint ) row format delimited fields terminated by '|' stored ...
CompEng's user avatar
  • 7,201
0 votes
0 answers
32 views

Zstd decompress with Armv7 32bit performance is too poor

I am using ZStd with armv7 32bit platform to decompress, but 128M data would require about 2 minutes to finish. But on armv8 64bit, it is only require about 2seconds, so what result armv7 32bit ...
lico's user avatar
  • 1
0 votes
0 answers
128 views

WinUI 3, Desktop application error during "Package and Publish"

I'm working on a WinUI 3 desktop application. During the option of "Package and Publish", I get an error of which I've attached the image (as the error details are too long, I thought of ...
OmkarD's user avatar
  • 1
0 votes
0 answers
300 views

Issue reinstalling package Phyloseq after updating R to v4.3.0: "fatal error: zstd.h: No such file or directory"

I just installed new Ubuntu updates, and apparently it contained a new version of R (4.3.0), which I didn't realize. I ended up having to reinstall some packages, but there is one that I am having ...
matheusb's user avatar
0 votes
0 answers
148 views

How do I decompress a zstd compressed payload in scala

Hi I have a payload that i need to decompress in scala val uncompressed = "" //payload to be decompressed val uncomprBytes = uncompressed.getBytes() val size = uncomprBytes.length val ...
Ankit Chawrai's user avatar
0 votes
0 answers
253 views

ROS topic communication Buffer overrun

I intended to use the zstd compression api to compress the data sent from the client side of ROS topic communication to the server side . When I wrote the code and ran the code after compiling, the ...
zwq's user avatar
  • 5
0 votes
1 answer
192 views

Error in AWS Lambda when using pyzstd module

I'm trying to use module pyzstd on Lambda service, but I get the following error: pyzstd module: Neither C implementation nor CFFI implementation can be imported. If pyzstd module is dynamically ...
datahack's user avatar
  • 625
0 votes
1 answer
270 views

duckdb getting 'Error: IO Error: Frame requires too much memory for decoding' error when reading very large files

I'm trying to read a month's worth of Reddit data from Pushshift. These files are around 30gigs, compressed (zst format) of json data. I am trying to convert these files to parquet. Here is the code: ...
Shahbaz's user avatar
  • 10.4k
-1 votes
1 answer
147 views

ZSTD throws an exception if src is less than or equal to 4 bytes

can you help me to solve a problem with ZSTD library compression. The thing is, when I tried to compress less than or equal to 4 bytes, the ZSTD_compress function returned me an exception but if i ...
Pavel12398's user avatar
1 vote
1 answer
287 views

How to add zstd and lz4 algorithm to tar on MacOS?

So my tar version on MacOS doesn't seem to support the zstd or lz4 algorithms, and I am unsure how to update tar or add support for it. My Ubuntu's tar version seems to have support for zstd.
Dr. Light's user avatar
  • 386
0 votes
1 answer
564 views

How to compress and decompress byte array with zstd algorithm in java

I'm trying to use ZSTD algorithm to compress and decompress byte array. I read the ZSTD document that will give more details about implementation, but I can't still implement sufficient method to ...
pan ta's user avatar
  • 3
4 votes
1 answer
508 views

Stream a .zst compressed file line by line

I am trying to sift through a big database that is compressed in a .zst. I am aware that I can simply just decompress it and then work on the resulting file, but that uses up a lot of space on my ssd ...
SimonUnderwood's user avatar
2 votes
0 answers
152 views

Only one package in the dependency graph may specify the same links value

I have two dependencies, actix-web and sled. Both rely on zstd, however they rely on zstd-sys v2.0.4 and zstd-sys 1.6.1 respectively. Is there a way I can "link" actix-web to one version of ...
blurt.washday's user avatar
2 votes
0 answers
104 views

Getting message "Compression negotiation not requested by client" in db connection logs

I am trying to use network compression feature of mongoDb using golang, I am trying to use Zstd compression, and I am using the following method at the time of db connection, opts := options.Client()....
sahil garg's user avatar
0 votes
1 answer
653 views

How do I transfer small files quickly over the network with zstd?

As the question states, I want to backup many small files and send them via ssh to a destination. Does rsync speed things up significantly vs tar?
Bob's user avatar
  • 729
0 votes
1 answer
157 views

Zstandart(zstd) - how use fucntion ZSTD_CCtx_setParameter

Tried to figure out how to restore values when using the ZSTD_CCtx setParameter function. I understood only on the example of setting the compression level, like this for example: cctx = ...
Serg_'s user avatar
  • 77
1 vote
1 answer
108 views

Zstandart(zstd) - checking return

Can you please explain how the "ZSTD_isError" function works? I check the return code from ZSTD_isError to be zero: ZSTD_cParameter cParam; ZSTD_bounds ZSTD_bounds_ = ZSTD_cParam_getBounds(...
Serg_'s user avatar
  • 77
0 votes
1 answer
251 views

Zstandart - compress with function reuse context

I'm trying to figure out when to use the Zstandart function, which, as written, reuses the context. Please explain what you mean in this case: `ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void); ...
Serg_'s user avatar
  • 77
0 votes
1 answer
549 views

How to correctly compress a vector using ZSTD simple API?

I'm new to C++ and I wanted to compress a vector via ZSTD compression library. I used ZSTD simple API ZSTD_compress and ZSTD_decompress in the same way as the example. But I found a wired issue that ...
zwq's user avatar
  • 5
3 votes
1 answer
957 views

How does zstandard compression behave when passed a size hint, instead of compressing a stream?

The zstd compressor can operate in streaming mode, or the total size to be compressed can be given in advance (for example, with the size parameter in this Python binding How does the library behave ...
joeforker's user avatar
  • 40.7k
0 votes
0 answers
72 views

Presto hive connector reads .zst file

for the following query, both Hive and Spark-SQL work fine, but the result returned by Presto (hive connector ) has wrong encoding/decoding. Wonders how should I set the hive connector, or presto ...
AppleCEO's user avatar
0 votes
0 answers
333 views

Spark driver fails when running zstd compression on multiple threads in databricks single node cluster

I'm using a single node databricks cluster with the following configuration: I need to write partitions from different dataframes in zstd compressed .csv files and upload them to DBFS. What I would ...
user14681827's user avatar
0 votes
2 answers
354 views

zstandard library linker error using conan

I think I'm missing something very simple but this is the setup I have: I'm using conan to install zstandard: [requires] ... zstd/1.5.1 with following config: [settings] os=Linux arch=x86_64 ...
MoneyBall's user avatar
  • 2,363
-1 votes
1 answer
89 views

Error when creating zstd file with ProcessBuilder on Java

I'm trying to use ProcessBuilder in order to create a tar.zst file that compresses a folder. However, I'm getting the following error after running this code: try { ProcessBuilder pb = new ...
avivblo's user avatar
  • 144
1 vote
1 answer
118 views

ZSTD equivalent of ZLIB deflateInit2 method

In a c++ project I have a function using deflateInit2 method of ZLIB. ZEXTERN int ZEXPORT deflateInit2 OF((z_streamp strm, int level, ...
Vahag Chakhoyan's user avatar
1 vote
1 answer
905 views

cannot enable redis-zstd in php:8.0.2-fpm-buster

In a container based on "php:8.0.2-fpm-buster" image, I run pecl install zstd echo "extension=zstd.so" > /path/to/php.ini then run pecl install redis enable zstd compression ...
Steve's user avatar
  • 41
3 votes
1 answer
1k views

How to change zstd level for qcow2 image?

There is an option compression_type=zstd to enable zstd compression for qcow2 according to wiki. But it always uses zstd level 3, how could I do to compress it with level 19? This image is read-only ...
kkocdko's user avatar
  • 300
0 votes
0 answers
491 views

Streaming decompression of data in a vector

I need to do the following procedure. Compress an input text into an array. Split the compressed output into multiple pieces with approx. the same length and store them in a vector. Decompress using ...
Alberto Tiraboschi's user avatar
1 vote
0 answers
386 views

Compression without dictionary

I have been testing the various compression algorithms with parquet files, and have settled on Zstd. Now as far as I understand Zstd uses adaptive dictionary unless one is explicitly specified, thus ...
SomewhatInterested's user avatar
2 votes
0 answers
364 views

Visual Studio lnk2001 error for boost iostreams::zstd

I am trying to build OpenMVS with Visual Studio 2017. I have installed boost 1.74, which is found as well. But when I try to build, I get the following error: Error LNK2001 unresolved external ...
brandbenni's user avatar
3 votes
1 answer
3k views

How do I create a dictionary object for passing to Zstd.compress?

I am using Zstd compression in Java for compressing a large JSON payload. I am using methods from the zstd-jni library for Java. I create a byte array out of the JSON string and use this method. ...
A. Ganesh's user avatar