Questions tagged [rocksdb]
RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads.
480
questions
0
votes
0
answers
39
views
The RocksDB state store of Spark Executors takes up a lot of space in Kubernetes
I am currently testing the benefits of using RocksDB as a State store for a (Scala) Spark application I am maintaining. It is running on a Kubernetes cluster.
(https://spark.apache.org/docs/3.2.0/...
0
votes
0
answers
44
views
MyRocks error in ubuntu 18.04.6. cant' build with cmake
i want to use MyRocks(MySQL + RocksDB). so, i have follow this MyRocks official manual(http://myrocks.io/docs/getting-started/). but i faced something wrong situation in cmake(RocksDB inherited in ...
1
vote
2
answers
50
views
how does RocksDB cache writes?
I have a few column families which have references to each other -- to construct a "full object" I need to join data across them. The upstream providing me with data often provides updates ...
0
votes
0
answers
20
views
Rocksdb change compaction style, the exsisted data will be correct?
If I open Rocksdb with leveled compaction style and insert some data. And then, I reopen Rocksdb with universal compaction style. The rocksdb can work correct or not?
https://github.com/facebook/...
0
votes
0
answers
14
views
Build TiKV's fork of rocksdb in arch linux
So i am trying to compile titan, which is a pulgin for rocksdb that enables K/V separate feature. However, it requires TiKV's fork of rocksdb rather than the original one of Facebook's.
I am doing ...
0
votes
0
answers
24
views
Is there any way to diff two snapshots in rocksdb?
RocksDB provides an API DB::GetSnapshot() to get a snapshot. A snapshot is actually an LSN, all keys with LSN less or equal than this LSN is visible, while all keys with LSN greater than this LSN is ...
1
vote
1
answer
31
views
c++ memory order in RocksDB source code void WriteThread::SetState(Writer* w, uint8_t new_state)
I've been reading through the source code of rocksdb lately, but I've run into a problem I can't figure out and would like to ask for advice
void WriteThread::SetState(Writer* w, uint8_t new_state) {
...
0
votes
0
answers
19
views
rocksdb v3.4 compilation failure, ::abs has not been declared
Try to compile rocksdb v3.4 with follow instructions on centOS7:
git checkout v3.4
make
Error:
...
g++ -g -Wall -Werror -Wsign-compare -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX ...
1
vote
1
answer
80
views
How can I get `fs::remove_dir_all` to remove a specific file upon test execution?
I am working on a project using Rust and RocksDB that creates a db log when certain tests are ran. Each time the tests are ran, the path gets created again and the information of both is saved. ...
2
votes
1
answer
91
views
RocksDB data loss
I have a stateful Kafka Streams application with logging disabled. The reason I had to disable it is because the aggregated state for some records can get very big, and sending an update to the ...
0
votes
2
answers
82
views
RocksDB. Backups take up more space than the database itself
I encountered a problem in which the backup takes up more space than the database itself.
233G rocks database size
651G rocks backup size
Both the database engine and the backup engine are ...
1
vote
0
answers
89
views
Flink + rocksdb taking a very long time to restore a large s3 checkpoint
I am running a low-parallellism (4 slots) job whose checkpoints can get very large.
In the example I'm going to present the checkpoint was 142 GB, saved in S3 and it took 40 minutes to restore from ...
0
votes
0
answers
12
views
How can I get an approximate quantile from rockdb at a cheap cost with currently available apis?
I have a rocksdb instance, I am sure no more writes are performed on it, and I am going to open it in read only mode.
Now I want to get a key of approximate x-quantile. For example, if I have keys ...
-1
votes
1
answer
73
views
Address sanitizer is not working with application which is using rocksdb
I am using rocksdb 4.13 version in a c++ project. To find memory issue I wanted to use libasan. But I could not test my application as it exits before running any test. It mostly points to rocksdb::DB:...
0
votes
0
answers
215
views
"ERROR: Failed building wheel for python-rocksdb" when building Docker Container
For my project, I run a Docker Container which needs to have python-rocksdb installed. This usually worked for me, but since 1 or 2 weeks, the building process fails with "... Cython.Compiler....
0
votes
0
answers
21
views
concurrent update on different documents in same collection with index
In the documentation here, under locks section, I see "However, if two concurrent transactions attempt to modify the same document or index entry, there will be a write-write conflict"
...
0
votes
0
answers
26
views
Rocksdb cache store in infinispan - huge size of .log file
I am using embedded infinispan cache 14 and rocksdb cache store for persistence. After couple of hours, I observed .log file in cache store if getting huge in size.
I have searched on it and used ...
0
votes
0
answers
135
views
State store vs Ktable in Kafka Streams
I'm new to Kafka and Kafka Streams. While I have gone through the concepts of Kafka and Kafka Streams and feel confident conceptually, there is one thing that's confusing me in and out. Yes, it's the ...
0
votes
1
answer
139
views
RocksDB compaction requires too much free space
We have a write intensive applications, which write many thousand of logs per hour to a MariaDB database in a RocksDB table. We want to manage the situations where the disk space is nearly filled up.
...
0
votes
0
answers
90
views
How to configure RocksDB matrics using kafka streams 3.2.0 in java?
I need to use metrics mentioned in https://cwiki.apache.org/confluence/display/KAFKA/KIP-471%3A+Expose+RocksDB+Metrics+in+Kafka+Stream in java.
Is there any sample document available for printing the ...
0
votes
0
answers
141
views
RocksDB Converting SST Files to Human-Readable Text
I possess a folder named myfolder which contains the following types of files:
sst files
log files
CURRENT file
IDENTITY file
LOCK file
LOG files
MANIFEST files
OPTIONS files
The output of the ...
0
votes
1
answer
70
views
How does key rebalancing in state store work in Kafka Streams with scaling out of partitions?
Let's suppose a Kafka streams that operates stateful has an input topic with 16 partitions and the number of instances (or tasks) is also 16. I understand that the default partitioner of the Kafka ...
3
votes
1
answer
181
views
Rocksdb compaction is very slow
I am doing rocksdb compaction using compactfiles api. I checked the compaction stats, the read and write is very slow.. I am seeing the read write and write rate to be around 10MBps. which is very low....
0
votes
0
answers
86
views
ClickHouse EmbeddedRocksDB full scan on range query
I try to use EmbeddedRocksDB engine in ClickHouse and I can't understand why EmbeddedRocksDB does full scan on range query.
To reproduce
CREATE TABLE sample_table
(
key String,
value String
) ...
1
vote
1
answer
101
views
Rocksdb seems to load full Ktable statestore in memory
In my kafka stream topology, I join a ktable with a kstream and it keeps crashing because of the memory: the join DSL creates one statestore for the Ktable, and when this statestore is small, the used ...
0
votes
1
answer
93
views
Not able to recover states after restart apche flink with Spring boot
Configuration configuration = new Configuration();
configuration.setBoolean("state.backend.local-recovery", true);
configuration.setString("state.checkpoints.dir&...
0
votes
1
answer
190
views
Kafka Stream Join memory keeps growing
I have some memory issues with a Kafka Stream that joins 2 Streams (one stream enrichs the other one). When the state store grows, the memory used (heap + off heap) grows too, and my application ...
0
votes
1
answer
89
views
Compile RocksDB to Native in Quarkus
n my application, I am using RocksDB. I would compile this application natively. However, I have this error:
{"refId":1,"exceptionType":"java.lang.UnsatisfiedLinkError",&...
0
votes
0
answers
83
views
Rocks DB benefits when using Faust
According to faust-streaming it is recommended to use Rocks DB for the fast recovery of the faust tables. Say there is one machine running a consumer with the faust table and Rocks DB. Rocks DB ...
0
votes
2
answers
290
views
Kafka Streams unexpectedly high memory usage
I'm running a Kafka Streams application in Kubernetes that simply reads a topic straight into a GlobalKTable and then offers an HTTP API to get the key-value pairs. The memory usage of this pod is for ...
0
votes
1
answer
147
views
How to figure out what's the size of ListState?
In my Flink project, there are two topics that I join query on both of them:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment ...
0
votes
1
answer
94
views
Safe to delete zero-byte .log file in RocksDB to preventing error_if_wal_file_exists error in rocksdb
Is it safe to delete a zero-sized write ahead log (WAL) .log file in RocksDB?
Is there a way to do this through the RocksDB API?
Context
I want to open a RocksDB database in read-only mode and set the ...
0
votes
0
answers
53
views
Facing NPE when upgrading scala from 2.11 to 2.12 in flink
Upgrading to flink 1.13.5_2.12 from 1.13.5_2.11 and skipped state but facing NPE when RocksDb state is updating.
Have a PriorityQueue which is ordered by Long, and when its updating state, an error ...
0
votes
2
answers
88
views
Do memtables ever actually use trees over skip lists?
I keep seeing articles that mention how memtables are usually backed by a self balancing tree like AVL tree or red-black tree.
Examples:
https://www.linkedin.com/pulse/data-structures-powering-our-...
0
votes
1
answer
58
views
Why does rocksdb need to perform directory-level fsync on files generated by compaction
Each sst file generated through compaction will perform the fsync operation of the file to ensure that the data is placed on the disk. Why do we need the fsync operation at the directory level?
I ...
0
votes
1
answer
526
views
RocksDB Metrics
I'm using RocksDB in my project with the SpringBoot Microservices. For monitoring & troubleshooting purposes, I want to export RocksDB metrics daily. I have exposed some metrics like Column Family,...
0
votes
1
answer
124
views
under what circumstances CURRENT file will be missing under rocksdb database directory
At times, our application fails to start with error "CURRENT : does not exist ( create_if_missing is false )". I am sure, it has some thing to do with our app shutdown. we are doing a delete ...
1
vote
1
answer
391
views
Flink RocksDB custom options factory config error disable block cache
I am running Flink 1.15.2 and am trying to define a custom options factory in RocksDB to disable the block cache.
Following the example from this blog post: https://shopify.engineering/optimizing-...
1
vote
0
answers
184
views
Problem in verifying the RocksDB StateStore
I wanted to test the RocksDB StateStore implementation and check if it really resists the OOM exception.
Before I explain my issue....
I compiled the spark version 3.2.0,
I'm running it on linux ...
1
vote
1
answer
231
views
RocksDB visualization tool
I would like to know if there is any basic RocksDB visualization tool. The only one I can see is the LDB tool which is a command line.
Thanks in advance,
Hak
0
votes
0
answers
83
views
Memory is not reclaimed when close the rocksdb instance
our project use rocksdb with rocksdbjni that is nice and easy to use.
but, some problems give me a headaches. when i close the rocksdb the momery in used doesn't properly reclaimed. It seems to ...
1
vote
2
answers
525
views
Using RocksDB instead of HashMap in Spring Boot project, what pitfalls should be avoided?
Background:
There is a Spring Boot service that needs to synchronize some data from Elasticsearch to memory during startup. The processed data will be used by @Service A.
The data needs to be ...
0
votes
0
answers
71
views
Kafka stream failover
What are the best practices to migrate kafka stream applications to another secondary cluster for failover purposes? And any precautions to take?
Also requirement is that application should continue ...
0
votes
1
answer
133
views
Best way to load a mmap dictionary in Python without deserializing
Context:
I've got Python processes running on the same container and I want to be able to share a read-only key-value object between them.
I'm aware I could use something like Redis to share that info,...
-1
votes
1
answer
103
views
Will rocksDB support nested keys for one value?
I am new to RocksDB. I am testing it on java spring boot(dependency: org.rocksdb.rocksdbjni). Will it support on storing nested keys for one value and retrieve a value using nested keys.
Example:
In ...
0
votes
0
answers
165
views
When does the Flink flush the data into disk using Rocksdb?
I am using state for processing the data using Flink.
I used the rocksDB because the data size to be stored is relatively large compared to the memory size.
I set up the rocksdb configuration and ...
1
vote
1
answer
237
views
Does Rocksdb support any mechanism/API to export the saved records in text file like JSON or CSV?
I want to use RocksDB as a storage. I want to export the database to human redable format like Json or CSV. Does Rocksdb support that ? If yes, it also possible to import the data back in database ?
0
votes
0
answers
123
views
Issues installing python package on windows
I'm trying to install python-rocksdb on a windows 10 machine and I'm coming across some issues with the install.
I have successfully installed rocksdb first using the vcpkg method.
When running pip ...
0
votes
0
answers
511
views
Build RocksDB static library using vcpkg and custom gcc on Windows
I am trying to create Kotlin/Native bindings for RocksDB. This requires that RocksDB is compiled using gcc and g++ version 9.2.0, because that is the only version that Kotlin/Native supports. If I ...
0
votes
1
answer
42
views
Effectirve rows removal in RocksDB
I need to loop over all key-values in RocksDB in order to fill my POD collection. I don't need to store key-values after retrieval. What should I use - DeleteRange() after loop or Delete() within loop?...