Questions tagged [rocksdb]

RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads.

Filter by
Sorted by
Tagged with
0 votes
0 answers
39 views

The RocksDB state store of Spark Executors takes up a lot of space in Kubernetes

I am currently testing the benefits of using RocksDB as a State store for a (Scala) Spark application I am maintaining. It is running on a Kubernetes cluster. (https://spark.apache.org/docs/3.2.0/...
Manfred Von Karma's user avatar
0 votes
0 answers
44 views

MyRocks error in ubuntu 18.04.6. cant' build with cmake

i want to use MyRocks(MySQL + RocksDB). so, i have follow this MyRocks official manual(http://myrocks.io/docs/getting-started/). but i faced something wrong situation in cmake(RocksDB inherited in ...
Hun Shim's user avatar
1 vote
2 answers
50 views

how does RocksDB cache writes?

I have a few column families which have references to each other -- to construct a "full object" I need to join data across them. The upstream providing me with data often provides updates ...
Dan's user avatar
  • 7,317
0 votes
0 answers
20 views

Rocksdb change compaction style, the exsisted data will be correct?

If I open Rocksdb with leveled compaction style and insert some data. And then, I reopen Rocksdb with universal compaction style. The rocksdb can work correct or not? https://github.com/facebook/...
zncleon's user avatar
0 votes
0 answers
14 views

Build TiKV's fork of rocksdb in arch linux

So i am trying to compile titan, which is a pulgin for rocksdb that enables K/V separate feature. However, it requires TiKV's fork of rocksdb rather than the original one of Facebook's. I am doing ...
DUmmY's user avatar
  • 1
0 votes
0 answers
24 views

Is there any way to diff two snapshots in rocksdb?

RocksDB provides an API DB::GetSnapshot() to get a snapshot. A snapshot is actually an LSN, all keys with LSN less or equal than this LSN is visible, while all keys with LSN greater than this LSN is ...
calvin's user avatar
  • 2,277
1 vote
1 answer
31 views

c++ memory order in RocksDB source code void WriteThread::SetState(Writer* w, uint8_t new_state)

I've been reading through the source code of rocksdb lately, but I've run into a problem I can't figure out and would like to ask for advice void WriteThread::SetState(Writer* w, uint8_t new_state) { ...
Li Tank's user avatar
  • 11
0 votes
0 answers
19 views

rocksdb v3.4 compilation failure, ::abs has not been declared

Try to compile rocksdb v3.4 with follow instructions on centOS7: git checkout v3.4 make Error: ... g++ -g -Wall -Werror -Wsign-compare -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX ...
minglyu's user avatar
  • 3,018
1 vote
1 answer
80 views

How can I get `fs::remove_dir_all` to remove a specific file upon test execution?

I am working on a project using Rust and RocksDB that creates a db log when certain tests are ran. Each time the tests are ran, the path gets created again and the information of both is saved. ...
andrewvious's user avatar
2 votes
1 answer
91 views

RocksDB data loss

I have a stateful Kafka Streams application with logging disabled. The reason I had to disable it is because the aggregated state for some records can get very big, and sending an update to the ...
AmsterdamLuis's user avatar
0 votes
2 answers
82 views

RocksDB. Backups take up more space than the database itself

I encountered a problem in which the backup takes up more space than the database itself. 233G rocks database size 651G rocks backup size Both the database engine and the backup engine are ...
Yakhya's user avatar
  • 3
1 vote
0 answers
89 views

Flink + rocksdb taking a very long time to restore a large s3 checkpoint

I am running a low-parallellism (4 slots) job whose checkpoints can get very large. In the example I'm going to present the checkpoint was 142 GB, saved in S3 and it took 40 minutes to restore from ...
Raúl García's user avatar
0 votes
0 answers
12 views

How can I get an approximate quantile from rockdb at a cheap cost with currently available apis?

I have a rocksdb instance, I am sure no more writes are performed on it, and I am going to open it in read only mode. Now I want to get a key of approximate x-quantile. For example, if I have keys ...
calvin's user avatar
  • 2,277
-1 votes
1 answer
73 views

Address sanitizer is not working with application which is using rocksdb

I am using rocksdb 4.13 version in a c++ project. To find memory issue I wanted to use libasan. But I could not test my application as it exits before running any test. It mostly points to rocksdb::DB:...
Halder's user avatar
  • 45
0 votes
0 answers
215 views

"ERROR: Failed building wheel for python-rocksdb" when building Docker Container

For my project, I run a Docker Container which needs to have python-rocksdb installed. This usually worked for me, but since 1 or 2 weeks, the building process fails with "... Cython.Compiler....
David's user avatar
  • 21
0 votes
0 answers
21 views

concurrent update on different documents in same collection with index

In the documentation here, under locks section, I see "However, if two concurrent transactions attempt to modify the same document or index entry, there will be a write-write conflict" ...
Prashanth's user avatar
0 votes
0 answers
26 views

Rocksdb cache store in infinispan - huge size of .log file

I am using embedded infinispan cache 14 and rocksdb cache store for persistence. After couple of hours, I observed .log file in cache store if getting huge in size. I have searched on it and used ...
Satish Shelake's user avatar
0 votes
0 answers
135 views

State store vs Ktable in Kafka Streams

I'm new to Kafka and Kafka Streams. While I have gone through the concepts of Kafka and Kafka Streams and feel confident conceptually, there is one thing that's confusing me in and out. Yes, it's the ...
eureka19's user avatar
  • 2,850
0 votes
1 answer
139 views

RocksDB compaction requires too much free space

We have a write intensive applications, which write many thousand of logs per hour to a MariaDB database in a RocksDB table. We want to manage the situations where the disk space is nearly filled up. ...
AshkanVZ's user avatar
  • 691
0 votes
0 answers
90 views

How to configure RocksDB matrics using kafka streams 3.2.0 in java?

I need to use metrics mentioned in https://cwiki.apache.org/confluence/display/KAFKA/KIP-471%3A+Expose+RocksDB+Metrics+in+Kafka+Stream in java. Is there any sample document available for printing the ...
shalaka's user avatar
0 votes
0 answers
141 views

RocksDB Converting SST Files to Human-Readable Text

I possess a folder named myfolder which contains the following types of files: sst files log files CURRENT file IDENTITY file LOCK file LOG files MANIFEST files OPTIONS files The output of the ...
Toami's user avatar
  • 3
0 votes
1 answer
70 views

How does key rebalancing in state store work in Kafka Streams with scaling out of partitions?

Let's suppose a Kafka streams that operates stateful has an input topic with 16 partitions and the number of instances (or tasks) is also 16. I understand that the default partitioner of the Kafka ...
tae's user avatar
  • 51
3 votes
1 answer
181 views

Rocksdb compaction is very slow

I am doing rocksdb compaction using compactfiles api. I checked the compaction stats, the read and write is very slow.. I am seeing the read write and write rate to be around 10MBps. which is very low....
suresh's user avatar
  • 4,124
0 votes
0 answers
86 views

ClickHouse EmbeddedRocksDB full scan on range query

I try to use EmbeddedRocksDB engine in ClickHouse and I can't understand why EmbeddedRocksDB does full scan on range query. To reproduce CREATE TABLE sample_table ( key String, value String ) ...
Стас Насибуллин's user avatar
1 vote
1 answer
101 views

Rocksdb seems to load full Ktable statestore in memory

In my kafka stream topology, I join a ktable with a kstream and it keeps crashing because of the memory: the join DSL creates one statestore for the Ktable, and when this statestore is small, the used ...
Devidb's user avatar
  • 67
0 votes
1 answer
93 views

Not able to recover states after restart apche flink with Spring boot

Configuration configuration = new Configuration(); configuration.setBoolean("state.backend.local-recovery", true); configuration.setString("state.checkpoints.dir&...
saif's user avatar
  • 1
0 votes
1 answer
190 views

Kafka Stream Join memory keeps growing

I have some memory issues with a Kafka Stream that joins 2 Streams (one stream enrichs the other one). When the state store grows, the memory used (heap + off heap) grows too, and my application ...
Devidb's user avatar
  • 67
0 votes
1 answer
89 views

Compile RocksDB to Native in Quarkus

n my application, I am using RocksDB. I would compile this application natively. However, I have this error: {"refId":1,"exceptionType":"java.lang.UnsatisfiedLinkError",&...
Diogo Dias's user avatar
0 votes
0 answers
83 views

Rocks DB benefits when using Faust

According to faust-streaming it is recommended to use Rocks DB for the fast recovery of the faust tables. Say there is one machine running a consumer with the faust table and Rocks DB. Rocks DB ...
Plesiosaurus's user avatar
0 votes
2 answers
290 views

Kafka Streams unexpectedly high memory usage

I'm running a Kafka Streams application in Kubernetes that simply reads a topic straight into a GlobalKTable and then offers an HTTP API to get the key-value pairs. The memory usage of this pod is for ...
Grisu47's user avatar
  • 530
0 votes
1 answer
147 views

How to figure out what's the size of ListState?

In my Flink project, there are two topics that I join query on both of them: StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment ...
newbie5050's user avatar
0 votes
1 answer
94 views

Safe to delete zero-byte .log file in RocksDB to preventing error_if_wal_file_exists error in rocksdb

Is it safe to delete a zero-sized write ahead log (WAL) .log file in RocksDB? Is there a way to do this through the RocksDB API? Context I want to open a RocksDB database in read-only mode and set the ...
Manuel's user avatar
  • 6,471
0 votes
0 answers
53 views

Facing NPE when upgrading scala from 2.11 to 2.12 in flink

Upgrading to flink 1.13.5_2.12 from 1.13.5_2.11 and skipped state but facing NPE when RocksDb state is updating. Have a PriorityQueue which is ordered by Long, and when its updating state, an error ...
ashwin gupta's user avatar
0 votes
2 answers
88 views

Do memtables ever actually use trees over skip lists?

I keep seeing articles that mention how memtables are usually backed by a self balancing tree like AVL tree or red-black tree. Examples: https://www.linkedin.com/pulse/data-structures-powering-our-...
jmtt's user avatar
  • 51
0 votes
1 answer
58 views

Why does rocksdb need to perform directory-level fsync on files generated by compaction

Each sst file generated through compaction will perform the fsync operation of the file to ensure that the data is placed on the disk. Why do we need the fsync operation at the directory level? I ...
qing zhao's user avatar
0 votes
1 answer
526 views

RocksDB Metrics

I'm using RocksDB in my project with the SpringBoot Microservices. For monitoring & troubleshooting purposes, I want to export RocksDB metrics daily. I have exposed some metrics like Column Family,...
Luffy's user avatar
  • 186
0 votes
1 answer
124 views

under what circumstances CURRENT file will be missing under rocksdb database directory

At times, our application fails to start with error "CURRENT : does not exist ( create_if_missing is false )". I am sure, it has some thing to do with our app shutdown. we are doing a delete ...
Ravikumar Tulugu's user avatar
1 vote
1 answer
391 views

Flink RocksDB custom options factory config error disable block cache

I am running Flink 1.15.2 and am trying to define a custom options factory in RocksDB to disable the block cache. Following the example from this blog post: https://shopify.engineering/optimizing-...
Sean L's user avatar
  • 52
1 vote
0 answers
184 views

Problem in verifying the RocksDB StateStore

I wanted to test the RocksDB StateStore implementation and check if it really resists the OOM exception. Before I explain my issue.... I compiled the spark version 3.2.0, I'm running it on linux ...
user3551056's user avatar
1 vote
1 answer
231 views

RocksDB visualization tool

I would like to know if there is any basic RocksDB visualization tool. The only one I can see is the LDB tool which is a command line. Thanks in advance, Hak
hak's user avatar
  • 49
0 votes
0 answers
83 views

Memory is not reclaimed when close the rocksdb instance

our project use rocksdb with rocksdbjni that is nice and easy to use. but, some problems give me a headaches. when i close the rocksdb the momery in used doesn't properly reclaimed. It seems to ...
kospi's user avatar
  • 355
1 vote
2 answers
525 views

Using RocksDB instead of HashMap in Spring Boot project, what pitfalls should be avoided?

Background: There is a Spring Boot service that needs to synchronize some data from Elasticsearch to memory during startup. The processed data will be used by @Service A. The data needs to be ...
haitao cc's user avatar
0 votes
0 answers
71 views

Kafka stream failover

What are the best practices to migrate kafka stream applications to another secondary cluster for failover purposes? And any precautions to take? Also requirement is that application should continue ...
Amy's user avatar
  • 1
0 votes
1 answer
133 views

Best way to load a mmap dictionary in Python without deserializing

Context: I've got Python processes running on the same container and I want to be able to share a read-only key-value object between them. I'm aware I could use something like Redis to share that info,...
Damien's user avatar
  • 1,963
-1 votes
1 answer
103 views

Will rocksDB support nested keys for one value?

I am new to RocksDB. I am testing it on java spring boot(dependency: org.rocksdb.rocksdbjni). Will it support on storing nested keys for one value and retrieve a value using nested keys. Example: In ...
Praveen's user avatar
0 votes
0 answers
165 views

When does the Flink flush the data into disk using Rocksdb?

I am using state for processing the data using Flink. I used the rocksDB because the data size to be stored is relatively large compared to the memory size. I set up the rocksdb configuration and ...
sclee1's user avatar
  • 1,151
1 vote
1 answer
237 views

Does Rocksdb support any mechanism/API to export the saved records in text file like JSON or CSV?

I want to use RocksDB as a storage. I want to export the database to human redable format like Json or CSV. Does Rocksdb support that ? If yes, it also possible to import the data back in database ?
Halder's user avatar
  • 45
0 votes
0 answers
123 views

Issues installing python package on windows

I'm trying to install python-rocksdb on a windows 10 machine and I'm coming across some issues with the install. I have successfully installed rocksdb first using the vcpkg method. When running pip ...
Fonty's user avatar
  • 239
0 votes
0 answers
511 views

Build RocksDB static library using vcpkg and custom gcc on Windows

I am trying to create Kotlin/Native bindings for RocksDB. This requires that RocksDB is compiled using gcc and g++ version 9.2.0, because that is the only version that Kotlin/Native supports. If I ...
aSemy's user avatar
  • 5,962
0 votes
1 answer
42 views

Effectirve rows removal in RocksDB

I need to loop over all key-values in RocksDB in order to fill my POD collection. I don't need to store key-values after retrieval. What should I use - DeleteRange() after loop or Delete() within loop?...
ilya's user avatar
  • 1,113

1
2 3 4 5
10