Questions tagged [hazelcast-jet]

Hazelcast Jet is a distributed computing platform for low-latency stream and batch processing. It benefits from integration with Hazelcast IMDG providing fast, scalable Big Data processing that’s lightweight and embeddable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Perform Left,Right and Inner Join in Hazelcast Jet in Stateless manner

I am trying to Join two datasoures using Hazelcast Jet but it seems to be done in Statefull form that it collect all data/grouping and then joins but can we join as it goes? Here below is my code: ...
user3458271's user avatar
0 votes
1 answer
41 views

Hazelcast Jet after rollingAggregate how to get each grouped values in one go only?

I am trying to to do the aggregation on the data but I am trying rollingAggregate but as it's streaming process it processes asyn but my need is I need to do sum of amount and group by id and country. ...
user3458271's user avatar
0 votes
0 answers
38 views

Hazelcast Jet not removing Jobs after completion?

I trying to join 2 data sources using Hazelcast Jet and once I start Hazelcast Jet instance I do not want to shutdown it as to start it every time when new job is submitted can be time consuming so I ...
user3458271's user avatar
0 votes
1 answer
25 views

Can Hazelcast jet be used for batch based workload?

We are working on a mainframe modernization program where we need to process 3-5 Billion records per day in 6 different cycles in 24 hours window. Records are stored in files in some specific format. ...
arpan paliwal's user avatar
0 votes
0 answers
14 views

what is purpose of JobConfig.addCustomClasspath() method in Hazelcastjet

I am trying to submit job to the hazelcast cluster and facing NoClassDefFoundError exception for the dependencies even though I have kept the dependencies in custom-lib directory and adding the jar ...
Swapnil's user avatar
  • 11
0 votes
0 answers
67 views

Hazelcast Jet multiple datasource join slow down machine and high memory usage

I have created a Spring Boot application in which we are joining multiple datasources using Hazelcast Jet. For small datasource it is working very good but as datasource records increases above 1 ...
user3458271's user avatar
0 votes
0 answers
26 views

Does Hazelcast Jet remoteMap sink make a putTransient call on the remote map?

I have 2 Hazelcast clusters both backed by same map Store. I am trying to get some data from one cluster Imap and push it to the map of the second cluster. What I want to know is if this makes a put ...
user12995979's user avatar
0 votes
0 answers
62 views

How to use Karaf with Hazelcast Jet in Java module?

It is little bit tricky to make understand but I will try. I am creating application in which multiple sources data is coming from different module which is deployed in karaf but when I am trying to ...
user3458271's user avatar
0 votes
1 answer
48 views

How does Hazelcast Jet assign task-to-CPU priority?

If I have the following code and let's say I'm running on 10 nodes of 32 cores each: IList<...> ds = ....; //large collection, eg 1e6 elements ds .map() //expensive computation .flatMap()//...
Jonathan Sylvester's user avatar
0 votes
1 answer
278 views

Hazelcast Jet Job is not able to access spring context

I am createing a spring boot application and in my application I am trying to use Hazelcast Jet in normal spring application it is working properly but while trying in spring boot application it is ...
user3458271's user avatar
1 vote
1 answer
161 views

Hazelcast Predicate SQL into map

I'm trying to build a method that deletes entries with attributes that are not null, but I keep failing in predicates and I don't know how to implement properly the predicate because hazelcast doesn't ...
killopolo's user avatar
0 votes
1 answer
181 views

How to read and write Identified Data Serializable in Hazelcast

i have a problem and the documentations and github examples doesn't provide a clear example about how to do it.... i have this class public class KeySerializable implements IdentifiedDataSerializable{ ...
killopolo's user avatar
0 votes
1 answer
188 views

Problem trying to configure an interface and connect to the cluster

Hello people and good day, i'm trying to build an interface so, the interface it will be an external client that i need to get access from my linux console and execute the commands, this is the ...
killopolo's user avatar
0 votes
2 answers
97 views

Can we read parquet file in Hazelcast jet?

I trying to read parquet file via Hazelcast for that I have written below code which is working fine, but do Hazelcast provide any in-build source to read parquet file? BatchSource<Object> ...
user3458271's user avatar
0 votes
0 answers
106 views

Aggregation and grouping using Hazelcast jet

I am trying to do grouping and aggregation using Hazelcast Jet but it is getting little bit slow as I have to loop through data twice one for creating groupingKey and after that aggregating all data ...
user3458271's user avatar
1 vote
0 answers
128 views

How to handle multiple requests to huge csv files?

I am working on on Java web application and in our application user can upload his csv files and than we allow them to create charts, table etc in our dashboard. But when user uploads huge csv files ...
user3458271's user avatar
0 votes
0 answers
107 views

Hazelcast Jet not allowing Tomcat to stop

I am using Hazelcast jet to some aggregation and grouping but after being idle for sometime and when I tried to stop my tomcat it is not allowing to stop my tomcat and I have restart my PC. Below is ...
user3458271's user avatar
1 vote
0 answers
46 views

How to change exception from ERROR to WARRNING in Hazelcast

How to change exception level from ERROR to WARNING in hazelcast. How to customise the hazelcast global exception handler? 2022-04-07 15:07:06,094 [ERROR] [hz.laughing_bhabha.partition-operation....
Imtiaz Ali's user avatar
0 votes
1 answer
189 views

Hazelcast getSql().execute() throws Unrecognized client message received with type: 0x210100

I am trying out Hazelcast client-server. So far, I Got up hazelcast server members Create a spring boot application and connected to hazelcast Inserted/updated/queried Imap in hazelcast instance Used ...
Maram Sreerama Reddy's user avatar
2 votes
2 answers
349 views

Hazelcast SQL interface slow performance HZ 4.2.2 vs HZ 5.0.2

Situation : We have a product with approx 30 attributes (String, Enum, Double) values We have iMap with indexes for all attributes IndexType.HASH for string value and IndexType.SORTED for double ...
Lukas Karasek's user avatar
1 vote
1 answer
60 views

How Hazecast BatchStage work with memory?

I am new to Hazelcast jet and I am trying to join to datas but I want to know how it will occupy memory and how much and what it best practice to do it? Here is my code: BatchStage<Map<String,...
user3458271's user avatar
0 votes
1 answer
96 views

Hazelcast Jet logs not working after configuration

I am new to Hazelcast and I am trying to change the logs location to file using logs4j but it seems to be not working. log file is getting created but there is no entry coming, please help how can I ...
user3458271's user avatar
0 votes
2 answers
43 views

Are Hazelcast Jet Reliable Topic Sinks idempotent? (Hazelcast fault-tolerance of a websocket source)

I cannot find this in the Hazelcast Jet 5.0 (or 4.x) documentation, so I hope someone can answer this here - can a reliable topic be used as an idempotent sink, for example to de-duplicate events ...
siddhadev's user avatar
  • 16.5k
0 votes
1 answer
96 views

Hazelcast multiple left joins not working for two it is working fine

I am new to Hazelcast Jet and I m tryin to join 4 datas but join is not working as accepted I am not sure what wrong I am doing as for two tables it is working fine. Please help My Code: ...
user3458271's user avatar
1 vote
1 answer
247 views

How to query embedded object field with sql from hazelcast?

I'm trying to get complex java object from hazelcast using sql approach. My data object contains embedded object like Person{ ... HumanName name; ... } HumanName { String firstName; } ...
Александр Селютин's user avatar
1 vote
1 answer
101 views

How to Sink output to the json file in Hazelcast Jet?

I am new to Hazelcast Jet and in my application I am using jdbc source, this source returning List of Map which I have to Sink in one of json file but when I use Sinks.json it create json directory ...
user3458271's user avatar
1 vote
1 answer
1k views

Java parallelStream java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker

I'm using Hazelcast Jet to do calculations over a large number of elements (~ 480 million). I'm starting using an IMap Sink containing the starting data. I'm filling the map using Collections2....
Nova Shadowflame's user avatar
1 vote
2 answers
262 views

How to use Connection Pooling with Hazelcast jet JDBC?

I am trying to use connection pooling with hazelcast jet jdbc but it is not allowing me to do it. I have Datasource bean from which I am fetching connection but it is not working. Here is my code: ...
user3458271's user avatar
0 votes
1 answer
101 views

Problem with data structures using Hazelcast and Hazelcast Jet

I'm having a few problems with jet and hazelcast, but my question is for logic purposes, I have a class which is going to bring all the data and when we start the client we got a queue and 2 maps, but ...
killopolo's user avatar
0 votes
1 answer
112 views

In Hazelcast jet how can we store IList to normal list as I have to sent it in Response?

I am new to Hazelcast jet and in my application on data I am doing some aggregation and getting data but I want to send that in rest response so how can I change it to normal list? public class ...
user3458271's user avatar
0 votes
1 answer
133 views

In Hazelcast jet how to Sink Map object to BatchSource?

I am new to Hazelcast jet and I am not getting how to Sink simple java.util.Map to BatchSource? I have tried below but not seems to be working. Map<String, Object> data =new HashMap<String, ...
user3458271's user avatar
0 votes
1 answer
171 views

Can't set hazelcast ManagedContext when using Jet.bootstrappedInstance()

I want to run hazelcast-jet embedded in my spring application (ie cluster is in process / NOT remote). I also want my jet jobs to be able to access spring beans from the ApplicationContext (see jet/...
lance-java's user avatar
  • 25.7k
0 votes
1 answer
591 views

Hazelcast Node multiple errors when run multiple nodes

i'm trying to launch several nodes with this programmatic configuration public class Server { private Server() { //TODO: MapStore/MapLoader config here MapConfig ...
killopolo's user avatar
0 votes
0 answers
94 views

Submit mutiple jobs in hazelcast jet

I have two separate jet pipeline. and I have to execute one by one. public static void main(String[] args) { JetInstance instance = Jet.newJetInstance(); // this job is for ...
h387518's user avatar
0 votes
2 answers
108 views

StreamSource vs BatchStage in hazelcast jet

I'm using CDC in my application. CDC always returns the object in the form of streams.StreamSource<ChangeRecord> source. My Requirement:- 1] Capture the db changes using cdc and store in the map....
vivek's user avatar
  • 115
2 votes
2 answers
274 views

Hazelcast Jet change data capture

I'm using Hazelcast Change data capture (CDC) in my application. (Reason I'm using CDC because if use jdbc or other alternative feature to load data into cache its taking to much of time). So CDC ...
vivek's user avatar
  • 115
2 votes
1 answer
383 views

Hazelcast Change Data Capture with Postgres

I'm trying to use CDC for my Postgres Database. and I have created simple project using Hazelcast Docs example. https://jet-start.sh/docs/tutorials/cdc-postgres public static void main(String[] args) {...
vivek's user avatar
  • 115
1 vote
1 answer
240 views

Hazelcast Jet "mapFn" must be Serializable Error

Good day, i'm having a problem implementing some code source from github to my project so... i'm trying to build a pipeline, and then create StreamStage to read the sources and build a queuesink with ...
killopolo's user avatar
0 votes
1 answer
111 views

Hazelcast jet data loading

I'm trying to load database data into hazelcast jet map instance public void sourceStage(JetInstance jet, Pipeline pipeLine){ SimpleDriverDataSource dataSource = new SimpleDriverDataSource(); ...
vivek's user avatar
  • 115
0 votes
1 answer
134 views

Problem Deserializing Hazelcast Aggregator Response

I have this custom Aggregator written, which aggregation response is a HashMap. I have a problem on client side deserializer fails in the error given below. Any clue what could be wrong? public class ...
aquawhale's user avatar
1 vote
0 answers
96 views

Hazelcast jet Python Exception: PythonService initialization failed

While executing python function but it's not working: Here is my code : BatchStage<Object> bd1 = AggregateData.aggregate(data, Object); bd1 = bd1.filter(k -> { // some ...
user3458271's user avatar
2 votes
1 answer
43 views

How to apply python on BatchStage Hazelcast Jet

What I am trying to do is, I am fetching the data from different source and doing grouping and join on them and after that I am getting result in BatchStage and now I have to perform python code on ...
user3458271's user avatar
0 votes
1 answer
1k views

Hazelcast adding address to blacklist and client not connecting

I am new to Hazelcast and in my application when I am starting Hazelcast member it is showing two addresses been blacklisted don't know why, what is the best configuration for Hazelcast clustering? ...
user3458271's user avatar
0 votes
1 answer
63 views

Convert BatchStage<List<Object>> to BatchStage<Object> in Hazelcast

In my hazelcast jet application I am performing grouping joins after that I am getting result in BatchStage<List> format but I want to convert it to BatchStage, I have tried flatMap but it ...
user3458271's user avatar
0 votes
1 answer
185 views

Hazelcast jet flatMap says: These transforms have nothing attached to them: [flat-map]

I am trying to filter BatchStage but it is not working it is giving me below error: java.lang.IllegalArgumentException: "filterFn" must be serializable at com.hazelcast.jet.impl.util....
user3458271's user avatar
2 votes
1 answer
201 views

Perform Left, Right and Inner join in Hazelcast Jet

I am working on Hazelcast jet application and I trying to join two Sources using Left, Right or Inner Join but I am stuck at below: Here is my code: BatchStage<Map<String,Object>> batch1= ...
user3458271's user avatar
0 votes
1 answer
202 views

How to control number of processors during Hazelcast Imap filter

I am new to Hazelcast. When Imap filter with predicate is triggered on a HazelCast Jet pipeline, in a VDI I see the porcessors count is only 2 and on my machine I see 12 porcessors. When there is only ...
Parthi's user avatar
  • 123
0 votes
1 answer
216 views

How to read CSV in Hazelcast jet in List<Map> or JsonArray format?

As I am new to hazelcast am I trying few thing but not getting result as I accepted, please help me out. Here is my below code which I am trying but not getting success. BatchSource<List> ...
user3458271's user avatar
0 votes
2 answers
165 views

Hazelcast Jet with Named Jdbc Query or ResultSet?

I am new to Hazelcast Jet and I am using Spring JdbcTemplate to execute my query, which uses named parameters in the query, but I am not sure how to use it with Hazelcast Jet. E.g. Hazelcast works in ...
user3458271's user avatar
3 votes
2 answers
274 views

wordcount test shows slowness in Flink

i am doing some benchmark comparison between streaming processing frameworks, I selected WordCount such "Hello world" task (with some twists) in this area, and tested Flink and Hazelcast Jet ...
Kuawiiii's user avatar
  • 151