Questions tagged [apache-pulsar]
Apache Pulsar is an open-source distributed pub-sub messaging system
308
questions
0
votes
0
answers
6
views
Pulsar: consuming from several partitioned topics
Let's consider the following example. We have two topics. Each of the topics has 4 partitions. Two consumers connect to these topics. We use key_shared subscription.
Is there any guarantee in pulsar ...
0
votes
0
answers
12
views
How can I enforce a custom order of messages in Pulsar?
I have 2 producers that will send similar messages to the same topic. In this case, one will send a DELETE for a specific ID/Key, the other will send a CREATE for a specific ID/key.
Producer 1:
...
0
votes
1
answer
28
views
How can I auto-generate a pulsar AvroSchema class from an existing model?
I'm running Apache Pulsar schemaless where the content is often changing. Now, there is some specific data for which I've written "data" classes (derived of SQLModel, which doesn't really ...
0
votes
1
answer
17
views
Multiple subscriber and go routine not work together
I'm trying to create multiple subscriber as much as go routine to consume message from pulsar but not sure why there is a problem.
package main
import (
"fmt"
"log"
&...
0
votes
1
answer
20
views
Apache Pulsar - Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds x amount - discarded
When I run a command via pulsar-admin, it works fine, but when I try to run the equivalent command via cURL or via the REST interface directly, I get:
2023-03-10T15:45:06,468-0700 [pulsar-io-19-8] ...
1
vote
1
answer
29
views
How pulsar can handle ack between broker and consumer
I was so curious about the case when consumer consume a message, then it send ack to broker. Broker receive the ack and send ack back to consumer but due to connection lost or some other issues that ...
1
vote
1
answer
42
views
Pulsar give feedback to producer
We are currently building a service with pulsar where multiple producers can send messages onto a topic. Now we want the producers to get feedback on the processing of their messages. We thought to ...
1
vote
0
answers
40
views
Apache Pulsar - 15k TPS with Payload size of 52kb?
It's been a month now and I am happy to share my POC experience with the Pulsar Community. Also have some scaling, stability, and configs-related concerns to resolve.
With Standalone Setup, I got ...
1
vote
1
answer
78
views
How to use Apache Pulsar with Spring Cloud Stream?
There is no official library of Apache Pulsar for Spring Cloud Stream.
I found Spring for Apache Pulsar, but it is only about Spring Boot.
In it's GitHub repository, there is a [spring-pulsar-spring-...
-1
votes
0
answers
40
views
After Pulsar installsation on client " org.apache.pulsar.common.util.netty.DnsResolverUtil
ava.lang.IllegalAccessException: class org.apache.pulsar.common.util.netty.DnsResolverUtil cannot access class sun.net.InetAddressCachePolicy (in module java.base) because module java.base does not ...
0
votes
1
answer
16
views
Can pulsar geo-replicate among different release versions, for example 2.8.x and 2.7.x?
Apache Pulsar supports geo-replication between clusters in different regions. I am wondering if there are any compatibility issues between clusters running different version of Pulsar.
The question is ...
1
vote
1
answer
47
views
Pulsar producer send_async() with callback function acknowledging the sent message
I have a use case where messages from an input_topic gets consumed and sent to a list of topics. I'm using producers[i].send_async(msg, callback=callback) where callback = lambda res, msg: consumer....
1
vote
1
answer
33
views
How to run Push-Queries with Apache Pulsar?
I love the push queries of Apache ksqlDB. https://developer.confluent.io/learn-kafka/ksqldb/push-queries-and-pull-queries/
It allows to get notified via HTTP2 of a new result of a query whenever the ...
1
vote
1
answer
38
views
Make sure messages goes to a specific consumer
I have multiple consumers subscribed to the same Pulsar topic. How do I make sure certain messages go to specific consumers?
The closest thing I understand is the key-shared consumer type. However, ...
0
votes
1
answer
78
views
How to send JsonArray data in apache-pulsar-client?
I am a beginner who just started developing pulsar-client with spring boot.
First of all, I learned the basics through pulsar doc and git, but I was stuck testing batch transmission of messages from ...
1
vote
2
answers
101
views
Multithreaded Use of Spring Pulsar
I am working on a project to read from our existing ElasticSearch instance and produce messages in Pulsar. If I do this in a highly multithreaded way without any explicit synchronization, I get many ...
1
vote
1
answer
90
views
Spring Pulsar listener does not receive messages
Edit: the issue was that my @PulsarListener was in a class that was not a Spring Bean, and as such did not get discovered. Adding a @Service annotation solved the problem.
I'm in a Spring Boot 3.0.1 ...
0
votes
0
answers
16
views
Issue loading a Jupyter notebook on AWS SageMaker
I have a Jupyter Notebook running on AWS SageMaker. One of the cells in the notebook was reading data row by row from a large (~5m rows) datastore.
I ran the cell and then stopped it after confirming ...
1
vote
1
answer
55
views
How to create an subscription with the Pulsar Admin Rest API with subscriptionType Key_Shared
Im trying to make a Subscription on an Apache Pulsar (2.10.1) Topic using the Admin Rest API and I would like to create it as the Type: Key_Shared or Shared.
Making an subscription to an existing ...
0
votes
0
answers
35
views
Apache Pulsar Helm install failed on managed GKEAutopilot cluster
reconciliation failed: Helm install failed: services is forbidden: User "system:serviceaccount:flux-system:helm-controller" cannot create resource "services" in API group │
│ &...
1
vote
2
answers
42
views
Could not find artifact org.apache.pulsar:pulsar-broker:pom:2.12.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots)
While following the instructions to setup IntelliJ for pulsar development I got the below error
Could not find artifact org.apache.pulsar:pulsar-broker:pom:2.12.0-SNAPSHOT in apache.snapshots (https://...
1
vote
0
answers
59
views
pulsar-admin sink localrun failure: ClientCnx Failed to authenticate the client. (tls auth enabled)
command:
bin/pulsar-admin sinks localrun -a connectors/pulsar-io-mongo-2.10.2.nar --tenant public --namespace default --inputs up-20wt --name mongo-sink --sink-config-file work/config.json --...
1
vote
0
answers
85
views
Pulsar MongoDB Debezium source connection to Mongo Atlas not working
I set up a pulsar container with docker compose, and a Mongo DB Debezium source container configuration file. I am attempting to connect to MongoDb Cluster on Atlas without success. Your help is ...
0
votes
1
answer
138
views
kubectl exec with curl Connection refused
i cannot execute successfully a curl command inside a running pod
$ kubectl exec -t pulsar-pulsar-manager-79c6769595-xd596 -- bash -c "CSRF_TOKEN=$(curl -v http://localhost:7750/pulsar-manager/...
3
votes
1
answer
91
views
Excluding consumer of a topic by sender in Pulsar
Intro
We're developing a system to support multiple real-time messages (chat) and updates (activity notifications).
That is, user A can receive via Web Socket messages for :
receiving new chat ...
1
vote
0
answers
67
views
Is there any Pulsar UI where messages can be updated/produced?
The only UI I found was Pulsar Manager , which don't even allow to see the messages in a topic.
By any chance is there a Pulsar UI with better features where messages can be manipulated?
1
vote
0
answers
67
views
How to move specific messages from one topic to another on demand in Pulsar (using pulsar-admin)?
I have a requirement of moving specific messages from DLQ to RETRY queue. I want to use pulsar-admin for this task.
I have message-id for all the messages.
~note: don't want to copy and produce ...
1
vote
1
answer
103
views
Custom Objects with Apache Pulsar and Spring Boot
I'm using Apache Pulsar for the first time but finding it hard to publish and listen to custom objects. I have defined schemas as directed by this reference pages but for some reason it’s not working.
...
1
vote
1
answer
161
views
Group messages by key for pulsar key_shared subscription type
Say I have an unbounded set of keys for messages published to a pulsar topic.
{k0, k1, ..., kn}
And a finite set of expected message categories, where the category information is part of the message ...
0
votes
1
answer
34
views
pulsar spock groovy test
How with groovy mock pulsar producer if I has code:
myProducer.newMessage().key(String.valueOf(1L)).value(nameBytes).send()
my producer is mock myProducer = Mock(Producer). But in the test I got ...
1
vote
1
answer
48
views
java, apache pulsar Topic partition
If I have a topic called var test = "test"
and I create a bean:
@Bean(name = "test")
public Producer<Test> testProducer(PulsarClient pulsarClient){
return pulsarClient....
0
votes
0
answers
15
views
Implementing an RPC system with apache pulsar
I would like to implement an RPC system with NodeJS and Java client using Apache Pulsar. Unless there is already one, would you have any advice to implement this?
1
vote
0
answers
28
views
apache pulsar Direct approach with Apache spark
There are 2 approaches of integrating apache kafka with apache spark as mentioned here with pros and cons.
From what i have read there are 2 spark connector for apache pulsar
org.apache.pulsar:pulsar-...
1
vote
1
answer
80
views
Pulsar JVM Dashboard Zookeeper show No Data on column direct memory
Describe the bug
To monitor a Pulsar cluster, I used grafana and in the Pulsar JVM dashboard, Zookeeper Direct Memory is show No Data. While other dashboards have data such as Bookie, Broker, Recovery....
0
votes
0
answers
41
views
data published to pulsar by apache nifi (pulsar-nifi-bundle) is not persisted in lakehouse using delta lake sink connector
Does anyone know if this pulsar nifi connector can publish to a deltasink ( delta lake sink connector )
https://github.com/david-streamlio/pulsar-nifi-bundle
I am getting data to pulsar with nifi and ...
0
votes
0
answers
35
views
Apache Pulsar Support Message Chunking for Python?
Does Apache Pulsar support message chunking in its Python client library?
https://pulsar.apache.org/api/python/2.10.1/pulsar.html#Client.create_producer <-- missing how to enable chunking
https://...
0
votes
1
answer
28
views
How can I create Pulsar stream from Python?
How can I create a Kafka stream directly from Python? I know that I need to use GQLAlchemy but I don't know the exact commands.
1
vote
2
answers
103
views
Spring JMS Integration with Apache Pulsar
Our current code is that we use MQ to receive messages from upstream application. JMS will consume messages from MQ and inserts into database.
The requirement is that instead of MQ we need to use ...
0
votes
0
answers
90
views
Pulsar Function Window Length Count
I am trying to write a pulsar function with python that executes every 10 samples of data.
Below is the sandbox code for the pulsar function
from pulsar import Function
class Func(Function):
def ...
1
vote
0
answers
59
views
Failed to decrypt data key x.pem to decrypt messages unable to process block at Consumer in apache pulsar with version v2.0.0-rc1-incubating
Producer publishes same messages to a topic encrypted with different public keys to consume by different applications with a set of public key and private key.
Apllication Consumer able to consume the ...
0
votes
0
answers
33
views
does Pulsar has data repair mechanism like read repair in Cassandra?
As we know in Cassandra, there is read repair which is used to make data consistent in quorum write.
In Pulsar, let's assume that Write Quorum Size (Qw) is 4, Ack Quorum Size (Qa) is 3.
If client try ...
0
votes
1
answer
141
views
PulsarClient.builder() extremely long error logger
I am trying to make a pulsar client written in java, however I get an error with the logger(slf4j). I thought that it was because my PropertyConfigurator was badly made, but even if I use the ...
1
vote
2
answers
209
views
Apache pulsar / Pulsar Manager
I have installed apache pulsar using helm chart in AWS EKS,
Reference : Apache Pulsar Helm Deployment on Kubernetes
also enabled pulsar manager and deployed pulsar manager UI, all pods are up and ...
0
votes
0
answers
17
views
Could not load file or assembly 'DotPulsar, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies
even though installed nuget package, runtime getting below error
Could not load file or assembly 'DotPulsar, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A ...
1
vote
2
answers
212
views
Delta Lake Sink Connector for Apache Pulsar with miniO throws (java.lang.IllegalArgumentException)
I am trying to run the new apache pulsar Lakehouse Sink Connector and I am getting java.lang.IllegalArgumentException
Below is my setup.
docker-compose.yaml file:
version: '3.7'
volumes:
mssql-data:
...
1
vote
0
answers
334
views
How did parameterizing this function make my Zio project stop working?
I've started writing a server that uses zio-http to forward messages from a Pulsar topic to a WebSocket. It was working fine, but I realized I was creating the consumer again when the socket was being ...
1
vote
1
answer
250
views
Apache Pulsar Client - Broker notification of Closed consumer - how to resume data feed?
TLDR: using python client library to subscribe to pulsar topic. logs show: 'broker notification of consumer closed' when something happens server-side. subscription appears to be re-established ...
0
votes
1
answer
164
views
pulsar-datadog configuration files into Data dog agent openmetrics.d directory not publishing
Has anyone tried to implement the pulsar-datadog configurations into their Kubernetes based Datadog agent. I've taken the steps to helm install the datadog agent to my kubernetes cluster and copied ...
0
votes
2
answers
126
views
Unexpect error 'Pulsar error: ServiceUnitNotReady' when try to connect to pulsar
Hello I was debuging a pulsar code, and I was working perfectly, but one time after a re-execution of the code suddenly show a error, of when try to connect:
Loading .env environment variables...
0000-...
0
votes
1
answer
162
views
Pulsar: Failure reason while sending message in DLQ
We are using apache-pulsar's retry topic and dead letter topic to handle failures in event processing. In case of a failure in processing, event is retried 3 times via retry topic and if it still ...