Questions tagged [apache-pulsar]
Apache Pulsar is an open-source distributed pub-sub messaging system
345
questions
0
votes
0
answers
10
views
Pulsar OIDC protocol_version error with TLSv1.3 restricted Keycloak
We've setup a Pulsar 3.1.0 cluster with OIDC authentication. This works fine until we try to connect to a Keycloak instance that is restricted to only accepts TLSv1.3, then we get a fatal alert: ...
0
votes
0
answers
9
views
How to get the queued tasks (not yet consumed messages) when using Apache Pulsar as a distributed task queue?
I am currently evaluating the suitability of Apache Pulsar for building a distributed task queue using a "shared" subscription.
One required feature is to get all queued tasks for certain ...
0
votes
0
answers
12
views
How to get the currently running tasks when using Apache Pulsar as a distributed task queue?
I am currently evaluating the suitability of Apache Pulsar for building a distributed task queue using a "shared" subscription.
One required feature is to get all currently running tasks for ...
0
votes
0
answers
19
views
How to limit the number of concurrently processed tasks when using Apache Pulsar as a distributed task queue?
I am currently evaluating the suitability of Apache Pulsar to build a distributed task queue using a "shared" subscription.
One required feature is to limit the number of concurrently ...
0
votes
0
answers
42
views
How to error handle partition rebalancing?
In the deployment environment, Apache Pulsar is used as MsgQ broker.
Goal is to select the Go client library that allow consumer to error handle amidst consumer group rebalancing.
Consumer group ...
0
votes
0
answers
12
views
how to keep track of certain events, e.g. when queues,consumers are created and deleted?
how to keep track of certain events, e.g. when queues,consumers are created and deleted ?
similarity to rabbitmq-event-exchange (https://github.com/rabbitmq/rabbitmq-event-exchange)
I have 30000 topic,...
-1
votes
0
answers
22
views
Issue - Apache Pulsar Schema Evolution with Spark `streamnative / pulsar-spark`
Pulsar version = 3.1.0
Spark version = 3.4.0
running things on standalone.
Step 1
Created a topic using pulsar admin CLI and Uploaded schema of topic
Written some data using Pulsar Python Client Or ...
0
votes
1
answer
33
views
Why is there a delay delivering messages from new topic when subscribed via regex?
I have a Python program that receives messages from Pulsar and updates a document database. This works great when the messages are published to a pre-existing topic. But when a new topic is created ...
0
votes
0
answers
7
views
Issue with Pulsar GeoReplication
I have followed the steps and completed the Geo replication setup as per the pulsar documentation
https://pulsar.apache.org/docs/3.1.x/administration-geo/
All the geo replication related setup was ...
0
votes
0
answers
20
views
the pulsar send look like succeed but really not when the grpc and pulsar in python?
When I use grpc and pulsar in python3:
if I import pulsar first and grpc second, the pulsar.send will work fine and I can consume the info.
if I import the grpc first and the pulsar after, the pulsar....
0
votes
0
answers
11
views
How to implement horizontal horizontal scaling with mutii-topic failover subscriptions in Pulsar
I have about 100 entities, each with the same set of topics associated with them (e.g A-foo, B-foo, A-bar, B-bar) on a Pulsar message bus. For a given entity/topic, I need to process messages in the ...
0
votes
0
answers
52
views
Pulsar issue with connecting to the bookie when running as standalone in docker
After starting cluster producer, consumer, and pulsar itself, it seems there are cases where pulsar is not accepting the messages.
Following is the error:
pulsar | 2023-08-12T09:09:48,860+...
0
votes
1
answer
37
views
Facing issue with C# client of Apache pulsar getting faulted
I am working on incorporating Pulsar C# client library and facing issues with integrating the library.
If I call "producer.Send", its getting faulted.
public class PulsarProducerExample
{
...
0
votes
0
answers
26
views
Topic consuming blocked on one failed message in Apache Pulsar
I have a problem with Key_Shared subscription.
If a consumer fails to process one message, processing of all other messages with other keys is also stalled. Including the other consumers of the ...
0
votes
0
answers
24
views
Issue with topic level policy setup in apache-pulsar
I have changed the topic level policy in the broker.conf and standalone.conf file
systemTopicEnabled=true
topicLevelPoliciesEnabled=true
When i execute the below command it's failing due to policy ...
0
votes
0
answers
32
views
Can open Telemetry be used to add observability in Apache Pulsar?
I'm doing a project with Apache Pulsar and I want to add logging and metrics to the project. Can open Telemetry be used to add observability in Apache Pulsar?
In terms of both adding metrics and ...
0
votes
1
answer
32
views
Can Pulsar support multiple schemas on a single topic
I'm interested in the Schema functionality provided by Pulsar, but I want to validate an assumption before moving forward.
Let's say I have an inventory management system with a single topic called ...
4
votes
2
answers
113
views
Can we expose Apache Pulsar metrics through Micrometer?
The broker metrics are exposed under "/metrics/" at port 8080. But, what about the client metrics (both consumer and producer)? Can we expose Apache Pulsar metrics through Micrometer (...
0
votes
2
answers
56
views
Pulsar client numIoThreads and numListenerThreads per consumer instance or for all consumer instances
Let's say I have the following pulsar config:
pulsar:
topicX:
client:
config:
numIoThreads: 3
numListenerThreads: 15
consumer:
config:
receiverQueueSize: ...
0
votes
1
answer
40
views
Could not get connection while getPartitionedTopicMetadata - Connection handshake failed
I have setup apache pulsar by following this tutorial: https://pulsar.apache.org/docs/3.0.x/deploy-aws/ but I use aws ansible dynamic inventory rather than terraform inventory in a tutorial. The ...
1
vote
0
answers
66
views
Apache Pulsar Open ID authentication: Why do I receive unauthorised when correct client credentials are passed?
I've been attempting to configure Apache Pulsar cluster to enable OpenID Connect authentication by following the documentation/SO answers. However I receive the following unauthorized error...
[main] ...
0
votes
0
answers
92
views
JVM Parameters Not Being Applied Despite Being Set in Kubernetes ConfigMap
I have a Pulsar cluster running an EKS (kubernetes) cluster. I've been advised to try and increase the JVM heap size on of the pods (called toolset) as advised by one of our support vendors:
You are ...
0
votes
1
answer
147
views
Why does Apache Pulsar fails to start the broker service when enabling OpenID Connect Authentication?
After following the documentation (and answers in SO questions 1 & 2) to enable OpenID Connection Authentication the pulsar.broker service fails to start and provides me with a flurry of 401 not ...
0
votes
1
answer
78
views
Apache Pulsar fails to enable OpenID authentication in function worker when configured as specified in documentation
When following the documentation on enabling OpenID Connect authentication in the Function Worker, you are presented with the following error.
[jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd]...
0
votes
1
answer
102
views
Apache Pulsar unable to validate issuer certificate when attempting to load OpenID Connect provider
I've attempted to follow the instructions on https://pulsar.apache.org/docs/3.0.x/security-openid-connect/ in order to enable OpenID Connect Authentication, but can't seem to get passed a certificate ...
0
votes
0
answers
59
views
Pulsar send() method has low data write efficiency
When I use a Python program to write data to Pulsar, the QPS of a single partition can only reach around 500.
I need to ensure that data is sent successfully and in order, I use the send() method of a ...
0
votes
1
answer
139
views
How to consume JSON object in Apache Pulsar using spring-pulsar-spring-boot-starter?
I am trying to implement a simple messaging queue using Spring Boot and Apache Pulsar. The system works fine when working with String datatype but not for a Custom Object. Below are the object I'm ...
1
vote
1
answer
110
views
Spring boot org.apache.pulsar.client.api.Message.getValue() seems to lock any line of code it's included in. How do I stop it from doing this?
I am using spring boot to connect to a Pulsar message broker but I cannot access the message payload no matter what class I use in the call to:
//...
void listen(<T> message)
//...
I'm using ...
0
votes
1
answer
61
views
404 Partitioned topic does not exist
I am trying to delete a Namespace in Apache Pulsar but I am getting the following error:
javax.ws.rs.NotFoundException: HTTP 404 Partitioned topic does not exist
The command I am running is:
pulsar-...
0
votes
0
answers
57
views
Instrumenting pulsar functions
Has anyone had success instrumenting pulsar functions with otel? I am currently attempting the following, but not having much success.
Starting a pulsar-standalone docker container with conf/...
0
votes
0
answers
38
views
Apache Pulsar 2.11 - how can we track Broker/Consumer command flow message metrics?
We are using Apache Pulsar 2.11 and seeing scenarios that round-robin approach is not working correctly - keeping some consumers idle while there is enough backlog.
I have good understanding of what ...
0
votes
0
answers
32
views
apache pulsar Transaction pending ack generate managedLedgerStats fail! issue
apache pulsar Transaction pending ack generate managedLedgerStats fail! issue
05:48:00.084 [prometheus-stats-43-1] WARN org.apache.pulsar.broker.stats.prometheus.TransactionAggregator - Transaction ...
-1
votes
1
answer
100
views
Apache pulsar not started in windows
I am getting below error while starting Apache pulsar broker in windows. I have Windows 10 and Java 11.
what is missing?
0
votes
1
answer
32
views
How to modify the producer cache properties for Spring Pulsar 0.2.0 producer?
Spring Pulsar 0.2.0
how to modify ProducerConfigProperties.cache parameter
i want to change cache properties through application.properties, such as:
spring.pulsar.producer.cache.expire-after-access=...
0
votes
1
answer
512
views
Pulsar Spring Boot Cannot get DNS TTL settings from sun.net.InetAddressCachePolicy
I am trying to initialize PulsarClient in spring-boot's configuration class using the following code but getting
Java Details:
<properties>
<java.version>20</java.version>
...
0
votes
0
answers
58
views
how to expose pulsar broker to out side of k8s cluster?
I have installed pulsar helm charts using this link https://pulsar.apache.org/docs/2.10.x/getting-started-helm/. I am trying to expose the broker with nodeport option. I tried to add the node port to ...
0
votes
0
answers
44
views
How to pass the TLS certs to the latest version of flink-connector-pulsar
I am trying to connect my Flink application to a Pulsar topic for ingesting data. The topic is active and i am able to ingest the data via a normal Java application.
When i try to use the Flink ...
0
votes
1
answer
34
views
Connecting to Data Streams for Memgraph Import
I need to decide which streaming technology we should implement for our next project. I can choose between Kafka, Redpanda, or Pulsar stream, and I need to import streams into Memgraph. How can I ...
0
votes
0
answers
79
views
How to configure function state storage for Pulsar running on Kubernetes
We have recently been working on migrating our Pulsar cluster to run on Kubernetes. We have successfully moved over basic functionality as well as our sinks, however we have been struggling with our ...
0
votes
1
answer
65
views
if apache pulsar is a push model whether will it send notification to consumers when the message received from producer?
Would like to know apache pulsar has the functionality to send notification to its consumers those who subscribed? or only it will send messages when consumer requested
0
votes
0
answers
26
views
How to send FullDocument from Debezium Mongodb connector in Pulsar
guys. I've tried to do the poc about Apache Pulsar for my project. I tried to connect MongoDB to Pulsar function via the connector for detect any change events in the database. Everytime I updated ...
1
vote
0
answers
123
views
Avro Schema from JSON document in golang
I tried to create JSON parser which will create a AVRO schema. I was unsuccessful in implementing it
I am looking for golang Code which **infers AVRO Schema from JSON **document. if I can get a python ...
1
vote
0
answers
78
views
Pulsar does a infinite redelivery of message when maxredelivery count is breached for DLQ
I have simple setup where in i have pulasr consumer subscribing to a topic with DLQ of maxredelivery count= 2 on shared subscription type.
So when the message is consumed we send the message via ...
1
vote
0
answers
92
views
Is it possible to subscribe to subscriptions in apache pulsar?
Motivation:
Given a producer for particular set of topics, A*.
The producer should only generate the A* messages if there are active subscribers to those A* topics.
Question:
Do subscriptions generate ...
1
vote
1
answer
184
views
What happens when a consumer has both a negative acknowledge and a retry policy set
For Apache Pulsar, are negative acknowledge and retry topic policies mutually exclusive, or do they somehow work together? What if you enable both? What would the failure progression of a single ...
1
vote
0
answers
70
views
how to disable client statistic message when using python pulsar client
I am using Pulsar python client to subscribe on pulsar.
I'd like to disable client side statistics msg.
However looks like there is no such function in python client.
While, in java we can do ...
3
votes
1
answer
216
views
Pulsar: consuming from several partitioned topics
Let's consider the following example. We have two topics. Each of the topics has 4 partitions. Two consumers connect to these topics. We use key_shared subscription.
Is there any guarantee in pulsar ...
0
votes
1
answer
115
views
How can I enforce a custom order of messages in Pulsar?
I have 2 producers that will send similar messages to the same topic. In this case, one will send a DELETE for a specific ID/Key, the other will send a CREATE for a specific ID/key.
Producer 1:
...
0
votes
1
answer
200
views
How can I auto-generate a pulsar AvroSchema class from an existing model?
I'm running Apache Pulsar schemaless where the content is often changing. Now, there is some specific data for which I've written "data" classes (derived of SQLModel, which doesn't really ...
0
votes
0
answers
50
views
Multiple subscriber and go routine not work together
I'm trying to create multiple subscriber as much as go routine to consume message from pulsar but not sure why there is a problem.
package main
import (
"fmt"
"log"
&...