Questions tagged [apache-pulsar]

Apache Pulsar is an open-source distributed pub-sub messaging system

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Pulsar OIDC protocol_version error with TLSv1.3 restricted Keycloak

We've setup a Pulsar 3.1.0 cluster with OIDC authentication. This works fine until we try to connect to a Keycloak instance that is restricted to only accepts TLSv1.3, then we get a fatal alert: ...
Davidas's user avatar
0 votes
0 answers
9 views

How to get the queued tasks (not yet consumed messages) when using Apache Pulsar as a distributed task queue?

I am currently evaluating the suitability of Apache Pulsar for building a distributed task queue using a "shared" subscription. One required feature is to get all queued tasks for certain ...
Marco Stahl's user avatar
0 votes
0 answers
12 views

How to get the currently running tasks when using Apache Pulsar as a distributed task queue?

I am currently evaluating the suitability of Apache Pulsar for building a distributed task queue using a "shared" subscription. One required feature is to get all currently running tasks for ...
Marco Stahl's user avatar
0 votes
0 answers
19 views

How to limit the number of concurrently processed tasks when using Apache Pulsar as a distributed task queue?

I am currently evaluating the suitability of Apache Pulsar to build a distributed task queue using a "shared" subscription. One required feature is to limit the number of concurrently ...
Marco Stahl's user avatar
0 votes
0 answers
42 views

How to error handle partition rebalancing?

In the deployment environment, Apache Pulsar is used as MsgQ broker. Goal is to select the Go client library that allow consumer to error handle amidst consumer group rebalancing. Consumer group ...
overexchange's user avatar
  • 15.8k
0 votes
0 answers
12 views

how to keep track of certain events, e.g. when queues,consumers are created and deleted?

how to keep track of certain events, e.g. when queues,consumers are created and deleted ? similarity to rabbitmq-event-exchange (https://github.com/rabbitmq/rabbitmq-event-exchange) I have 30000 topic,...
zmy's user avatar
  • 1
-1 votes
0 answers
22 views

Issue - Apache Pulsar Schema Evolution with Spark `streamnative / pulsar-spark`

Pulsar version = 3.1.0 Spark version = 3.4.0 running things on standalone. Step 1 Created a topic using pulsar admin CLI and Uploaded schema of topic Written some data using Pulsar Python Client Or ...
RakeshV's user avatar
  • 442
0 votes
1 answer
33 views

Why is there a delay delivering messages from new topic when subscribed via regex?

I have a Python program that receives messages from Pulsar and updates a document database. This works great when the messages are published to a pre-existing topic. But when a new topic is created ...
Brian Richardson's user avatar
0 votes
0 answers
7 views

Issue with Pulsar GeoReplication

I have followed the steps and completed the Geo replication setup as per the pulsar documentation https://pulsar.apache.org/docs/3.1.x/administration-geo/ All the geo replication related setup was ...
Sathiskumar Perumal's user avatar
0 votes
0 answers
20 views

the pulsar send look like succeed but really not when the grpc and pulsar in python?

When I use grpc and pulsar in python3: if I import pulsar first and grpc second, the pulsar.send will work fine and I can consume the info. if I import the grpc first and the pulsar after, the pulsar....
shaominyao's user avatar
0 votes
0 answers
11 views

How to implement horizontal horizontal scaling with mutii-topic failover subscriptions in Pulsar

I have about 100 entities, each with the same set of topics associated with them (e.g A-foo, B-foo, A-bar, B-bar) on a Pulsar message bus. For a given entity/topic, I need to process messages in the ...
user1941996's user avatar
0 votes
0 answers
52 views

Pulsar issue with connecting to the bookie when running as standalone in docker

After starting cluster producer, consumer, and pulsar itself, it seems there are cases where pulsar is not accepting the messages. Following is the error: pulsar | 2023-08-12T09:09:48,860+...
Sameer Joshi's user avatar
0 votes
1 answer
37 views

Facing issue with C# client of Apache pulsar getting faulted

I am working on incorporating Pulsar C# client library and facing issues with integrating the library. If I call "producer.Send", its getting faulted. public class PulsarProducerExample { ...
Sameer Joshi's user avatar
0 votes
0 answers
26 views

Topic consuming blocked on one failed message in Apache Pulsar

I have a problem with Key_Shared subscription. If a consumer fails to process one message, processing of all other messages with other keys is also stalled. Including the other consumers of the ...
user1380915's user avatar
0 votes
0 answers
24 views

Issue with topic level policy setup in apache-pulsar

I have changed the topic level policy in the broker.conf and standalone.conf file systemTopicEnabled=true topicLevelPoliciesEnabled=true When i execute the below command it's failing due to policy ...
Sathiskumar Perumal's user avatar
0 votes
0 answers
32 views

Can open Telemetry be used to add observability in Apache Pulsar?

I'm doing a project with Apache Pulsar and I want to add logging and metrics to the project. Can open Telemetry be used to add observability in Apache Pulsar? In terms of both adding metrics and ...
Scoby's user avatar
  • 158
0 votes
1 answer
32 views

Can Pulsar support multiple schemas on a single topic

I'm interested in the Schema functionality provided by Pulsar, but I want to validate an assumption before moving forward. Let's say I have an inventory management system with a single topic called ...
Doug Ferguson's user avatar
4 votes
2 answers
113 views

Can we expose Apache Pulsar metrics through Micrometer?

The broker metrics are exposed under "/metrics/" at port 8080. But, what about the client metrics (both consumer and producer)? Can we expose Apache Pulsar metrics through Micrometer (...
Scoby's user avatar
  • 158
0 votes
2 answers
56 views

Pulsar client numIoThreads and numListenerThreads per consumer instance or for all consumer instances

Let's say I have the following pulsar config: pulsar: topicX: client: config: numIoThreads: 3 numListenerThreads: 15 consumer: config: receiverQueueSize: ...
Thomas Verhoeven's user avatar
0 votes
1 answer
40 views

Could not get connection while getPartitionedTopicMetadata - Connection handshake failed

I have setup apache pulsar by following this tutorial: https://pulsar.apache.org/docs/3.0.x/deploy-aws/ but I use aws ansible dynamic inventory rather than terraform inventory in a tutorial. The ...
BOB's user avatar
  • 1
1 vote
0 answers
66 views

Apache Pulsar Open ID authentication: Why do I receive unauthorised when correct client credentials are passed?

I've been attempting to configure Apache Pulsar cluster to enable OpenID Connect authentication by following the documentation/SO answers. However I receive the following unauthorized error... [main] ...
raah's user avatar
  • 158
0 votes
0 answers
92 views

JVM Parameters Not Being Applied Despite Being Set in Kubernetes ConfigMap

I have a Pulsar cluster running an EKS (kubernetes) cluster. I've been advised to try and increase the JVM heap size on of the pods (called toolset) as advised by one of our support vendors: You are ...
Hofbr's user avatar
  • 868
0 votes
1 answer
147 views

Why does Apache Pulsar fails to start the broker service when enabling OpenID Connect Authentication?

After following the documentation (and answers in SO questions 1 & 2) to enable OpenID Connection Authentication the pulsar.broker service fails to start and provides me with a flurry of 401 not ...
raah's user avatar
  • 158
0 votes
1 answer
78 views

Apache Pulsar fails to enable OpenID authentication in function worker when configured as specified in documentation

When following the documentation on enabling OpenID Connect authentication in the Function Worker, you are presented with the following error. [jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd]...
raah's user avatar
  • 158
0 votes
1 answer
102 views

Apache Pulsar unable to validate issuer certificate when attempting to load OpenID Connect provider

I've attempted to follow the instructions on https://pulsar.apache.org/docs/3.0.x/security-openid-connect/ in order to enable OpenID Connect Authentication, but can't seem to get passed a certificate ...
raah's user avatar
  • 158
0 votes
0 answers
59 views

Pulsar send() method has low data write efficiency

When I use a Python program to write data to Pulsar, the QPS of a single partition can only reach around 500. I need to ensure that data is sent successfully and in order, I use the send() method of a ...
ke du's user avatar
  • 1
0 votes
1 answer
139 views

How to consume JSON object in Apache Pulsar using spring-pulsar-spring-boot-starter?

I am trying to implement a simple messaging queue using Spring Boot and Apache Pulsar. The system works fine when working with String datatype but not for a Custom Object. Below are the object I'm ...
Neha P's user avatar
  • 13
1 vote
1 answer
110 views

Spring boot org.apache.pulsar.client.api.Message.getValue() seems to lock any line of code it's included in. How do I stop it from doing this?

I am using spring boot to connect to a Pulsar message broker but I cannot access the message payload no matter what class I use in the call to: //... void listen(<T> message) //... I'm using ...
Dan's user avatar
  • 45
0 votes
1 answer
61 views

404 Partitioned topic does not exist

I am trying to delete a Namespace in Apache Pulsar but I am getting the following error: javax.ws.rs.NotFoundException: HTTP 404 Partitioned topic does not exist The command I am running is: pulsar-...
HLT's user avatar
  • 327
0 votes
0 answers
57 views

Instrumenting pulsar functions

Has anyone had success instrumenting pulsar functions with otel? I am currently attempting the following, but not having much success. Starting a pulsar-standalone docker container with conf/...
Damo's user avatar
  • 1,519
0 votes
0 answers
38 views

Apache Pulsar 2.11 - how can we track Broker/Consumer command flow message metrics?

We are using Apache Pulsar 2.11 and seeing scenarios that round-robin approach is not working correctly - keeping some consumers idle while there is enough backlog. I have good understanding of what ...
JTBS's user avatar
  • 11
0 votes
0 answers
32 views

apache pulsar Transaction pending ack generate managedLedgerStats fail! issue

apache pulsar Transaction pending ack generate managedLedgerStats fail! issue 05:48:00.084 [prometheus-stats-43-1] WARN org.apache.pulsar.broker.stats.prometheus.TransactionAggregator - Transaction ...
Nithin SUbbaraj's user avatar
-1 votes
1 answer
100 views

Apache pulsar not started in windows

I am getting below error while starting Apache pulsar broker in windows. I have Windows 10 and Java 11. what is missing?
Chowdappa's user avatar
  • 1,580
0 votes
1 answer
32 views

How to modify the producer cache properties for Spring Pulsar 0.2.0 producer?

Spring Pulsar 0.2.0 how to modify ProducerConfigProperties.cache parameter i want to change cache properties through application.properties, such as: spring.pulsar.producer.cache.expire-after-access=...
DeveloperV's user avatar
0 votes
1 answer
512 views

Pulsar Spring Boot Cannot get DNS TTL settings from sun.net.InetAddressCachePolicy

I am trying to initialize PulsarClient in spring-boot's configuration class using the following code but getting Java Details: <properties> <java.version>20</java.version> ...
silentsudo's user avatar
  • 6,750
0 votes
0 answers
58 views

how to expose pulsar broker to out side of k8s cluster?

I have installed pulsar helm charts using this link https://pulsar.apache.org/docs/2.10.x/getting-started-helm/. I am trying to expose the broker with nodeport option. I tried to add the node port to ...
Ravi's user avatar
  • 424
0 votes
0 answers
44 views

How to pass the TLS certs to the latest version of flink-connector-pulsar

I am trying to connect my Flink application to a Pulsar topic for ingesting data. The topic is active and i am able to ingest the data via a normal Java application. When i try to use the Flink ...
Bauddhik's user avatar
0 votes
1 answer
34 views

Connecting to Data Streams for Memgraph Import

I need to decide which streaming technology we should implement for our next project. I can choose between Kafka, Redpanda, or Pulsar stream, and I need to import streams into Memgraph. How can I ...
Moraltox's user avatar
  • 539
0 votes
0 answers
79 views

How to configure function state storage for Pulsar running on Kubernetes

We have recently been working on migrating our Pulsar cluster to run on Kubernetes. We have successfully moved over basic functionality as well as our sinks, however we have been struggling with our ...
Thomas Turner's user avatar
0 votes
1 answer
65 views

if apache pulsar is a push model whether will it send notification to consumers when the message received from producer?

Would like to know apache pulsar has the functionality to send notification to its consumers those who subscribed? or only it will send messages when consumer requested
ilavarasan M's user avatar
0 votes
0 answers
26 views

How to send FullDocument from Debezium Mongodb connector in Pulsar

guys. I've tried to do the poc about Apache Pulsar for my project. I tried to connect MongoDB to Pulsar function via the connector for detect any change events in the database. Everytime I updated ...
zosmex's user avatar
  • 3
1 vote
0 answers
123 views

Avro Schema from JSON document in golang

I tried to create JSON parser which will create a AVRO schema. I was unsuccessful in implementing it I am looking for golang Code which **infers AVRO Schema from JSON **document. if I can get a python ...
Anand Narendra's user avatar
1 vote
0 answers
78 views

Pulsar does a infinite redelivery of message when maxredelivery count is breached for DLQ

I have simple setup where in i have pulasr consumer subscribing to a topic with DLQ of maxredelivery count= 2 on shared subscription type. So when the message is consumed we send the message via ...
Sanjay's user avatar
  • 11
1 vote
0 answers
92 views

Is it possible to subscribe to subscriptions in apache pulsar?

Motivation: Given a producer for particular set of topics, A*. The producer should only generate the A* messages if there are active subscribers to those A* topics. Question: Do subscriptions generate ...
phreed's user avatar
  • 1,769
1 vote
1 answer
184 views

What happens when a consumer has both a negative acknowledge and a retry policy set

For Apache Pulsar, are negative acknowledge and retry topic policies mutually exclusive, or do they somehow work together? What if you enable both? What would the failure progression of a single ...
Zach Blocker's user avatar
1 vote
0 answers
70 views

how to disable client statistic message when using python pulsar client

I am using Pulsar python client to subscribe on pulsar. I'd like to disable client side statistics msg. However looks like there is no such function in python client. While, in java we can do ...
wood's user avatar
  • 29
3 votes
1 answer
216 views

Pulsar: consuming from several partitioned topics

Let's consider the following example. We have two topics. Each of the topics has 4 partitions. Two consumers connect to these topics. We use key_shared subscription. Is there any guarantee in pulsar ...
Victor Denisov's user avatar
0 votes
1 answer
115 views

How can I enforce a custom order of messages in Pulsar?

I have 2 producers that will send similar messages to the same topic. In this case, one will send a DELETE for a specific ID/Key, the other will send a CREATE for a specific ID/key. Producer 1: ...
Brian Lee's user avatar
0 votes
1 answer
200 views

How can I auto-generate a pulsar AvroSchema class from an existing model?

I'm running Apache Pulsar schemaless where the content is often changing. Now, there is some specific data for which I've written "data" classes (derived of SQLModel, which doesn't really ...
feder's user avatar
  • 1,869
0 votes
0 answers
50 views

Multiple subscriber and go routine not work together

I'm trying to create multiple subscriber as much as go routine to consume message from pulsar but not sure why there is a problem. package main import ( "fmt" "log" &...
The Lannisters's user avatar

1
2 3 4 5
7