Questions tagged [apache-pulsar]

Apache Pulsar is an open-source distributed pub-sub messaging system

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Pulsar: consuming from several partitioned topics

Let's consider the following example. We have two topics. Each of the topics has 4 partitions. Two consumers connect to these topics. We use key_shared subscription. Is there any guarantee in pulsar ...
0 votes
0 answers
12 views

How can I enforce a custom order of messages in Pulsar?

I have 2 producers that will send similar messages to the same topic. In this case, one will send a DELETE for a specific ID/Key, the other will send a CREATE for a specific ID/key. Producer 1: ...
0 votes
1 answer
28 views

How can I auto-generate a pulsar AvroSchema class from an existing model?

I'm running Apache Pulsar schemaless where the content is often changing. Now, there is some specific data for which I've written "data" classes (derived of SQLModel, which doesn't really ...
  • 1,775
0 votes
1 answer
17 views

Multiple subscriber and go routine not work together

I'm trying to create multiple subscriber as much as go routine to consume message from pulsar but not sure why there is a problem. package main import ( "fmt" "log" &...
0 votes
1 answer
20 views

Apache Pulsar - Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds x amount - discarded

When I run a command via pulsar-admin, it works fine, but when I try to run the equivalent command via cURL or via the REST interface directly, I get: 2023-03-10T15:45:06,468-0700 [pulsar-io-19-8] ...
  • 4,580
1 vote
1 answer
29 views

How pulsar can handle ack between broker and consumer

I was so curious about the case when consumer consume a message, then it send ack to broker. Broker receive the ack and send ack back to consumer but due to connection lost or some other issues that ...
1 vote
1 answer
42 views

Pulsar give feedback to producer

We are currently building a service with pulsar where multiple producers can send messages onto a topic. Now we want the producers to get feedback on the processing of their messages. We thought to ...
  • 13
1 vote
0 answers
40 views

Apache Pulsar - 15k TPS with Payload size of 52kb?

It's been a month now and I am happy to share my POC experience with the Pulsar Community. Also have some scaling, stability, and configs-related concerns to resolve. With Standalone Setup, I got ...
  • 31
1 vote
1 answer
78 views

How to use Apache Pulsar with Spring Cloud Stream?

There is no official library of Apache Pulsar for Spring Cloud Stream. I found Spring for Apache Pulsar, but it is only about Spring Boot. In it's GitHub repository, there is a [spring-pulsar-spring-...
-1 votes
0 answers
40 views

After Pulsar installsation on client " org.apache.pulsar.common.util.netty.DnsResolverUtil

ava.lang.IllegalAccessException: class org.apache.pulsar.common.util.netty.DnsResolverUtil cannot access class sun.net.InetAddressCachePolicy (in module java.base) because module java.base does not ...
0 votes
1 answer
16 views

Can pulsar geo-replicate among different release versions, for example 2.8.x and 2.7.x?

Apache Pulsar supports geo-replication between clusters in different regions. I am wondering if there are any compatibility issues between clusters running different version of Pulsar. The question is ...
1 vote
1 answer
47 views

Pulsar producer send_async() with callback function acknowledging the sent message

I have a use case where messages from an input_topic gets consumed and sent to a list of topics. I'm using producers[i].send_async(msg, callback=callback) where callback = lambda res, msg: consumer....
1 vote
1 answer
33 views

How to run Push-Queries with Apache Pulsar?

I love the push queries of Apache ksqlDB. https://developer.confluent.io/learn-kafka/ksqldb/push-queries-and-pull-queries/ It allows to get notified via HTTP2 of a new result of a query whenever the ...
  • 1,775
1 vote
1 answer
38 views

Make sure messages goes to a specific consumer

I have multiple consumers subscribed to the same Pulsar topic. How do I make sure certain messages go to specific consumers? The closest thing I understand is the key-shared consumer type. However, ...
0 votes
1 answer
78 views

How to send JsonArray data in apache-pulsar-client?

I am a beginner who just started developing pulsar-client with spring boot. First of all, I learned the basics through pulsar doc and git, but I was stuck testing batch transmission of messages from ...
1 vote
2 answers
101 views

Multithreaded Use of Spring Pulsar

I am working on a project to read from our existing ElasticSearch instance and produce messages in Pulsar. If I do this in a highly multithreaded way without any explicit synchronization, I get many ...
  • 101
1 vote
1 answer
90 views

Spring Pulsar listener does not receive messages

Edit: the issue was that my @PulsarListener was in a class that was not a Spring Bean, and as such did not get discovered. Adding a @Service annotation solved the problem. I'm in a Spring Boot 3.0.1 ...
0 votes
0 answers
16 views

Issue loading a Jupyter notebook on AWS SageMaker

I have a Jupyter Notebook running on AWS SageMaker. One of the cells in the notebook was reading data row by row from a large (~5m rows) datastore. I ran the cell and then stopped it after confirming ...
  • 59
1 vote
1 answer
55 views

How to create an subscription with the Pulsar Admin Rest API with subscriptionType Key_Shared

Im trying to make a Subscription on an Apache Pulsar (2.10.1) Topic using the Admin Rest API and I would like to create it as the Type: Key_Shared or Shared. Making an subscription to an existing ...
0 votes
0 answers
35 views

Apache Pulsar Helm install failed on managed GKEAutopilot cluster

reconciliation failed: Helm install failed: services is forbidden: User "system:serviceaccount:flux-system:helm-controller" cannot create resource "services" in API group │ │ &...
1 vote
2 answers
42 views

Could not find artifact org.apache.pulsar:pulsar-broker:pom:2.12.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots)

While following the instructions to setup IntelliJ for pulsar development I got the below error Could not find artifact org.apache.pulsar:pulsar-broker:pom:2.12.0-SNAPSHOT in apache.snapshots (https://...
  • 746
1 vote
0 answers
59 views

pulsar-admin sink localrun failure: ClientCnx Failed to authenticate the client. (tls auth enabled)

command: bin/pulsar-admin sinks localrun -a connectors/pulsar-io-mongo-2.10.2.nar --tenant public --namespace default --inputs up-20wt --name mongo-sink --sink-config-file work/config.json --...
  • 11
1 vote
0 answers
85 views

Pulsar MongoDB Debezium source connection to Mongo Atlas not working

I set up a pulsar container with docker compose, and a Mongo DB Debezium source container configuration file. I am attempting to connect to MongoDb Cluster on Atlas without success. Your help is ...
0 votes
1 answer
138 views

kubectl exec with curl Connection refused

i cannot execute successfully a curl command inside a running pod $ kubectl exec -t pulsar-pulsar-manager-79c6769595-xd596 -- bash -c "CSRF_TOKEN=$(curl -v http://localhost:7750/pulsar-manager/...
  • 424
3 votes
1 answer
91 views

Excluding consumer of a topic by sender in Pulsar

Intro We're developing a system to support multiple real-time messages (chat) and updates (activity notifications). That is, user A can receive via Web Socket messages for : receiving new chat ...
  • 2,266
1 vote
0 answers
67 views

Is there any Pulsar UI where messages can be updated/produced?

The only UI I found was Pulsar Manager , which don't even allow to see the messages in a topic. By any chance is there a Pulsar UI with better features where messages can be manipulated?
1 vote
0 answers
67 views

How to move specific messages from one topic to another on demand in Pulsar (using pulsar-admin)?

I have a requirement of moving specific messages from DLQ to RETRY queue. I want to use pulsar-admin for this task. I have message-id for all the messages. ~note: don't want to copy and produce ...
1 vote
1 answer
103 views

Custom Objects with Apache Pulsar and Spring Boot

I'm using Apache Pulsar for the first time but finding it hard to publish and listen to custom objects. I have defined schemas as directed by this reference pages but for some reason it’s not working. ...
1 vote
1 answer
161 views

Group messages by key for pulsar key_shared subscription type

Say I have an unbounded set of keys for messages published to a pulsar topic. {k0, k1, ..., kn} And a finite set of expected message categories, where the category information is part of the message ...
0 votes
1 answer
34 views

pulsar spock groovy test

How with groovy mock pulsar producer if I has code: myProducer.newMessage().key(String.valueOf(1L)).value(nameBytes).send() my producer is mock myProducer = Mock(Producer). But in the test I got ...
1 vote
1 answer
48 views

java, apache pulsar Topic partition

If I have a topic called var test = "test" and I create a bean: @Bean(name = "test") public Producer<Test> testProducer(PulsarClient pulsarClient){ return pulsarClient....
0 votes
0 answers
15 views

Implementing an RPC system with apache pulsar

I would like to implement an RPC system with NodeJS and Java client using Apache Pulsar. Unless there is already one, would you have any advice to implement this?
1 vote
0 answers
28 views

apache pulsar Direct approach with Apache spark

There are 2 approaches of integrating apache kafka with apache spark as mentioned here with pros and cons. From what i have read there are 2 spark connector for apache pulsar org.apache.pulsar:pulsar-...
  • 5,395
1 vote
1 answer
80 views

Pulsar JVM Dashboard Zookeeper show No Data on column direct memory

Describe the bug To monitor a Pulsar cluster, I used grafana and in the Pulsar JVM dashboard, Zookeeper Direct Memory is show No Data. While other dashboards have data such as Bookie, Broker, Recovery....
0 votes
0 answers
41 views

data published to pulsar by apache nifi (pulsar-nifi-bundle) is not persisted in lakehouse using delta lake sink connector

Does anyone know if this pulsar nifi connector can publish to a deltasink ( delta lake sink connector ) https://github.com/david-streamlio/pulsar-nifi-bundle I am getting data to pulsar with nifi and ...
  • 63
0 votes
0 answers
35 views

Apache Pulsar Support Message Chunking for Python?

Does Apache Pulsar support message chunking in its Python client library? https://pulsar.apache.org/api/python/2.10.1/pulsar.html#Client.create_producer <-- missing how to enable chunking https://...
  • 510
0 votes
1 answer
28 views

How can I create Pulsar stream from Python?

How can I create a Kafka stream directly from Python? I know that I need to use GQLAlchemy but I don't know the exact commands.
  • 850
1 vote
2 answers
103 views

Spring JMS Integration with Apache Pulsar

Our current code is that we use MQ to receive messages from upstream application. JMS will consume messages from MQ and inserts into database. The requirement is that instead of MQ we need to use ...
  • 11
0 votes
0 answers
90 views

Pulsar Function Window Length Count

I am trying to write a pulsar function with python that executes every 10 samples of data. Below is the sandbox code for the pulsar function from pulsar import Function class Func(Function): def ...
1 vote
0 answers
59 views

Failed to decrypt data key x.pem to decrypt messages unable to process block at Consumer in apache pulsar with version v2.0.0-rc1-incubating

Producer publishes same messages to a topic encrypted with different public keys to consume by different applications with a set of public key and private key. Apllication Consumer able to consume the ...
  • 11
0 votes
0 answers
33 views

does Pulsar has data repair mechanism like read repair in Cassandra?

As we know in Cassandra, there is read repair which is used to make data consistent in quorum write. In Pulsar, let's assume that Write Quorum Size (Qw) is 4, Ack Quorum Size (Qa) is 3. If client try ...
0 votes
1 answer
141 views

PulsarClient.builder() extremely long error logger

I am trying to make a pulsar client written in java, however I get an error with the logger(slf4j). I thought that it was because my PropertyConfigurator was badly made, but even if I use the ...
1 vote
2 answers
209 views

Apache pulsar / Pulsar Manager

I have installed apache pulsar using helm chart in AWS EKS, Reference : Apache Pulsar Helm Deployment on Kubernetes also enabled pulsar manager and deployed pulsar manager UI, all pods are up and ...
0 votes
0 answers
17 views

Could not load file or assembly 'DotPulsar, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies

even though installed nuget package, runtime getting below error Could not load file or assembly 'DotPulsar, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A ...
1 vote
2 answers
212 views

Delta Lake Sink Connector for Apache Pulsar with miniO throws (java.lang.IllegalArgumentException)

I am trying to run the new apache pulsar Lakehouse Sink Connector and I am getting java.lang.IllegalArgumentException Below is my setup. docker-compose.yaml file: version: '3.7' volumes: mssql-data: ...
  • 63
1 vote
0 answers
334 views

How did parameterizing this function make my Zio project stop working?

I've started writing a server that uses zio-http to forward messages from a Pulsar topic to a WebSocket. It was working fine, but I realized I was creating the consumer again when the socket was being ...
  • 3,383
1 vote
1 answer
250 views

Apache Pulsar Client - Broker notification of Closed consumer - how to resume data feed?

TLDR: using python client library to subscribe to pulsar topic. logs show: 'broker notification of consumer closed' when something happens server-side. subscription appears to be re-established ...
  • 11
0 votes
1 answer
164 views

pulsar-datadog configuration files into Data dog agent openmetrics.d directory not publishing

Has anyone tried to implement the pulsar-datadog configurations into their Kubernetes based Datadog agent. I've taken the steps to helm install the datadog agent to my kubernetes cluster and copied ...
  • 746
0 votes
2 answers
126 views

Unexpect error 'Pulsar error: ServiceUnitNotReady' when try to connect to pulsar

Hello I was debuging a pulsar code, and I was working perfectly, but one time after a re-execution of the code suddenly show a error, of when try to connect: Loading .env environment variables... 0000-...
  • 4,478
0 votes
1 answer
162 views

Pulsar: Failure reason while sending message in DLQ

We are using apache-pulsar's retry topic and dead letter topic to handle failures in event processing. In case of a failure in processing, event is retried 3 times via retry topic and if it still ...
  • 63

1
2 3 4 5
7