136

I have not worked on kafka much but wanted to build data pipeline in GCE. So we wanted to know Kafka vs PUB/Sub. Basically I want to know how message consistency, message availability, message reliability is maintained in both Kafka and Pub/sub

Thanks

1

3 Answers 3

144

In addition to Google Pub/Sub being managed by Google and Kafka being open source, the other difference is that Google Pub/Sub is a message queue (e.g. Rabbit MQ) where as Kafka is more of a streaming log. You can't "re-read" or "replay" messages with Pubsub. (EDIT - as of 2019 Feb, you CAN replay messages and seek backwards in time to a certain timestamp, per comment below)

With Google Pub/Sub, once a message is read out of a subscription and ACKed, it's gone. In order to have more copies of a message to be read by different readers, you "fan-out" the topic by creating "subscriptions" for that topic, where each subscription will have an entire copy of everything that goes into the topic. But this also increases cost because Google charges Pub/Sub usage by the amount of data read out of it.

With Kafka, you set a retention period (I think it's 7 days by default) and the messages stay in Kafka regardless of how many consumers read them. You can add a new consumer (a.k.a. subscriber), and have it start consuming from the front of the topic any time you want. You can also set the retention period to be infinite, and then you can basically use Kafka as an immutable datastore, as described here: http://stackoverflow.com/a/22597637/304262

Amazon AWS Kinesis is a managed version of Kafka whereas I think of Google Pubsub as a managed version of Rabbit MQ. Amazon SNS with SQS is also similar to Google Pubsub (SNS provides the fanout and SQS provides the queueing).

8
  • 7
    Replay is a critical feature in most event-oriented architectures. In addition, Kafka adds a sequence number to messages and therefore becomes the authoritative source of sequence. Commented Aug 12, 2017 at 15:37
  • 5
    The way to accomplish "replay" with a message queue system like PubSub is by doing fan-out of the topic to more subscriptions (i.e. make more copies of the messages), and each consumer consumes their own subscription at their own pace. I suppose you can have a subscription that's just meant for replay when you need it. To do the same thing with Kafka, you would create a new consumer and start consuming from the front (since Kafka doesn't make a copy of the messages, it just gives each consumer their own "pointer" offset to keep track of what was already read)
    – gunit
    Commented Aug 16, 2017 at 23:20
  • 8
    Kinesis can be thought of as a managed service that is semantically similar to Kafka, but it's inaccurate to say that it's a "managed version of Kafka". For an actual "managed Kafka", see Confluent Cloud confluent.io/confluent-cloud Commented Aug 22, 2018 at 22:42
  • 11
    Cloud Pub/Sub recently added support for replaying previously acknowledged messages. The quickstart guide and blog post explain how to use the feature. Commented Feb 5, 2019 at 15:05
  • 3
    Suppose you have a listener. It wakes up upon receipt of a message, does some work, and then crashes. What happens to the event it was working on? WIth replay+sequence numbers, you can always store last known good state + sequence number and pick up where you left off. Commented Feb 6, 2021 at 23:16
46

I have been reading the answers above and I would like to complement them, because I think there are some details pending:

Fully Managed System Both system can have fully managed version in the cloud. Google provides Pubsub and there are some fully managed Kafka versions out there that you can configure on the cloud and On-prem.

Cloud vs On-prem I think this is a real difference between them, because Pubsub is only offered as part of the GCP ecosystem whereas Apache Kafka you can use as a both Cloud service and On-prem service (doing the cluster configuration by yourself)

Message duplication - With Kafka you will need to manage the offsets of the messages by yourself, using an external storage, such as, Apache Zookeeper. In that way you can track the messages read so far by the Consumers. Pubsub works using acknowledging the message, if your code doesn't acknowledge the message before the deadline, the message is sent again, that way you can avoid duplicated messages or another way to avoid is using Cloud Dataflow PubsubIO.

Retention policy Both Kafka and Pubsub have options to configure the maximum retention time, by default, I think is 7 days.

Consumers Group vs Subscriptions Be careful how you read messages in both systems. Pubsub use subscriptions, you create a subscription and then you start reading messages from that subscription. Once a message is read and acknowledge, the message for that subscription is gone. Kafka use the concept of "consumer group" and "partition", every consumer process belongs to a group and when a message is read from a specific partition, then any other consumer process which belongs to the same "consumer group" will not be able to read that message (that is because the offset eventually will increase). You can see the offset as a pointer which tells the processes which message have to read.

I think there is not a correct answer for your question, it will really depends on what you will need and the constrains you have (below are some examples of the escenarios):

  • If the solution must be in GCP, obviously use Google Cloud Pubsub. You will avoid all the settings efforts or pay extra for a fully automated system that Kafka requires.

  • If the solution should require process data in Streaming way but also needs to support Batch processing (eventually), it is a good idea to use Cloud Dataflow + Pubsub.

  • If the solution require to use some Spark processing, you could explore Spark Streaming (which you can configure Kafka for the stream processing)

In general, both are very solid Stream processing systems. The point which make the huge difference is that Pubsub is a cloud service attached to GCP whereas Apache Kafka can be used in both Cloud and On-prem.

Update (April 6th 2021):

4
  • 10
    I think this can be misleading; Unless you want to write your own library on the Kafka wire protocol, the existing clients already provide configurable mechanisms to deal with committing the offsets. Also comitted offsets are not kept in Zookeeper but in a special topic "__consumer_offsets" which is replicated among the brokers. This is a good read: confluent.io/blog/…
    – Zoltan
    Commented Jan 1, 2020 at 14:24
  • 5
    Indeed I really don't understand your statement about storing offsets manually: With Kafka you will need to manage the offsets of the messages by yourself, using an external storage, such as, Apache Zookeeper => Downvoting
    – Fares
    Commented Dec 27, 2020 at 23:15
  • retention part is wrong: for Kafka it can be set to infinite Commented Jun 1, 2022 at 8:37
  • There is some misconception about offset management in Kafka. Prior to Kafka2.8 (KIP-500) the Zk was integral part of Kafka cluster. So there is no "managing yourself" part, all done by the Kafka internally. Unless you are interested in "re-inventing the wheel" there is no reason to use some parts of Kafka. Since Kafka2.8 even this part was removed and Zk no longer needed. But tgose are kafka internals and your code should be agnostic to it. Commented Nov 15, 2022 at 3:54
23

One big difference between Kafka vs. Cloud Pub/Sub is that Cloud Pub/Sub is fully managed for you. You don't have to worry about machines, setting up clusters, fine tune parameters etc. which means that a lot of DevOps work is handled for you and this is important, especially when you need to scale.

3
  • 26
    That's not really a difference because there are multiple vendors that offer Kafka as a fully managed service too. The difference perhaps that that Google PubSub is only available as a service in Googles Cloud so there is no on prem version nor is there a managed service running in other cloud providers like AWS or Azure. Commented Jun 10, 2017 at 2:32
  • 3
    "Google PubSub is only available as a service in Googles Cloud" that is incorrect...your application is not tied to being deployed in Google App Engine..you can connect and publish to GooglePub/Sub" from any client as long as you securely connect to it via a 'service account".
    – Jeryl Cook
    Commented Feb 16, 2018 at 19:07
  • 25
    @JerylCook I think he just means you can't install google's pub/sub on prem Commented Apr 27, 2018 at 22:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.