13

Almost all of the examples of pub/sub available over web uses String messages.

How do I publish message other than text messages like Java Object, JSON or AVRO to topic and then consume it through subscription.

Many Thanks Pari

4 Answers 4

8

The following code lets you publish a message in Pub/Sub using JSON:

topic_path = 'your-topic-id'

publisher = pubsub_v1.PublisherClient()

record = {
    'Key1': 'Value1',
    'Key2': 'Value2',
    'Key3': 'Value3',
    'Key4': 'Value4'
}

data = json.dumps(record).encode("utf-8")
future = publisher.publish(topic_path, data)
print(f'published message id {future.result()}')

I hope that It helps you.

2
  • This answer should be accepted. All the others are workarounds.
    – AJ AJ
    Commented Dec 21, 2022 at 9:04
  • 1
    To consume this in a cloud function it's json.loads(base64.b64decode(cloud_event.data["message"]["data"]).decode()) as per docs
    – citynorman
    Commented Feb 19 at 14:39
7

You cannot really do that directly, as you can see here and here the published message has to be a bytestring. What you can do instead is load your file, encode it to utf-8 and then publish it. Afterwards, when pulling, you can dump the message data to a file. For instance, using python for json, you would need to publish like so:

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic_name)

with open(json_path) as f:
    data = str(json.load(f))

data = data.encode('utf-8')
publisher.publish(topic_path, data=data)

Then you can pull and dump like so:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project, subscription_name)

def callback(message):
    with open('received_message.json', 'w') as outfile:
        json.dump(message.data, outfile)

subscriber.subscribe(subscription_path, callback=callback)
1
  • Thanks for the answer but I guess, there could be better options. Lets wait for more suggestions.
    – Pari
    Commented Jul 3, 2018 at 15:21
4

publish JSON with Node - publishMessage:

const {PubSub} = require('@google-cloud/pubsub')
const pubsub = new PubSub()

const json = {
  foo: 'bar'
}

await pubsub.topic('my-topic').publishMessage({json})
0

As of this writing

The node.js @google-cloud/pubsub library only accepts buffer objects and you'll need to stringify.

https://github.com/googleapis/nodejs-pubsub/blob/master/samples/topics.js

async function publishMessage(topicName, data) {

  // Imports the Google Cloud client library
  const {PubSub} = require('@google-cloud/pubsub');

  // Creates a client
  const pubsub = new PubSub();

  // Publishes the message as a string, e.g. "Hello, world!" or JSON.stringify(someObject)
  const dataBuffer = Buffer.from(data);

  const messageId = await pubsub
    .topic(topicName)
    .publisher()
    .publish(dataBuffer);
  console.log(`Message ${messageId} published.`);
}

This might change soon!

Please follow this request/issue: https://github.com/googleapis/nodejs-pubsub/issues/121

The library may soon be modified to accept non buffer objects and buffer them for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.