Almost all of the examples of pub/sub available over web uses String messages.
How do I publish message other than text messages like Java Object, JSON or AVRO to topic and then consume it through subscription.
Many Thanks Pari
The following code lets you publish a message in Pub/Sub using JSON:
topic_path = 'your-topic-id'
publisher = pubsub_v1.PublisherClient()
record = {
'Key1': 'Value1',
'Key2': 'Value2',
'Key3': 'Value3',
'Key4': 'Value4'
}
data = json.dumps(record).encode("utf-8")
future = publisher.publish(topic_path, data)
print(f'published message id {future.result()}')
I hope that It helps you.
You cannot really do that directly, as you can see here and here the published message has to be a bytestring. What you can do instead is load your file, encode it to utf-8
and then publish it. Afterwards, when pulling, you can dump the message data to a file. For instance, using python for json, you would need to publish like so:
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic_name)
with open(json_path) as f:
data = str(json.load(f))
data = data.encode('utf-8')
publisher.publish(topic_path, data=data)
Then you can pull and dump like so:
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
def callback(message):
with open('received_message.json', 'w') as outfile:
json.dump(message.data, outfile)
subscriber.subscribe(subscription_path, callback=callback)
publish JSON with Node - publishMessage:
const {PubSub} = require('@google-cloud/pubsub')
const pubsub = new PubSub()
const json = {
foo: 'bar'
}
await pubsub.topic('my-topic').publishMessage({json})
As of this writing
The node.js @google-cloud/pubsub library only accepts buffer objects and you'll need to stringify.
https://github.com/googleapis/nodejs-pubsub/blob/master/samples/topics.js
async function publishMessage(topicName, data) {
// Imports the Google Cloud client library
const {PubSub} = require('@google-cloud/pubsub');
// Creates a client
const pubsub = new PubSub();
// Publishes the message as a string, e.g. "Hello, world!" or JSON.stringify(someObject)
const dataBuffer = Buffer.from(data);
const messageId = await pubsub
.topic(topicName)
.publisher()
.publish(dataBuffer);
console.log(`Message ${messageId} published.`);
}
This might change soon!
Please follow this request/issue: https://github.com/googleapis/nodejs-pubsub/issues/121
The library may soon be modified to accept non buffer objects and buffer them for you.