All Questions

Filter by
Sorted by
Tagged with
0 votes
1 answer
30 views

GCP Pub Sub topics

My application is based on 2 GCP cloud functions which consume messages from different topics. The first cloud function consumes a message from the 1st topic about object creation. The second cloud ...
Arman's user avatar
  • 1
0 votes
0 answers
28 views

Unable initialise pub/sub with SparkSession

SO I'm adding Pub/Sub connector in SparkSession to use spark streaming. But every time it gives error that Pub/Sub data source not found. How to fix it ! from pyspark.sql import SparkSession # ...
Chidananda Nayak's user avatar
0 votes
0 answers
20 views

KeyError in Apache Beam while reading from pubSub,'ref_PCollection_PCollection_6'

I am working on a usecase where I need to read from pubSub and create a streaming pipeline. I am using colab notebook, to write and test the pipeline. But, when I am reading from pubSub, its throwing ...
Amarjeet's user avatar
0 votes
0 answers
18 views

Google Cloud Function Connection Error when Deployed but Works in Inline Editor

I have a Google Cloud Function written in Python that is designed to export data from BigQuery to Google Cloud Storage. The function works perfectly fine when I test it within the inline editor of ...
Miniman005's user avatar
0 votes
0 answers
42 views

Can I ack/nack message after the streaming pull timeout exceeds?

I am writing a python program using Google CLoud Pub/Sub subscription, following this guide. Now I have the following code. subscriber = pubsub_v1.SubscriberClient(my_credentials) subscription_path = ...
hyperbola's user avatar
  • 124
0 votes
1 answer
21 views

Google PubSub Lite one subscriber with multiple partitions

I wanted to understand what is the behaviour of Google PubSub Lite when there is only 1 subscriber subscribed to a topic that has 2 partitions. Whether the subscriber receive the data from both the ...
nijrajgelani-crest's user avatar
1 vote
1 answer
37 views

Pull PubSub Message through Proxy server - Python

I am using this below script to pull message, I have developed as per document but facing error import json from googleapiclient.discovery import build from httplib2 import Http import httplib2 from ...
Moses01's user avatar
  • 152
0 votes
0 answers
43 views

PubSub async publishing

I wanted to make sure (haven't found it anywhere in the documentation), If I am using PubSub async publishing as part of my service handling request logic (Python Django) - *handling request* publish ...
Itai Bar's user avatar
  • 717
0 votes
1 answer
75 views

Writing to GCS using Cloud Functions with python gives OSError: [Errno 30] Read-only file system

I'm trying to convert a dataframe to xlsx, and writing it to Google Cloud storage using Cloud functions, but it's not working. It worked when I tested and deployed, but when I tried to trigger with ...
Zathara's user avatar
0 votes
0 answers
54 views

PUBSUB push subscription(cloud run endpoint) acknowledging to avoid retries

I have a cloud run endpoint that gets triggered by a push type pubsub subscription and it recives the user specified message thorugh publsiher,I am trying to get the message acknowledged by the ...
neelima's user avatar
0 votes
1 answer
57 views

PubSub Error : AssertionError: A message queue should exist for every ordered message in flight

currently I am developing publisher and subscriber in python program, this program is quite simple which i found somewhere in github def receive_messages(project_id, subscription_name): subscriber ...
lifetolearn22's user avatar
0 votes
1 answer
235 views

Google cloud function finishes with status ok even when I return 500 error code

I am using python for a google cloud function. When an exception occures I catch it and reutrn error code 500 but the log shows that the funcion finishes with status "OK". This is the code: ...
blob's user avatar
  • 475
2 votes
1 answer
219 views

How to use PublisherServiceAsyncClient

I am trying to use AsyncPublisherClient from pubsublite pip install google-cloud-pubsublite==1.8.3 from google.cloud import pubsublite_v1 pubsub = pubsublite_v1.AsyncPublisherClient() error: Module ...
Rodrigo's user avatar
  • 516
0 votes
1 answer
108 views

Python Pubsub Subscriber client not pulling messages when filter is applied on subscription

I am using Pubsub streaming pull subscription in my Python web application. When I am not applying any subscription filter, the subscriber client is able to successfully pull messages from the ...
Sumit Desai's user avatar
  • 1,672
0 votes
1 answer
1k views

Getting asyncio.run() cannot be called from a running event loop

I have a python application which is running inside a uvicorn server. I have created a Pub/sub subscriber and trying to enable it from my main.py. I am using a streaming-pull subscription. Now, my ...
Sumit Desai's user avatar
  • 1,672
0 votes
1 answer
118 views

How to get all messages from GCloud Pubsub?

I am writing a scheduled cloud function, which subscribes to a pubsub topic and pulls all the messages from the pubsub until the queue is free. I want to ack some messages but nack the others so they ...
Abdullah Sheikh's user avatar
0 votes
0 answers
130 views

Join pub/sub real time data with BQ side input slowly changing data

I am working on a use case where I need to join real time events (read from pub/sub) that are processed using a sliding window with a slowly changing data (read from BQ). This is the code to read BQ ...
khalil's user avatar
  • 37
0 votes
0 answers
100 views

Slow File Download from Google Cloud Storage while Using Pub/Sub Subscription in Python

I am working on a Python script that subscribes to a Google Cloud Pub/Sub topic, and upon receiving a message, it triggers the download of files from a Google Cloud Storage bucket. The code is ...
vlad Hohlov's user avatar
0 votes
0 answers
23 views

How to handle number of threads increasing linearly with number of topics in streaming pull?

I have a Python script that takes a list of topics from the user and listens for events published to any of those topics. When an event is received, the script executes a specified action, such as ...
Harsh Nasit's user avatar
0 votes
1 answer
120 views

Consume messages from Pub/Sub with Cloud Functions untill topic is empty

I'm facing a technical challenge with a message processing system using a Pub/Sub topic, where my Cloud Function processes messages individually. However, I've encountered a scenario where if there ...
kbenzakrilelp's user avatar
0 votes
1 answer
105 views

How to ignore a message in google pubsub and do not acknolwedge it back?

Hi I am working with a topic and subscription where I want to acknowledge only specific messages in one topic subscription and ignore the rest of the messages.Is there a way or process how we can ...
akshay dirisala's user avatar
0 votes
1 answer
108 views

How to stop python running in parallel execution

I am using the below script to get the message from Google Pub/Sub. When I execute the code, it is not waiting for the sleep time mentioned in the function time_sleep(). My expected result is, Message ...
Moses's user avatar
  • 226
0 votes
0 answers
118 views

Python unit test patch and mock?

I am trying to unit test my Workers class functions inside my workers.py module while testing file_download() which basically manipulates the message and makes another its own message and publish it ...
Shivam Kumar's user avatar
1 vote
1 answer
161 views

How can I receive firebase messages using python (or API)

I have a fleet of devices running python and need to be able to send messages to/from them each individually. I know that it is possible to send a message to an individual InstanceID token, but I can'...
Daniel Watrous's user avatar
1 vote
3 answers
92 views

Dataflow pipeline creation fails due to lack of resources in current zone

I've set up a Dataflow pipeline that ingests messages from Pub/Sub, converts to a dict and prints the messages. Here is the script I've written: import apache_beam as beam import logging import ...
Joe Moore's user avatar
  • 1,961
0 votes
0 answers
35 views

Is there any way to mark a new email from a specific email address with a label using gmail api or through pub sub model in python?

Problem: I need to find a way to mark all the emails(past and future) with a specific label that my code just created, I have the label id and I can mark all the past emails but I am unable to find a ...
Madhur Jodhwani's user avatar
0 votes
0 answers
103 views

Google pubsub client can't subscribe with Locust

I'm trying to implement a GCloudSubUser for the Python library Locust. I'm implementing this custom User because I need the user to: send a message to a topic of Google Pub/sub wait for my API's ...
JPFrancoia's user avatar
  • 5,232
1 vote
1 answer
223 views

Pub/Sub push subscription not forwarding message with schema?

Introduction I've created an Avro Schema which I attached to a Topic. To this Topic I've added two subscriptions, a BigQuery and a Push. BigQuery Subscription is working great, and the table is ...
Joao Leme's user avatar
0 votes
1 answer
73 views

Function not defined on Google Dataflow

when i run the following code, that it takes CSV files from a GCS bucket and, after validating them it publishes messages to PubSub, using the DirectRunner with Python it works fine: import ...
Amedeo Tortora's user avatar
0 votes
0 answers
407 views

Mocking Google PubSub instance for testing publish and subscribe method

I have a module 'initialize' which returns a pub sub instance based on users input currently returns 2(Google PubSub and RabbitMQ pub sub). def create_pubsub( required input parameters ) -> ...
Shivam Kumar's user avatar
0 votes
1 answer
23 views

PubSub Google - I am unable to publish messages in unique signatures

I have the following code snippet in python that publishes messages to pubsub queues from the content of a file that falls into cloud storage. The logic is as follows: If the file falls in the ...
Guilherme Duarte's user avatar
1 vote
1 answer
241 views

Import error Google pub/sub on gen1 Google cloud function

I tried many ways to from google.cloud import pubsub_v1 in python 3.11 but each time when I check function logs it says: ImportError: cannot import name 'pubsub_v1' from 'google.cloud' (unknown ...
Aliosha kolioshap's user avatar
2 votes
1 answer
288 views

Workload Identity stops working when using gevent monkey patching

I have an application which includes a Flask API server and a worker to process messages from PubSub. These run as separate containers on separate pods in Kubernetes. I've migrated to using Workload ...
M4C4R's user avatar
  • 336
0 votes
1 answer
397 views

GCP From pubsub message to trigger multiple instance of cloud function

I have a situation where I want to publish a message to Pubsub, which triggers multiple instances of the same cloud function. here is some information: In the pub-sub message, I have project ids ...
Waqas Sarwar MVP's user avatar
0 votes
1 answer
804 views

Google cloud PubSub keeps throwing timeout error on production

It's my first time using google pubsub on production. We are using the Python client and deploying projects with docker and k8s. The code snippet is just as the sample code provided by google. os....
DYX's user avatar
  • 31
0 votes
1 answer
346 views

Extract column values from pubsub output using apache beam in python

I am trying to extract data from PubSub subscription and finally, once the data is extracted I want to do some transformation. Currently, it's in bytes format. I have tried multiple ways to extract ...
Data writer's user avatar
1 vote
0 answers
500 views

How do I seek using python in GCP Pubsub?

I am trying to use the Google Python pubsub client to seek to now in a subscription: seek_request_dict = { "subscription":subscription, "time":current_time ...
schoon's user avatar
  • 3,104
0 votes
1 answer
578 views

Publish Pandas Dataframe rows as PubSub Message

I have a task to publish the rows of a pandas dataframe as a pubsub message. What is the best to do this? My pandas dataframe could consist of around 1 million records. This is an example I have found ...
Ashok KS's user avatar
  • 679
0 votes
1 answer
1k views

Cloud Function to retrieve attributes from PubSub - Python

I created a GCP Cloud Functions which is triggered by a PubSub topic. import base64 def hello_pubsub(event, context): pubsub_message = base64.b64decode(event['data']).decode('utf-8') print(...
Ashok KS's user avatar
  • 679
0 votes
1 answer
1k views

Understanding Open Telemetry Integration to google cloud pub sub in python

I am curious about how distributed tracing can be provided to a message from a publisher and how this is received in the subscriber part just to get the possibility to get track about what could be ...
bgarcial's user avatar
  • 3,085
0 votes
2 answers
387 views

Dataflow Job to start based on PubSub Notification - Python

I am writing a Dataflow job which reads from BigQuery and does a few transformations. data = ( pipeline | beam.io.ReadFromBigQuery(query=''' SELECT * FROM `bigquery-public-data....
Ashok KS's user avatar
  • 679
-2 votes
1 answer
171 views

PubSub Publisher Ingestion

I am trying to ingest an avro file from a gcs to pubsub. Have some layman questions. What are the options to send a file as a message in PubSub? Like can we send the whole file as a message or only ...
mehere's user avatar
  • 1,526
0 votes
2 answers
256 views

How to handle lack of messages from Pub/Sub in Google Dataflow

The below is just random example code: import argparse import apache_beam as beam from apache_beam.transforms import window from apache_beam.transforms.trigger import AfterWatermark, AccumulationMode ...
Shahid Thaika's user avatar
0 votes
0 answers
85 views

How to implement an auto-scaling solution requiring synchronous subscription?

I have to process requests published in a Pub/ Sub Topic by the users of my "service" in a Python application, having a main loop. Each instance of the service will only be able to process ...
Torsten Knodt's user avatar
0 votes
1 answer
174 views

How to add the message content to the results in Google Pub/Sub?

I have the following code, based on Google's official API def publish_messages_with_error_handler(project_id: str = GOOGLE_CLOUD_PROJECT_ID, topic_id: str = ...
Guilherme Massoqueto's user avatar
0 votes
1 answer
938 views

Google cloud PubSub service not working (Python)

I am trying to use the pub sub service on my python application. When I am running the code it get stuck on the last publisher line for some reason and the code never end. The subscriber seems fine....
bob's user avatar
  • 75
0 votes
0 answers
366 views

GCP Cloud Function works, but gives errors

I have a cloud function which works on my local machine but when I put it live on GCP Cloud Functions I get errors and it crashes. But the data that is mean to be used and collected is done so ...
DrewS's user avatar
  • 93
1 vote
1 answer
13k views

TypeError: Function missing 1 required positional argument: 'self'

I have a function in GCP that I have working on my local machine, but when i push it onto GCP Cloud Functions I keep getting an error. the function collects a message from PubSub and then uses the IDs ...
DrewS's user avatar
  • 93
1 vote
1 answer
124 views

How do i save one subscription pub/sub message to three different big query tables in apache beam in python?

I have created three process and want to save data but this is not working for me. Below is my custom pardo: subscription = "pub/sub/writeToBigquery" dataset_table1="A" ...
user avatar
0 votes
1 answer
428 views

GCP Cloud Function PubSub not passing message to topic

I am trying to write a GCP Cloud Function which collects some data from an API call then stores selected data points in Firestore. I also would like to pass the event IDs to PubSub topic so that I can ...
DrewS's user avatar
  • 93

1
2 3 4 5
7