Questions tagged [google-cloud-storage]

Google Cloud Storage is a RESTful service for storing and accessing your data on Google's infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. ONLY PROGRAMMING RELATED QUESTIONS ARE ON-TOPIC. Questions which are not specifically programming related are OFF-TOPIC for Stack Overflow.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Is there any way to get the latest S3 file by GCP StorageTransfer?

We're running Airflow/Composer on GCP and trying to get the latest S3 file by any python operator. Currently we're getting S3 file once a day, they can happen to export several files in the same day ...
Sachiko's user avatar
  • 840
0 votes
1 answer
20 views

CSV file formatting doesn't work for GCP BlobWriter

I have same data as list of dict and want to import these in a CSV file in a GCP bucket. I'm importing these object as streaming: blob = defaults.bucket.blob(filename) csv_writer = BlobWriter(blob, ...
Zura Begadze's user avatar
0 votes
0 answers
30 views

Firestore: subcollection pretends to be empty

friends. Really need your help on this one. I've developed an iOS app that uses Firestore database and Storage. Everything works as intended. The thing I'm stuck with is propagating changes using ...
Michael Ros's user avatar
  • 1,171
0 votes
0 answers
19 views

Why am I getting "e.on is not a function" when uploading a file from React app to Google Cloud Storage?

I'm using a file input component from the React-Bootstrap library to select an image file from my machine. Then, I'm attempting to upload the image to a Google Cloud Storage bucket from memory. I'm ...
Jesse's user avatar
  • 25
0 votes
1 answer
32 views

Google Cloud IAM: website hosted in Cloud Storage to trigger Cloud Function (Gen 2)

I'm working inside Google Cloud, completely inside single project. My current issue is associated with corrected access rights only. I've developed a simple website hosted on Cloud Storage and having ...
Mike's user avatar
  • 3
0 votes
1 answer
24 views

How do I iterate through JSON files stored in GCP bucket in different folders. Example; | Bucket/Dict/Folder2/file.json Bucket/Dict/Folder1/file.json

I have dumped JSON files from DOCAI to GCP but each file is stored in individual folder, although they are in the same bucket on Cloud Storage. I am not able to iterate through the JSON files stored ...
Vedant Patil's user avatar
0 votes
0 answers
29 views

Conditional Binding for Objects in Google Cloud Storage Buckets

I am working with Google Cloud Storage (GCS) buckets and I need to set up conditional access control for objects in these buckets. Specifically, I want to give viewers access to objects in a certain ...
Matias Lopez's user avatar
0 votes
0 answers
15 views

Best practices for optimizing storage performance in a cloud environment? [closed]

What are the best practices for optimizing storage performance in a cloud environment? Efficient storage management is crucial for the overall performance and cost-effectiveness of our cloud-based ...
adamsmithad's user avatar
0 votes
1 answer
35 views

GCP: can we use output of one GSUTIL command embedded in other GSUTIL command

I have a requirement where have a file in GCS and I need to remove the header and trailer of a file and load the data into other GCS bucket. I have tried using something like this, but its not working....
Mani Shankar.S's user avatar
0 votes
0 answers
35 views

How to export BigQuery table to parquet using SQL and keep columns mode

I'm trying to export BQ table to GCS in the parquet format. Can't use API because of the size limits, so it's gotta be SQL query. The problem is when using SQL query, parquet ignores table columns ...
AlienDeg's user avatar
  • 1,290
0 votes
0 answers
14 views

Setting cors on my GCP bucket not making a difference when accessing data from my bucket

I set all my GCP bucket items to be accessible to the public. I also set cors on my bucket using gsutil cors set cors-config.json gs://myBucketName. My cors config file looks like this: [{"origin&...
seriously's user avatar
  • 1,232
-3 votes
0 answers
27 views

How to store the information? [closed]

how to store the information of a value that is the result of a function in the cloud so that when I refresh the page that value continues in the same place, in javascript So far I've tried Local ...
Nahuel Garcia's user avatar
0 votes
1 answer
35 views

Move files from private google bucket to a google drive

well we basically store some files on buckets in Google cloud, now I started to work on a feature to schedule tasks and move files, so what we want is the user to provide an api key/secret of google ...
Prince Hernandez's user avatar
0 votes
0 answers
39 views

Aliyun OSS to GCP Cloud Storage migration

I am trying to migrate files from Aliyun OSS to GCP Cloud Storage. As I see, Transfer service just supports AWS and Azure Blob storage natively. Workaround will be to use Rclone for this job, but is ...
zigbee's user avatar
  • 1
0 votes
1 answer
48 views

Uploading files to GCS using Multithread keeps running

Am trying to upload 36k+ CSV files to GCS from a java application. This application is hosted as an appengine in the same GCP. As the number of files are more, am using parallelStream.foreach in java ...
Dhivya Sadasivam's user avatar
0 votes
1 answer
46 views

When attempting to create a Data Store for GCP's "Search & Conversaion", storage.objects.get permission is missing

For Google Cloud Platform's Search & Conversation, I am trying to create a "Data Store" using Google Cloud Storage (GCS) as the source. Step 1 is to choose the source (GCS in this case), ...
Jane Wayne's user avatar
  • 8,317
0 votes
1 answer
52 views

Spring-Batch Reader, Processor and Writer

I need to create a Spring Batch job that reads various data from MySQL, puts this data into a .csv file, and then sends this file to Google Cloud Storage. I would like to understand the architecture. ...
Pedro Buttenbender's user avatar
0 votes
0 answers
32 views

Error: I can't download files larger than 70KB to my Google Cloud Storage bucket

I'm having an issue with GCS and Bun. It turns out that I want to download an image from my bucket, and I'm encountering an error. I'm using the .download() method from the GCS library, but it gives ...
Pato P's user avatar
  • 1
0 votes
1 answer
33 views

unable to store getDownloadUrl in state

const [values, setValues] = useState({ description: "", postImgUrl: "" }); const [imgPreview, setImgPreview] = useState(); const handleChange = (e) => { setImgPreview(...
user1450966's user avatar
-1 votes
1 answer
38 views

Google cloud sql - Storage size reset [closed]

I am using google console and I earlier had 2.5 TB of data in my DB and hence the auto increase configuration of GCP increased the storage to 2.73 TB. So I am being billed for 2.73 TB storage. ...
vigneshkct118's user avatar
0 votes
1 answer
21 views

gcsfs for google cloud storage throws error

I am trying to export google earth engine data to my google cloud storage bucket using a python tool called ml4floods. It uses gcsfs at its backend for this task. but when I set all necessary ...
Muhammed Omer's user avatar
0 votes
1 answer
64 views

GCS download attempt results in "cannot read property of null" on state.buffer[0].length

I'm trying to download a Google Cloud Storage object to /tmp in a Google Cloud Run service running Node.js. I followed this tutorial and have the following code: exports.resizeImage = async (eventData)...
mightimaus's user avatar
0 votes
0 answers
26 views

How to store file in csv format with header from google cloud cluster to google storage bucket?

I am currently working on a Springboot project which need me to submit a hive job with query to google cluster and retrieve data in a google bucket.However, it seems not accomplishable to contain a ...
Kailiang Chen's user avatar
0 votes
0 answers
33 views

How to connect/use GCS buckets as a catalog and warehouse for Apache Iceberg via pyspark?

I'm completely new to iceberg via pyspark, and I'm having difficulties trying to connect to a gcs bucket in order to store my tables (and use a gcs bucket as a catalog) I've been trying to connect to ...
Osama Iqbal's user avatar
0 votes
1 answer
49 views

How to delete files from Storage with Firebase Functions?

I have Firestore database with documents structure like this startDate - String endDate - String pages - Array(String) and Storage database with files by following path: /pages/fileID I run cron job ...
Heimdallr's user avatar
  • 211
0 votes
0 answers
70 views

FetchError: Invalid response body while trying to fetch https://www.googleapis.com/oauth2/v4/token throwing after upgrading to nodejs 16

FetchError: Invalid response body while trying to fetch https://www.googleapis.com/oauth2/v4/token error is throwing in google-cloud/storage bucket.getFiles() function after upgrading to Nodejs 16. It ...
Bipin Chaudhary's user avatar
0 votes
2 answers
37 views

How to know when a google cloud function has finished processing?

I want to use a Google Cloud Function that triggers when an image is uploaded to a storage bucket - the functional will resize the image and generate a set of images at different sizes to render ...
Zachary Wright's user avatar
0 votes
1 answer
58 views

Google cloud storage bad encoding of UTF-8 files

I have a .txt file encoded in UTF-8 in google cloud storage. When I visit the public URL of the file, I get text with malformed non-ascii(czech) characters. I am sure about the encoding of the file, ...
Tomáš Šíma's user avatar
0 votes
1 answer
39 views

Google cloud storage: How to generate access_key_id and secret_access_key programatically

Currently we are using AWS s3 service. We are giving access of bucket to user by sharing access_key_id and secret_access_key with them. below is our flow to generate these keys. example https://docs....
user2274074's user avatar
  • 1,007
0 votes
2 answers
49 views

What is the IP Network range for Google Cloud Storage service?

Is there a way to retrieve the IP ranges used by the GCP Storage service? When I dig storage.googleapis.com I get a list of IPs that is not included in the range present here. My scenario: I'm ...
Eugen's user avatar
  • 2,310
0 votes
1 answer
150 views

Mount Bucket on Google Storage on login node in google slurm cluster

I'm using the Google Slurm cluster. All the nodes, including the login nodes in the cluster, have mounts to the home directory on the controller. I can my Google bucket with the gcsfuse command to /...
Izik's user avatar
  • 788
0 votes
0 answers
24 views

Is it possible to deny access to a Google Storage bucket to all users/service accounts except for specific few?

I have a Google Storage bucket created via terraform. This bucket has a number of service accounts and user groups that are allowed to view/edit the objects via inherited project permissions. Some of ...
Neara's user avatar
  • 3,693
0 votes
0 answers
30 views

Firebase Firestore doc.data() doesn't return anything

I'm implementing a firebase function that takes metadata from a store in firestore, then get all files of that store folder in storage and compares this data with the one stored in the metadata. If ...
yieniggu's user avatar
  • 394
0 votes
1 answer
56 views

Cannot Connect to Google Cloud Storage from Google Compute Engine Virtual Machine

I have looked at posts with similar issues, telling me to update API permissions, which I have: I have also granted permission to the service agent in my Google Cloud storage bucket: I have tried ...
BX07's user avatar
  • 135
0 votes
0 answers
22 views

Can you apply a lifecycle policy to every item in a GCS Bucket that DOES NOT match a prefix?

I know that GCS supports conditional lifecycle policies. One of the conditions is matchesPrefix. However, I need to do the opposite. I have a bucket that gets a lot of unnecessary metadata, but one ...
Dasph's user avatar
  • 420
0 votes
0 answers
31 views

google cloud storage signed url for download facing CORS error

My react app makes POST request to server /signedUrl for creating a short-lived signed url that can be used to fetch files from firebase storage bucket. My server is configured to allow request from ...
Parvez Mia's user avatar
1 vote
1 answer
24 views

Monitoring firebase storage uploads

I have a flutter app that takes a picture every 5 seconds and stores it under a custom folder inside the bucket. There are 5 instances of this app running in different places, each one uploading the ...
yieniggu's user avatar
  • 394
0 votes
1 answer
67 views

How can I configure the lifecycle management for automatic deletion of logs in Google Cloud Platform's logging storage?

I have a few Google Cloud projects that accumulate a substantial amount of logs, around 200GB per month, and this is expected to nearly double in the next period. Initially, I assumed it would be ...
Borislav Gizdov's user avatar
3 votes
0 answers
80 views

Firebase Storage outage from Norway / Europe? [closed]

Our customers in Norway are currently getting 403 Forbidden on all Cloud Storage reads. This means our service is completely down for these customers. It's working fine for customers in Canada & ...
xaphod's user avatar
  • 6,616
0 votes
0 answers
21 views

Specifying schema in cloud function that uploads csv's to bigquery

'm encountering an issue with a Cloud Function that triggers upon the addition of a new CSV file to my Cloud Storage bucket. These CSV files are downloaded from a third-party database for my company, ...
Lauren Johnston's user avatar
0 votes
2 answers
42 views

load image from google cloud bucket url in flutter

I have a list of products from the api response which has image url for the product. The url is like https://storage.googleapis.com/bucket-1/logos/My%20Product.png but I am not able to access this ...
dev's user avatar
  • 130
0 votes
1 answer
25 views

In Google Cloud Storage, is it possible to change the Bucket Reference from starting with gs:// to data://, http://, or https://?

I'm trying to use links to my files in Google Cloud Storage in external APIs. However, all the links to the files in Google Cloud Storage start with gs:// instead of https://, and the APIs are unable ...
melee's user avatar
  • 61
0 votes
0 answers
11 views

When Uploading Image from Google Cloud to FabricJS Canvas- Tainted canvases may not be exported

How can I resolve the "SecurityError: Failed to execute 'toDataURL' on 'HTMLCanvasElement'" issue in Fabric.js when loading images from Google Cloud Storage onto a canvas? Additionally, how ...
tally's user avatar
  • 1
1 vote
1 answer
65 views

Security rules for firestore and firebase storage - avoid external sign-up

I'm developing a React Web App that uses a Firestore database to store some collections and Firebase storage for images. It's my first time using Firebase. It's time about time to go into production ...
Francisco Venes's user avatar
0 votes
1 answer
23 views

Google Cloud signed URL invalidation

The Google Cloud signed URL documentation says: "Anyone who knows the URL can access the resource until the expiration time for the URL is reached or the key used to sign the URL is rotated."...
Chris Gonzalez's user avatar
1 vote
1 answer
51 views

Bash script to delete Google cloud storage bucket does not work on the list of buckets

I am running the following command in bash to list all google cloud stoarage buckets that matches a specific pattern and delete all of them except one that is created at last. When running the below ...
Ananth's user avatar
  • 99
0 votes
0 answers
14 views

Access public storage from a Firestore client?

I'm writing a Firestore app on iOS and I want to access this public store https://console.cloud.google.com/storage/browser/29f98e10-a489-4c82-ae5e-489dbcd4912f;tab=configuration?prefix=&...
ygini's user avatar
  • 115
0 votes
0 answers
60 views

Cloud Storage - No 'Access-Control-Allow-Origin' header is present on the requested resource

This question has been asked before, but in my case, things are a little different. See, I already ran the following command: gsutil cors set cors-json-file.json gs://(repo) But the Javascript is ...
Roddy P. Carbonell's user avatar
0 votes
1 answer
31 views

Uploading image to firebase storage and getting URL from storage to user database. But photoURL overwritten everything in user database

I tried to upload image to profile page. I successfully uploaded image to storage and got photoURL and updated user database. But after I submit there is only photoURL and other user data like ...
Jaraahai's user avatar
0 votes
0 answers
39 views

Change metadata in google-cloud-storage python library

How to change metadata for the main metadata fields of a file in Google Cloud Storage? I want to modify the primary metadata fields, such as content-disposition or content-encoding. My setup: google-...
Alex_the_Shadowalker's user avatar

1
2 3 4 5
219