Questions tagged [bigtable]

Bigtable is a distributed storage system (built by Google) for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers.

Filter by
Sorted by
Tagged with
0 votes
0 answers
51 views

How to find failed inserts while writing to Bigtable using dataflow jobs?

I am writing a pipeline to migrate data from GCS to Bigtable. Data is in json format. Pipeline works fine but number of records written by dataflow job and Count I get from BigQuery using BigTable as ...
0 votes
1 answer
41 views

Key-Value databases that support timestamp-based querying? [closed]

When I worked at Google, I used a Bigtable a lot, and ever since I left I've been missing it. One of the nice things about Bigtable is that its data model had time-based versioning built in. In other ...
  • 26.3k
0 votes
0 answers
35 views

Large Temporal Cache that need to be Cleared on JVM

I have a in house data pipeline that need to process streams of events in a given window (few minutes) at a time. A window contains 10s of GBs of data (at least 30 million records). As internal state ...
  • 7,716
1 vote
0 answers
58 views

How fastly read non contiguous rows from cloud Bigtable from non-google-cloud data center

I have a table exists in Bigtable (BT) with huge data. I have a REST API. During that API execution, I query BT and process the results and send back to client. I was trying to read huge amount of ...
1 vote
0 answers
67 views

cloud big table adds an additional / when setting a hex value

I am trying to set a value in cloud big table using the terminal "cbt". The values are in hex and for some reason it adds an additional / when I submit it. I want it to look like ='\x00\x00\...
  • 11k
1 vote
1 answer
31 views

How to delete record from Bigtable

I have datapipeline in dataflow; and am trying to delete record from bigtable using rowkey. I tried couple of ways using; ex - However am unable to successfully delete record, can i get some sample ...
0 votes
1 answer
181 views

Querying Bigtable row keys by suffix?

I would like to perform some housekeeping on a Bigtable table with a Python script. My row key uses date as a suffix. Is there any way to query Bigtable based on the row key suffix?
0 votes
1 answer
50 views

BigTable - getting total number of rows

I'm using PHP client for working with BigTable instance. Is there a way to fetch total number of rows from the table? So far the solution I found is to use Filter::pass() and manually loop through all ...
  • 2,890
0 votes
1 answer
159 views

springboot with GCP BigTable

i am a new bee to GCP and trying to develop springboot rest api that connects to GCP Bigtable, do we have a quickstart guide that helps in development.
  • 19
1 vote
1 answer
144 views

How does Cloud Bigtable read rows that are non-contiguous?

Given a large number of known row keys. How does bigtable read(not a scan operation) those rows? Does it read the rows one after the other or all at once? If I have a large number of non-contiguous ...
0 votes
1 answer
45 views

Bigtable schema design views

I have a bigtable schema design that's proving a bit hard to scale in terms of reads. I would appreciate a quick look and some advice on bad design patterns. Here's a quick overview: screenshot of ...
0 votes
1 answer
72 views

Array Aggregation in Bigquery with Run Length Encoded Arrays

I've got a BigQuery table that looks something like this: -------------------------------------------------------- | col1 | cols 2-10 | col 11 | -----------------------------...
  • 198
0 votes
1 answer
87 views

Migrate BigTable table in PostgreSQL table

I want to move out of BigTable because it is costly. I have one table of 5GB and want to migrate it to a PostgreSQL database. So far I have found ways to migrate data in BigTable but not out of it, ...
  • 11
0 votes
1 answer
185 views

Batch delete BigTable tables and BigQuery datasets

I searched around to find a way to Batch delete BigTable tables and BigQuery datasets (using python's library) without any luck up to now. Is anyone aware of an efficient way to do that? I looked into ...
  • 288
0 votes
1 answer
34 views

How can I change the grpc.max_receive_message_length configuration in Google bigtable (nodejs)?

I am getting 8 RESOURCE_EXHAUSTED: Received message larger than max (5014229 vs. 4194304) when I call the following get method. const [ row ] = await table.row(key).get(); I've tried to pass in grpc....
  • 1
0 votes
0 answers
135 views

Adding multiple column families in Bigtable from an Apache Beam pipeline

I have this beam pipeline where I am trying to read stuff from BigQuery and put it in Bigtable. With the Dataflow Connector for Bigtable, it is not possible to create new column families on the fly. ...
-1 votes
1 answer
84 views

How to setup staging, pre-prod for google dataflow jobs?

Say we have a dataflow job: Written in Apache Beam Java SDK which is a Gradle project. Uses pubsub stream as input, writes results to bigtable and writes logs to BigQuery. As with deploying a server,...
1 vote
1 answer
200 views

when to use BigTableIO.write() API vs CloudBigtableIO.CloudBigtableSingleTableBufferedWriteFn API

I see two APIs available for writing to GCP BigTable : BigTableIO.write() vs CloudBigtableIO.CloudBigtableSingleTableBufferedWriteFn. I am working on a Dataflow pipeline to read data from one ...
0 votes
1 answer
189 views

Schema design for Google BigTable

In my project, Im using Google BigQuery that holds loots of data. The BigQuery columns are: account_id, session_id, transaction_id, username, event, timestamp. In my dashboard, Im fetching the entire ...
  • 71
0 votes
1 answer
282 views

How to write to BigTable using Apache Beam direct-runner in java?

I have been trying to get Apache Beam direct runner to write to BigTable but it seems like there is a problem. There is no failure or confirmation errors on the terminal when I run gradle run. My ...
0 votes
2 answers
231 views

hbase using google cloud storage (bucket)

I'm new to hbase. New to explore few options. Can I use gcs file system as backing store for hbase on google cloud? I know there's a google's big table, assuming setting up our own cluster with gcs ...
0 votes
0 answers
69 views

Bigtable test garbage collection in emulator & get most recent cell in a row

I'm wanting to test the garbage collection in Bigtable emulator. When I set the GCRULES.maxAge(Duration.ofSeconds(1)) My cells didn't deleted when I readRowAsync My schema: user_id as row_key, events ...
  • 61
0 votes
2 answers
68 views

BigTable data schema design

I am learning about BigTable, and trying to design a good schema for it user there will be an ID(unique), in time this user will receive many events(these events don't have an Id, only the timestamp ...
  • 61
0 votes
1 answer
123 views

Deciding whether to use column versions or new rows in GCP Bigtable

Let’s say I want to store a snapshot of a User’s Settings every time they make a change. These snapshots would allow the User to see a history of the changes they made to their settings. Would it be ...
  • 59
0 votes
1 answer
120 views

Regarding bigtable row transactions

I would like to understand how BigTable row level transactions work for the following scenario? Lets say row has the key 'k' and the value as set of elements {E1, E2} case a) Request A read the row ...
1 vote
0 answers
66 views

Does Google Bigtable has support for stored procedures?

Is there support for stored procedures in Google Bigtable, or support for any similar mechanism for saving multi-statement queries?
1 vote
0 answers
180 views

Cloud Bigtable: CheckAndMutateRow : row filter is failing with errors even after using ExactValueFilter

My goal is to update the existing value only if there is any change with the column value. Even though I am using ExactValueFilter still it is considered as a regex and failing with ) missing in my ...
1 vote
1 answer
359 views

BigTable: will column qualifier or timestamp filter cause the full scan?

Say I have a table that contains billions of records. On a particular day there are only 7 records though. If I filter it by timestamp range (as described here), will it cause the full table scan? ...
  • 2,754
0 votes
1 answer
212 views

Bigtable InstantiatingGrpcChannelProvider does not define or inherit an implementation of the resolved method

In JDK 11, the following code: CredentialsProvider credentialsProvider = FixedCredentialsProvider.create( ServiceAccountCredentials.fromStream(new FileInputStream("my_sa.json"))); ...
  • 1,212
1 vote
0 answers
287 views

Slow data reading from Google BigTable

Airflow 1.10.14 and composer 1.15.2, google Bigtable in GCP I'm getting this issue {taskinstance.py:1152} ERROR - <_MultiThreadedRendezvous of RPC that terminated with status = StatusCode....
0 votes
1 answer
244 views

The "Parquet Files on Cloud Storage to Cloud Bigtable" DataFlow template cannot read parquet files

I'm trying to move a parquet file that is written out in R using the arrow library to BigTable. I have validated the arrow package instalation and made sure that the snappy codec is available using ...
  • 3,687
0 votes
2 answers
335 views

Bigtable schema - multiple columns or rows?

I'm designing a Bigtable schema, where I am trying to optimize read performance. I'm looking for some advice on which of these 2 options would perform better: Single row, with multiple columns (...
  • 97
0 votes
1 answer
278 views

Python: deleting a row from Bigtable by its row-key

this is simple: I want to delete a row, by its row-key, from Bigtable, using Python. The documentation for this is woefully lacking. How is this done?
  • 7,245
0 votes
1 answer
633 views

How to write BigTable table data into a pandas dataframe?

I am trying to read a GCP BigTable - table to a pandas dataframe, and currently, the function I am using to fetch rows from BigTable is read_rows(), which returns PartialRowData. Code: from google....
1 vote
1 answer
559 views

How to connect to GCP BigTable using Python

I am connecting to my GCP BigTable instance using python (google-cloud-bigtable library), after setting up "GOOGLE_APPLICATION_CREDENTIALS" in my environment variables. I'm successful at ...
0 votes
2 answers
178 views

How does Bigtable handle network partition?

Bigtable is listed as CP system below NoSQL: What does it mean for MongoDB or BigTable to not always be "Available" This means all read gets the latest write and the system functions during ...
0 votes
1 answer
308 views

BigTable schema design

Playing around with BigTable and trying to understand if my schema design has flaws. We have thousands of sensors that are submitting data with different frequencies. Between once a second and every ...
  • 1,972
1 vote
1 answer
249 views

Which is better performant bigtable schema: Single column multi cells vs Multi columns single cell?

I need to store user interactions for 7 days in an existing BigTable table whose row key is user identifier. There are two types of interactions and we should be able to retrieve interaction history ...
  • 811
0 votes
1 answer
165 views

Error With RowKey Definition on Confluent BigTable Sink Connector

I'm trying to use the BigTable Sink Connector from Confluent to read data from kafka and write it into my BigTable Instance, but I'm receiving the following message error: org.apache.kafka.connect....
0 votes
1 answer
114 views

Label cells within Bigtable

How to label cell value with two labels ? Is it possible or is there a limitation. For example label the cell value with x,y. How to label cell value with two labels ? Is it possible or is there a ...
1 vote
0 answers
54 views

decode snappy messages using Bigtable cbt

I have some data in bigtable, but the messages are all snappy encoded. Does anyone know a way to decode it? cbt -project project-1 -instance test read events count=10 prefix=test123 This gives me ...
  • 133
0 votes
1 answer
823 views

Querying BigTable data with BigQuery: how to make a correct table definition file

I try to set up external tables for BigQuery, the source is BigTable. I can create a table but while executing a sql statment i get this error: Error while reading data, error message: Error detected ...
  • 21
0 votes
1 answer
97 views

Storing multiple cells from one row vs one cell and multiple contiguous rows in bigtable

What is the pros and cons in storing multiple cells in a single row key vs multiple rows each with one cell in bigtable? For example is there benefit of storing columnFamily1 ...
0 votes
1 answer
506 views

GCP moving Bigtable data to BigQuery

In GCP I would like to know if it possible to transfer/move data from Bigtable to BigQuery. i.e lets say I want all data greater than 1 year should be moved from Bigtable to BigQuery. Is this doable? ...
  • 4,686
0 votes
1 answer
1k views

What is column family in gcp bigtable and how is data stored in bigtable?

In cassandra column families are just table What's the difference between creating a table and creating a columnfamily in Cassandra? but it seems like column family is referring to something else ...
1 vote
1 answer
346 views

Bigtable: mutating multiple values into single column in a single column family

I've tried to insert 100 values into a column test_col in column family test_cf in a row key test-123 The problem is that I successfully inserted 100 values into Bigtable. However, The number of ...
  • 700
1 vote
1 answer
944 views

Can I read bigtable records with cbt by time?

I want to read the last (latest) records written to bigtable with cbt. The docs don't mention which order cbt read reads records in, however. I don't know what the row key(s) would be. Is there a ...
  • 9,437
0 votes
1 answer
329 views

Google Cloud Bigtable: Storing multiple rows for a row key

In Bigtable, I am trying to create a column family corresponding to a row key in this format shown below. Under the preferences column, there are multiple cells. Note that these are not multiple ...
0 votes
1 answer
182 views

Is there any way to backup all the tables from bigtable instance at once?

we have created a single instance and we want to backup all the tables in that single instance using gcloud command as we have created a pipeline to do that backup thing but while running the backup ...
1 vote
1 answer
285 views

Get last row of a range in bigtable?

I would like to get the last row of a range in Bigtable. Example: key: prefix:00000001 prefix:00000002 prefix:00000003 What would be the best way to get prefix:00000003? Thanks for your help
  • 94

1
2 3 4 5
11