Questions tagged [bigtable]
Bigtable is a distributed storage system (built by Google) for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers.
513
questions
0
votes
0
answers
51
views
How to find failed inserts while writing to Bigtable using dataflow jobs?
I am writing a pipeline to migrate data from GCS to Bigtable. Data is in json format. Pipeline works fine but number of records written by dataflow job and Count I get from BigQuery using BigTable as ...
0
votes
1
answer
41
views
Key-Value databases that support timestamp-based querying? [closed]
When I worked at Google, I used a Bigtable a lot, and ever since I left I've been missing it. One of the nice things about Bigtable is that its data model had time-based versioning built in. In other ...
0
votes
0
answers
35
views
Large Temporal Cache that need to be Cleared on JVM
I have a in house data pipeline that need to process streams of events in a given window (few minutes) at a time. A window contains 10s of GBs of data (at least 30 million records). As internal state ...
1
vote
0
answers
58
views
How fastly read non contiguous rows from cloud Bigtable from non-google-cloud data center
I have a table exists in Bigtable (BT) with huge data. I have a REST API. During that API execution, I query BT and process the results and send back to client. I was trying to read huge amount of ...
1
vote
0
answers
67
views
cloud big table adds an additional / when setting a hex value
I am trying to set a value in cloud big table using the terminal "cbt". The values are in hex and for some reason it adds an additional / when I submit it. I want it to look like
='\x00\x00\...
1
vote
1
answer
31
views
How to delete record from Bigtable
I have datapipeline in dataflow; and am trying to delete record from bigtable using rowkey.
I tried couple of ways using; ex -
However am unable to successfully delete record, can i get some sample ...
0
votes
1
answer
181
views
Querying Bigtable row keys by suffix?
I would like to perform some housekeeping on a Bigtable table with a Python script. My row key uses date as a suffix. Is there any way to query Bigtable based on the row key suffix?
0
votes
1
answer
50
views
BigTable - getting total number of rows
I'm using PHP client for working with BigTable instance. Is there a way to fetch total number of rows from the table?
So far the solution I found is to use Filter::pass() and manually loop through all ...
0
votes
1
answer
159
views
springboot with GCP BigTable
i am a new bee to GCP and trying to develop springboot rest api that connects to GCP Bigtable, do we have a quickstart guide that helps in development.
1
vote
1
answer
144
views
How does Cloud Bigtable read rows that are non-contiguous?
Given a large number of known row keys. How does bigtable read(not a scan operation) those rows? Does it read the rows one after the other or all at once? If I have a large number of non-contiguous ...
0
votes
1
answer
45
views
Bigtable schema design views
I have a bigtable schema design that's proving a bit hard to scale in terms of reads.
I would appreciate a quick look and some advice on bad design patterns.
Here's a quick overview:
screenshot of ...
0
votes
1
answer
72
views
Array Aggregation in Bigquery with Run Length Encoded Arrays
I've got a BigQuery table that looks something like this:
--------------------------------------------------------
| col1 | cols 2-10 | col 11 |
-----------------------------...
0
votes
1
answer
87
views
Migrate BigTable table in PostgreSQL table
I want to move out of BigTable because it is costly. I have one table of 5GB and want to migrate it to a PostgreSQL database. So far I have found ways to migrate data in BigTable but not out of it, ...
0
votes
1
answer
185
views
Batch delete BigTable tables and BigQuery datasets
I searched around to find a way to Batch delete BigTable tables and BigQuery datasets (using python's library) without any luck up to now.
Is anyone aware of an efficient way to do that?
I looked into ...
0
votes
1
answer
34
views
How can I change the grpc.max_receive_message_length configuration in Google bigtable (nodejs)?
I am getting 8 RESOURCE_EXHAUSTED: Received message larger than max (5014229 vs. 4194304) when I call the following get method.
const [ row ] = await table.row(key).get();
I've tried to pass in grpc....
0
votes
0
answers
135
views
Adding multiple column families in Bigtable from an Apache Beam pipeline
I have this beam pipeline where I am trying to read stuff from BigQuery and put it in Bigtable. With the Dataflow Connector for Bigtable, it is not possible to create new column families on the fly.
...
-1
votes
1
answer
84
views
How to setup staging, pre-prod for google dataflow jobs?
Say we have a dataflow job:
Written in Apache Beam Java SDK which is a Gradle project.
Uses pubsub stream as input, writes results to bigtable and writes logs to BigQuery.
As with deploying a server,...
1
vote
1
answer
200
views
when to use BigTableIO.write() API vs CloudBigtableIO.CloudBigtableSingleTableBufferedWriteFn API
I see two APIs available for writing to GCP BigTable : BigTableIO.write() vs CloudBigtableIO.CloudBigtableSingleTableBufferedWriteFn. I am working on a Dataflow pipeline to read data from one ...
0
votes
1
answer
189
views
Schema design for Google BigTable
In my project, Im using Google BigQuery that holds loots of data.
The BigQuery columns are:
account_id, session_id, transaction_id, username, event, timestamp.
In my dashboard, Im fetching the entire ...
0
votes
1
answer
282
views
How to write to BigTable using Apache Beam direct-runner in java?
I have been trying to get Apache Beam direct runner to write to BigTable but it seems like there is a problem.
There is no failure or confirmation errors on the terminal when I run gradle run.
My ...
0
votes
2
answers
231
views
hbase using google cloud storage (bucket)
I'm new to hbase.
New to explore few options.
Can I use gcs file system as backing store for hbase on google cloud?
I know there's a google's big table, assuming setting up our own cluster with gcs ...
0
votes
0
answers
69
views
Bigtable test garbage collection in emulator & get most recent cell in a row
I'm wanting to test the garbage collection in Bigtable emulator.
When I set the GCRULES.maxAge(Duration.ofSeconds(1))
My cells didn't deleted when I readRowAsync
My schema:
user_id as row_key,
events ...
0
votes
2
answers
68
views
BigTable data schema design
I am learning about BigTable, and trying to design a good schema for it
user there will be an ID(unique), in time this user will receive many events(these events don't have an Id, only the timestamp ...
0
votes
1
answer
123
views
Deciding whether to use column versions or new rows in GCP Bigtable
Let’s say I want to store a snapshot of a User’s Settings every time they make a change. These snapshots would allow the User to see a history of the changes they made to their settings.
Would it be ...
0
votes
1
answer
120
views
Regarding bigtable row transactions
I would like to understand how BigTable row level transactions work for the following scenario?
Lets say row has the key 'k' and the value as set of elements {E1, E2}
case a) Request A read the row ...
1
vote
0
answers
66
views
Does Google Bigtable has support for stored procedures?
Is there support for stored procedures in Google Bigtable, or support for any similar mechanism for saving multi-statement queries?
1
vote
0
answers
180
views
Cloud Bigtable: CheckAndMutateRow : row filter is failing with errors even after using ExactValueFilter
My goal is to update the existing value only if there is any change with the column value. Even though I am using ExactValueFilter still it is considered as a regex and failing with ) missing in my ...
1
vote
1
answer
359
views
BigTable: will column qualifier or timestamp filter cause the full scan?
Say I have a table that contains billions of records.
On a particular day there are only 7 records though. If I filter it by timestamp range (as described here), will it cause the full table scan?
...
0
votes
1
answer
212
views
Bigtable InstantiatingGrpcChannelProvider does not define or inherit an implementation of the resolved method
In JDK 11, the following code:
CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(
ServiceAccountCredentials.fromStream(new FileInputStream("my_sa.json")));
...
1
vote
0
answers
287
views
Slow data reading from Google BigTable
Airflow 1.10.14 and composer 1.15.2, google Bigtable in GCP
I'm getting this issue
{taskinstance.py:1152} ERROR - <_MultiThreadedRendezvous of RPC that terminated with
status = StatusCode....
0
votes
1
answer
244
views
The "Parquet Files on Cloud Storage to Cloud Bigtable" DataFlow template cannot read parquet files
I'm trying to move a parquet file that is written out in R using the arrow library to BigTable. I have validated the arrow package instalation and made sure that the snappy codec is available using ...
0
votes
2
answers
335
views
Bigtable schema - multiple columns or rows?
I'm designing a Bigtable schema, where I am trying to optimize read performance. I'm looking for some advice on which of these 2 options would perform better:
Single row, with multiple columns (...
0
votes
1
answer
278
views
Python: deleting a row from Bigtable by its row-key
this is simple: I want to delete a row, by its row-key, from Bigtable, using Python. The documentation for this is woefully lacking. How is this done?
0
votes
1
answer
633
views
How to write BigTable table data into a pandas dataframe?
I am trying to read a GCP BigTable - table to a pandas dataframe, and currently, the function I am using to fetch rows from BigTable is read_rows(), which returns PartialRowData.
Code:
from google....
1
vote
1
answer
559
views
How to connect to GCP BigTable using Python
I am connecting to my GCP BigTable instance using python (google-cloud-bigtable library), after setting up "GOOGLE_APPLICATION_CREDENTIALS" in my environment variables. I'm successful at ...
0
votes
2
answers
178
views
How does Bigtable handle network partition?
Bigtable is listed as CP system below
NoSQL: What does it mean for MongoDB or BigTable to not always be "Available"
This means all read gets the latest write and the system functions during ...
0
votes
1
answer
308
views
BigTable schema design
Playing around with BigTable and trying to understand if my schema design has flaws.
We have thousands of sensors that are submitting data with different frequencies. Between once a second and every ...
1
vote
1
answer
249
views
Which is better performant bigtable schema: Single column multi cells vs Multi columns single cell?
I need to store user interactions for 7 days in an existing BigTable table whose row key is user identifier. There are two types of interactions and we should be able to retrieve interaction history ...
0
votes
1
answer
165
views
Error With RowKey Definition on Confluent BigTable Sink Connector
I'm trying to use the BigTable Sink Connector from Confluent to read data from kafka and write it into my BigTable Instance, but I'm receiving the following message error:
org.apache.kafka.connect....
0
votes
1
answer
114
views
Label cells within Bigtable
How to label cell value with two labels ? Is it possible or is there a limitation. For example label the cell value with x,y.
How to label cell value with two labels ? Is it possible or is there a ...
1
vote
0
answers
54
views
decode snappy messages using Bigtable cbt
I have some data in bigtable, but the messages are all snappy encoded. Does anyone know a way to decode it?
cbt -project project-1 -instance test read events count=10 prefix=test123
This gives me ...
0
votes
1
answer
823
views
Querying BigTable data with BigQuery: how to make a correct table definition file
I try to set up external tables for BigQuery, the source is BigTable. I can create a table but while executing a sql statment i get this error:
Error while reading data, error message: Error detected ...
0
votes
1
answer
97
views
Storing multiple cells from one row vs one cell and multiple contiguous rows in bigtable
What is the pros and cons in storing multiple cells in a single row key vs multiple rows each with one cell in bigtable?
For example is there benefit of storing
columnFamily1 ...
0
votes
1
answer
506
views
GCP moving Bigtable data to BigQuery
In GCP I would like to know if it possible to transfer/move data from Bigtable to BigQuery. i.e lets say I want all data greater than 1 year should be moved from Bigtable to BigQuery. Is this doable?
...
0
votes
1
answer
1k
views
What is column family in gcp bigtable and how is data stored in bigtable?
In cassandra column families are just table
What's the difference between creating a table and creating a columnfamily in Cassandra?
but it seems like column family is referring to something else ...
1
vote
1
answer
346
views
Bigtable: mutating multiple values into single column in a single column family
I've tried to insert 100 values into a column test_col in column family test_cf in a row key test-123
The problem is that I successfully inserted 100 values into Bigtable.
However, The number of ...
1
vote
1
answer
944
views
Can I read bigtable records with cbt by time?
I want to read the last (latest) records written to bigtable with cbt. The docs don't mention which order cbt read reads records in, however. I don't know what the row key(s) would be.
Is there a ...
0
votes
1
answer
329
views
Google Cloud Bigtable: Storing multiple rows for a row key
In Bigtable, I am trying to create a column family corresponding to a row key in this format shown below.
Under the preferences column, there are multiple cells. Note that these are not multiple ...
0
votes
1
answer
182
views
Is there any way to backup all the tables from bigtable instance at once?
we have created a single instance and we want to backup all the tables in that single instance using gcloud command as we have created a pipeline to do that backup thing but while running the backup ...
1
vote
1
answer
285
views
Get last row of a range in bigtable?
I would like to get the last row of a range in Bigtable. Example:
key:
prefix:00000001
prefix:00000002
prefix:00000003
What would be the best way to get prefix:00000003? Thanks for your help