Questions tagged [google-bigquery]
Google BigQuery is a Google Cloud Platform product providing serverless queries of petabyte-scale data sets using SQL. BigQuery provides multiple read-write pipelines, and enables data analytics that transform how businesses analyze data.
18,064
questions
0
votes
1answer
24 views
How to bring column data to rows horizontally?
I have a dataset like this:
Number Job Title Date
001 Data analyst 01.01.2018
001 Senior Data Analyst 01.01.2019
001 ...
0
votes
2answers
15 views
How to use JSON_EXTRACT without having a key name?
how do I extract value frm key "Nome" from JSON using JSON_EXTRACT in google bigquery?
I cannot use the key 135 in the query because it is dynamic (Like this JSON_EXTRACT(vista, '$.Agencia....
-2
votes
0answers
27 views
Tell if event in the right order [closed]
I am looking to understand when the dataset from "the Data" is not following the correct sequences marked by the sequence table below
Step Sort
dis 1
prop 2
demo 3
neg 4
...
0
votes
1answer
7 views
How to expand each row in a tableto each distinct value in another table
So I have a a very basic table with one column with several values in it. I have another table (we can call it tableabc the structure is irrelevant). For each row in tableabc I want it exploded to ...
1
vote
3answers
27 views
How to filter table by if any of the columns contains a x value
How to write a where clause that filters any rows that equal to Not or NO.
The challenge is, the data has to many columns, so I need to avoid writing a specific where for each column like where a not ...
0
votes
2answers
32 views
Need help joining incremental data to a fact table in an incremental manor
TableA
ID
Counter
Value
1
1
10
1
2
28
1
3
34
1
4
22
1
5
80
2
1
15
2
2
50
2
3
39
2
4
33
2
5
99
TableB
StartDate
EndDate
2020-01-01
2020-01-11
2020-01-02
2020-01-12
2020-01-03
2020-01-13
...
0
votes
0answers
14 views
BigQuery Timestamp cast to Date regardless of Timezone
I have a table with a Timestamp column in BigQuery. This timestamp comes from systems that are located in different time zones. I don't know what the time zone is for each record and can't add the ...
0
votes
0answers
8 views
BigQuery - array elements as rows
I need to load data from API to BigQuery. Since data returned is deeply nested and has dynamic number of arrays and names (guid), I found it‘s easier to load it as CSV and then pull data using SELECT ...
-2
votes
0answers
34 views
How do I left join in a specific order?
I am trying to join a status history table to show when an item entered and left the state 'Call backs'. The problem I am having is how to join the leaving state in the correct order. This is my code ...
-2
votes
3answers
36 views
Need the last value of a Gap and Island
I am a bit stuck right now, I made good headway on this query, but I am not getting the result I expect. I am using BQ Standard SQL. The Table Looks like this.
ID - Unique ID to an account
Stage - ...
0
votes
1answer
20 views
Cluster table by nested fields
I have worked with partitioned and clustered tables before. I always use simple fields for this, but now I have the necessity to cluster my table by some nested fields. This is a reduced example of my ...
0
votes
0answers
6 views
Issue when migrating from non-partitioned to field partitioned table BigQuery
I was trying to convert a non-partitioned table to field partitioned table. However, when creating the backup table, only few data are copied from the original table. This This would lead to loss of a ...
0
votes
1answer
14 views
How can I find the implementation code for streaming data from Cloud Storage into BiqQuery?
When I try to run the relevant code in Cloud Shell that would allow the streaming function to be deployed it claims that the source folder containing the streaming function itself does not exist.
The ...
0
votes
1answer
19 views
Error when exporting from BigQuery to MySQL
I am trying to export a table from BigQuery to Google Cloud MySQL database.
I found this operator called BigQueryToMySqlOperator (documented here https://airflow.apache.org/docs/apache-airflow-...
1
vote
1answer
22 views
BigQuery User Defined Function Error - Unable to Call UDF In Query
I am trying to use a user defined function that I have created in my BQ dataset.
Here is the UDF:
CREATE OR REPLACE FUNCTION `project.dataset.CountHolidays`(START_DATE DATETIME,
SLA INT64, OPERATION ...
-4
votes
0answers
28 views
Seeking efficient SQL query for max count of record pairs between distinct 'user_ids' of same 'kind' w/in 1 second of each other, for each user_id
The problem:
Table T has fields user_id (STRING), kind (STRING) and time (TIMESTAMP), and has tens of millions of records with timestamps ranging within one hour. Let a pair be any pair of records ...
0
votes
0answers
14 views
Write multiple tables to bigquery by filtering
I am reading from a jdbc connection converting to tablerows and then uploading the data to bigquery with the following code.
The data I am expecting has multiple columns, one of them called "...
-4
votes
0answers
24 views
Please help, stuck trying to build a retention curve [closed]
I have data exported from Google analytics
event_date event_timestamp event_name event_params.key
20210719 1626696283344631 session_start ga_session_id
...
0
votes
0answers
18 views
Pandas to BigQuery upload fails due to InvalidSchema error
I'm using Pands to_gbq to append a dataframe to a big query table as I have done successfully in the past using this (I only explicitly declared one field in the schema so it would recognize it as a ...
-1
votes
1answer
19 views
How to query all tables in dataset and add an identifier?
I use this query
SELECT * FROM `project.DATASET.*`
to select all the data in DATASET,
There is a way to add a new column to identify which table belong each record?
-1
votes
1answer
15 views
MIN() MAX() BigQuery - Unexpected behaviour
The result of querying
SELECT
Type
, val
, MIN(val) over (partition by Type) as min_val
, MAX(val) over (partition by Type) as max_val
FROM tabA
Gives the unexpected output
Type val min_val max_val ...
-1
votes
1answer
47 views
My Web app is being blocked by CORS policy even after installing CORS in my Express app
I have a flutter web app that I am running on localhost to debug, my web app posts data to my Firebase Cloud functions API which then sends data to Google Bigquery to create a table, although I have ...
0
votes
1answer
18 views
How to convert string to float in Google Big Query ? (SAFE_CAST is not working) [duplicate]
I need to convert a column of type string to float so that I could perform mathematical calculations like >, <, = etc on it. I tried SAFE_CAST, but it results nothing. Could anyone please help ...
0
votes
0answers
7 views
Unable to connect Power BI Service to Google BigQuery - Login Loop
We have 2 PowerBI Gateways (on different servers). One was setup about 6 months ago and the other today.
The one setup 6 months ago works fine... we can even delete the BigQuery connection from this ...
0
votes
1answer
15 views
google.cloud.bigquery.Client() ignoring provided scopes, resulting in Permission denied while getting Drive credentials
I am trying to query data stored in Drive via the google.cloud.bigquery Python library.
I've followed Google's guide for Querying Drive Data.
Thus, my code looks like this:
import google.auth
from ...
0
votes
0answers
22 views
Failure to serialize json to table row during Dataflow Job to stream data to BigQuery
I'm using a Dataflow Job template to stream data from a Pub/Sub Subscription to BigQuery. From each JSON file I need to transform the values and output multiple table rows at once to a BQ table. A ...
1
vote
2answers
27 views
Cannot generate date array with adding missing dates and fill with previously known value
Pls.help me to finish query.
I have a table:
select
date(datetime_from) as datetime_from,
listing_id,
price_cents_usd
from `data_marts.pg_listings_log`
with the result:
id
...
1
vote
3answers
43 views
SQL BigQuery: For the current month, count number of distinct CustomerIDs in the previous 3 month
The following is the table with distinct CustomerID and Trunc_Date(Date,MONTH) called Date.
DATE
CustomerID
2021-01-01
111
2021-01-01
112
2021-02-01
111
2021-03-01
113
2021-03-01
115
2021-04-01
...
0
votes
0answers
6 views
BigQuery not storing events from Crashlytics after billing plan change
On 30.06 we have changed our billing status to Blaze (Pay as you go), starting from that date, we don't have any data (events) stored on BigQuery.
This issue is happening on both iOS and Android ...
0
votes
1answer
23 views
SQL - Get unique values by key selected by condition
I want to clean a dataset because there are repeated keys that should not be there. Although the key is repeated, other fields do change. On repetition, I want to keep those entries whose country ...
0
votes
0answers
13 views
BigQuery self-join to add last year, while taking gap years into consideration
I'm trying to add a Last Year column to my revenue table by a self-join with DATE_ADD(LYI.date, INTERVAL 1 YEAR) .
However, on the 28th of February on the year after a gap year (2021), a double record ...
-1
votes
1answer
29 views
creating a new column in bigquery sql using another column
Please consider this example dataset
#standardSQL
WITH abc AS (
SELECT 1234 id, 'Aug' month, 'P' value UNION ALL
SELECT 1234, 'Sept', 'P' UNION ALL
SELECT 3456, 'Aug', 'D' UNION ALL
SELECT ...
-1
votes
2answers
32 views
Selecting top most row in Bigquery based on conditions
I have a huge table, where sometimes 1 product ID has multiple specifications. I want to select the newest but unfortunately, I don't have the date information. please consider this example dataset
...
0
votes
0answers
9 views
Bigquery | How can I create table like google analytics bigquery table
I want to create table like google analytics bigquery below:
Google analytics table info
Google analytics table on the menu
There are lots of tables as each dates. and this tables look like merged.
I ...
-3
votes
2answers
38 views
How to use sql query to find percentage on the basis of certain conditions?
please consider the table below for reference
student name
class
marks
total marks
David
10-B
50
100
Leo
10-B
20
200
Cris
11-B
23
150
Lynn
10-B
100
240
Rachel
11-B
210
500
Ronda
9-B
43
400
So ...
0
votes
0answers
10 views
Exporting BigQuery Audiences from Google Analytics 4 (i.e. GA4) data for remarketing in Google Ads
With BigQuery, I have connected to the GA4 and studied my own website traffic and create propensity scores for the users to predict who will have higher chances to make online purchase.
And now I have ...
0
votes
1answer
11 views
Transpose Rows into column in bigquery
i am having trouble to transpose row into column as i wanted
having data like
rowid error
1 4126
2 26
3 42
4 1451
5 12214126
wanted like
rowid error wanted like
1 4126 41
...
0
votes
0answers
11 views
Does copying a table require the target table to have the same schema?
When copying data from one table to another using bq cp is it a requirement that the two tables have an identical schema?
My reason for asking is that I'm putting together a process to restore a table ...
0
votes
1answer
15 views
Select rows from array of nested objects and remove rows with some matched columns
This is kind of a weirdly specific use case, but I'm struggling to figure out how to only retrieve one row based on some columns within a nested struct, and also remove any subsequent rows that may ...
0
votes
3answers
31 views
Group on Previous Value
I have data that repeats is name but the dates are different. I want to group on the step name and date range but not lose some of the pertinent information if it went backwards a stage.
Currently ...
-1
votes
1answer
20 views
Do they support implicit transactions for single DDL statements (CREATE, DROP, ALTER) in BigQuery?
Do they support implicit transactions for single DDL statements (CREATE, DROP, ALTER) in BigQuery? I know some other Cloud database platform only support subset of DDL statements regarding autocommit ...
0
votes
2answers
22 views
Bigquery delete rows that are present in another table
I want to delete all the rows from table t1 which are present in table t2.
table_1 is as follows
a b c
1 4 3
3 334 3
5 4 5
6 5 4
4 85 3
7 332 54
8 46 6
45 42 5
7 576 6
...
0
votes
0answers
25 views
Data Transfers to Big Query error “Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1”
Hello im currently trying to establish daily data transfers from Google Cloud Storage to Big Query tables. Theses tables are just meant to store raw data (JSON files) and i unnest them later with ...
-1
votes
2answers
48 views
SQL BigQuery: Identifying customers to stopped buying the products
I have the following table where I have DISTINCT CustomerID and Date_Trunc(TIME,MONTH) shown below. If a customer has many transactions within the same month, it's only represented in our table as one ...
0
votes
0answers
12 views
Keep getting Error: Cannot find module '@google-cloud/bigquery' in my Firebase cloud function even after installing it through npm [duplicate]
I keep getting an error when i call my firebase cloud function, when i checked the logs on firebase console it says
Error: Cannot find module '@google-cloud/bigquery'
I installed it using
npm ...
0
votes
1answer
20 views
Bigquery: How to write Multiline SQL UDF?
I would like to create a SQL UDF in bigquery where I could perform certain operations and then return the value.
For eg, In SQL Server, I could create a UDF as follows:
CREATE FUNCTION east_or_west (
...
0
votes
1answer
41 views
Im failing to authenticate Big query from my node.js firebase functions index.js file
I am using the big query API in my firebase functions index.js file, but I keep getting an internal server error when initializing the Big query instance, my first attempt was to include my service ...
0
votes
0answers
12 views
In Big Query, is there a way to take the latest row of data while building a struct over a window?
Hi and thanks for your time
I'm trying to run a process that backfills data between the partition dates of '2021-06-08' and '2021-06-10'.
Each partition requires 10 days of data
I want to produce a ...
0
votes
0answers
15 views
Is it possible to set a query size limit quota for BigQuery ML (BQML)?
It's possible to set query size limits for BigQuery API on project and user-level, see https://cloud.google.com/bigquery/quotas
As I understand it, this includes BQML. The costs between BQ and BQML ...
1
vote
1answer
33 views
Rolling median in bigquery
I have monthly expenditure data in BigQuery for some customers, with the following structure:
CREATE TABLE if not EXISTS monthly_spend (
user_id int,
transaction_month DATE,
spend float
);
...