Questions tagged [google-bigquery]

Google BigQuery is a Google Cloud Platform product providing serverless queries of petabyte-scale data sets using SQL. BigQuery provides multiple read-write pipelines, and enables data analytics that transform how businesses analyze data.

Filter by
Sorted by
Tagged with
0
votes
1answer
24 views

How to bring column data to rows horizontally?

I have a dataset like this: Number Job Title Date 001 Data analyst 01.01.2018 001 Senior Data Analyst 01.01.2019 001 ...
0
votes
2answers
15 views

How to use JSON_EXTRACT without having a key name?

how do I extract value frm key "Nome" from JSON using JSON_EXTRACT in google bigquery? I cannot use the key 135 in the query because it is dynamic (Like this JSON_EXTRACT(vista, '$.Agencia....
-2
votes
0answers
27 views

Tell if event in the right order [closed]

I am looking to understand when the dataset from "the Data" is not following the correct sequences marked by the sequence table below Step Sort dis 1 prop 2 demo 3 neg 4 ...
0
votes
1answer
7 views

How to expand each row in a tableto each distinct value in another table

So I have a a very basic table with one column with several values in it. I have another table (we can call it tableabc the structure is irrelevant). For each row in tableabc I want it exploded to ...
1
vote
3answers
27 views

How to filter table by if any of the columns contains a x value

How to write a where clause that filters any rows that equal to Not or NO. The challenge is, the data has to many columns, so I need to avoid writing a specific where for each column like where a not ...
0
votes
2answers
32 views

Need help joining incremental data to a fact table in an incremental manor

TableA ID Counter Value 1 1 10 1 2 28 1 3 34 1 4 22 1 5 80 2 1 15 2 2 50 2 3 39 2 4 33 2 5 99 TableB StartDate EndDate 2020-01-01 2020-01-11 2020-01-02 2020-01-12 2020-01-03 2020-01-13 ...
0
votes
0answers
14 views

BigQuery Timestamp cast to Date regardless of Timezone

I have a table with a Timestamp column in BigQuery. This timestamp comes from systems that are located in different time zones. I don't know what the time zone is for each record and can't add the ...
0
votes
0answers
8 views

BigQuery - array elements as rows

I need to load data from API to BigQuery. Since data returned is deeply nested and has dynamic number of arrays and names (guid), I found it‘s easier to load it as CSV and then pull data using SELECT ...
-2
votes
0answers
34 views

How do I left join in a specific order?

I am trying to join a status history table to show when an item entered and left the state 'Call backs'. The problem I am having is how to join the leaving state in the correct order. This is my code ...
-2
votes
3answers
36 views

Need the last value of a Gap and Island

I am a bit stuck right now, I made good headway on this query, but I am not getting the result I expect. I am using BQ Standard SQL. The Table Looks like this. ID - Unique ID to an account Stage - ...
0
votes
1answer
20 views

Cluster table by nested fields

I have worked with partitioned and clustered tables before. I always use simple fields for this, but now I have the necessity to cluster my table by some nested fields. This is a reduced example of my ...
0
votes
0answers
6 views

Issue when migrating from non-partitioned to field partitioned table BigQuery

I was trying to convert a non-partitioned table to field partitioned table. However, when creating the backup table, only few data are copied from the original table. This This would lead to loss of a ...
0
votes
1answer
14 views

How can I find the implementation code for streaming data from Cloud Storage into BiqQuery?

When I try to run the relevant code in Cloud Shell that would allow the streaming function to be deployed it claims that the source folder containing the streaming function itself does not exist. The ...
0
votes
1answer
19 views

Error when exporting from BigQuery to MySQL

I am trying to export a table from BigQuery to Google Cloud MySQL database. I found this operator called BigQueryToMySqlOperator (documented here https://airflow.apache.org/docs/apache-airflow-...
1
vote
1answer
22 views

BigQuery User Defined Function Error - Unable to Call UDF In Query

I am trying to use a user defined function that I have created in my BQ dataset. Here is the UDF: CREATE OR REPLACE FUNCTION `project.dataset.CountHolidays`(START_DATE DATETIME, SLA INT64, OPERATION ...
-4
votes
0answers
28 views

Seeking efficient SQL query for max count of record pairs between distinct 'user_ids' of same 'kind' w/in 1 second of each other, for each user_id

The problem: Table T has fields user_id (STRING), kind (STRING) and time (TIMESTAMP), and has tens of millions of records with timestamps ranging within one hour. Let a pair be any pair of records ...
0
votes
0answers
14 views

Write multiple tables to bigquery by filtering

I am reading from a jdbc connection converting to tablerows and then uploading the data to bigquery with the following code. The data I am expecting has multiple columns, one of them called "...
-4
votes
0answers
24 views

Please help, stuck trying to build a retention curve [closed]

I have data exported from Google analytics event_date event_timestamp event_name event_params.key 20210719 1626696283344631 session_start ga_session_id ...
0
votes
0answers
18 views

Pandas to BigQuery upload fails due to InvalidSchema error

I'm using Pands to_gbq to append a dataframe to a big query table as I have done successfully in the past using this (I only explicitly declared one field in the schema so it would recognize it as a ...
-1
votes
1answer
19 views

How to query all tables in dataset and add an identifier?

I use this query SELECT * FROM `project.DATASET.*` to select all the data in DATASET, There is a way to add a new column to identify which table belong each record?
-1
votes
1answer
15 views

MIN() MAX() BigQuery - Unexpected behaviour

The result of querying SELECT Type , val , MIN(val) over (partition by Type) as min_val , MAX(val) over (partition by Type) as max_val FROM tabA Gives the unexpected output Type val min_val max_val ...
-1
votes
1answer
47 views

My Web app is being blocked by CORS policy even after installing CORS in my Express app

I have a flutter web app that I am running on localhost to debug, my web app posts data to my Firebase Cloud functions API which then sends data to Google Bigquery to create a table, although I have ...
0
votes
1answer
18 views

How to convert string to float in Google Big Query ? (SAFE_CAST is not working) [duplicate]

I need to convert a column of type string to float so that I could perform mathematical calculations like >, <, = etc on it. I tried SAFE_CAST, but it results nothing. Could anyone please help ...
0
votes
0answers
7 views

Unable to connect Power BI Service to Google BigQuery - Login Loop

We have 2 PowerBI Gateways (on different servers). One was setup about 6 months ago and the other today. The one setup 6 months ago works fine... we can even delete the BigQuery connection from this ...
0
votes
1answer
15 views

google.cloud.bigquery.Client() ignoring provided scopes, resulting in Permission denied while getting Drive credentials

I am trying to query data stored in Drive via the google.cloud.bigquery Python library. I've followed Google's guide for Querying Drive Data. Thus, my code looks like this: import google.auth from ...
0
votes
0answers
22 views

Failure to serialize json to table row during Dataflow Job to stream data to BigQuery

I'm using a Dataflow Job template to stream data from a Pub/Sub Subscription to BigQuery. From each JSON file I need to transform the values and output multiple table rows at once to a BQ table. A ...
1
vote
2answers
27 views

Cannot generate date array with adding missing dates and fill with previously known value

Pls.help me to finish query. I have a table: select date(datetime_from) as datetime_from, listing_id, price_cents_usd from `data_marts.pg_listings_log` with the result: id ...
1
vote
3answers
43 views

SQL BigQuery: For the current month, count number of distinct CustomerIDs in the previous 3 month

The following is the table with distinct CustomerID and Trunc_Date(Date,MONTH) called Date. DATE CustomerID 2021-01-01 111 2021-01-01 112 2021-02-01 111 2021-03-01 113 2021-03-01 115 2021-04-01 ...
0
votes
0answers
6 views

BigQuery not storing events from Crashlytics after billing plan change

On 30.06 we have changed our billing status to Blaze (Pay as you go), starting from that date, we don't have any data (events) stored on BigQuery. This issue is happening on both iOS and Android ...
0
votes
1answer
23 views

SQL - Get unique values by key selected by condition

I want to clean a dataset because there are repeated keys that should not be there. Although the key is repeated, other fields do change. On repetition, I want to keep those entries whose country ...
0
votes
0answers
13 views

BigQuery self-join to add last year, while taking gap years into consideration

I'm trying to add a Last Year column to my revenue table by a self-join with DATE_ADD(LYI.date, INTERVAL 1 YEAR) . However, on the 28th of February on the year after a gap year (2021), a double record ...
-1
votes
1answer
29 views

creating a new column in bigquery sql using another column

Please consider this example dataset #standardSQL WITH abc AS ( SELECT 1234 id, 'Aug' month, 'P' value UNION ALL SELECT 1234, 'Sept', 'P' UNION ALL SELECT 3456, 'Aug', 'D' UNION ALL SELECT ...
-1
votes
2answers
32 views

Selecting top most row in Bigquery based on conditions

I have a huge table, where sometimes 1 product ID has multiple specifications. I want to select the newest but unfortunately, I don't have the date information. please consider this example dataset ...
0
votes
0answers
9 views

Bigquery | How can I create table like google analytics bigquery table

I want to create table like google analytics bigquery below: Google analytics table info Google analytics table on the menu There are lots of tables as each dates. and this tables look like merged. I ...
-3
votes
2answers
38 views

How to use sql query to find percentage on the basis of certain conditions?

please consider the table below for reference student name class marks total marks David 10-B 50 100 Leo 10-B 20 200 Cris 11-B 23 150 Lynn 10-B 100 240 Rachel 11-B 210 500 Ronda 9-B 43 400 So ...
0
votes
0answers
10 views

Exporting BigQuery Audiences from Google Analytics 4 (i.e. GA4) data for remarketing in Google Ads

With BigQuery, I have connected to the GA4 and studied my own website traffic and create propensity scores for the users to predict who will have higher chances to make online purchase. And now I have ...
0
votes
1answer
11 views

Transpose Rows into column in bigquery

i am having trouble to transpose row into column as i wanted having data like rowid error 1 4126 2 26 3 42 4 1451 5 12214126 wanted like rowid error wanted like 1 4126 41 ...
0
votes
0answers
11 views

Does copying a table require the target table to have the same schema?

When copying data from one table to another using bq cp is it a requirement that the two tables have an identical schema? My reason for asking is that I'm putting together a process to restore a table ...
0
votes
1answer
15 views

Select rows from array of nested objects and remove rows with some matched columns

This is kind of a weirdly specific use case, but I'm struggling to figure out how to only retrieve one row based on some columns within a nested struct, and also remove any subsequent rows that may ...
0
votes
3answers
31 views

Group on Previous Value

I have data that repeats is name but the dates are different. I want to group on the step name and date range but not lose some of the pertinent information if it went backwards a stage. Currently ...
-1
votes
1answer
20 views

Do they support implicit transactions for single DDL statements (CREATE, DROP, ALTER) in BigQuery?

Do they support implicit transactions for single DDL statements (CREATE, DROP, ALTER) in BigQuery? I know some other Cloud database platform only support subset of DDL statements regarding autocommit ...
0
votes
2answers
22 views

Bigquery delete rows that are present in another table

I want to delete all the rows from table t1 which are present in table t2. table_1 is as follows a b c 1 4 3 3 334 3 5 4 5 6 5 4 4 85 3 7 332 54 8 46 6 45 42 5 7 576 6 ...
0
votes
0answers
25 views

Data Transfers to Big Query error “Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1”

Hello im currently trying to establish daily data transfers from Google Cloud Storage to Big Query tables. Theses tables are just meant to store raw data (JSON files) and i unnest them later with ...
-1
votes
2answers
48 views

SQL BigQuery: Identifying customers to stopped buying the products

I have the following table where I have DISTINCT CustomerID and Date_Trunc(TIME,MONTH) shown below. If a customer has many transactions within the same month, it's only represented in our table as one ...
0
votes
0answers
12 views

Keep getting Error: Cannot find module '@google-cloud/bigquery' in my Firebase cloud function even after installing it through npm [duplicate]

I keep getting an error when i call my firebase cloud function, when i checked the logs on firebase console it says Error: Cannot find module '@google-cloud/bigquery' I installed it using npm ...
0
votes
1answer
20 views

Bigquery: How to write Multiline SQL UDF?

I would like to create a SQL UDF in bigquery where I could perform certain operations and then return the value. For eg, In SQL Server, I could create a UDF as follows: CREATE FUNCTION east_or_west ( ...
0
votes
1answer
41 views

Im failing to authenticate Big query from my node.js firebase functions index.js file

I am using the big query API in my firebase functions index.js file, but I keep getting an internal server error when initializing the Big query instance, my first attempt was to include my service ...
0
votes
0answers
12 views

In Big Query, is there a way to take the latest row of data while building a struct over a window?

Hi and thanks for your time I'm trying to run a process that backfills data between the partition dates of '2021-06-08' and '2021-06-10'. Each partition requires 10 days of data I want to produce a ...
0
votes
0answers
15 views

Is it possible to set a query size limit quota for BigQuery ML (BQML)?

It's possible to set query size limits for BigQuery API on project and user-level, see https://cloud.google.com/bigquery/quotas As I understand it, this includes BQML. The costs between BQ and BQML ...
1
vote
1answer
33 views

Rolling median in bigquery

I have monthly expenditure data in BigQuery for some customers, with the following structure: CREATE TABLE if not EXISTS monthly_spend ( user_id int, transaction_month DATE, spend float ); ...

1
2 3 4 5
362