Questions tagged [google-bigquery]

Google BigQuery is a Google Cloud Platform product providing serverless queries of petabyte-scale data sets using SQL. BigQuery provides multiple read-write pipelines, and enables data analytics that transform how businesses analyze data.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
13 views

Dataset boreal-fort-402703:bigquery-public-data was not found in location US

Dataset boreal-fort-402703:bigquery-public-data was not found in location US The following is the full query. I could not identify what did I overlooked. I am sure I did put in all the required table ...
Sam LK's user avatar
  • 1
0 votes
0 answers
12 views

Difficulty Configuring GA4 dbt Package in BigQuery

I'm a non-technical marketer new to dbt, trying to leverage the GA4 dbt package for querying my Google Analytics 4 data in BigQuery. I've successfully connected my BigQuery projectto dbt. According to ...
ali izadi's user avatar
  • 143
0 votes
0 answers
11 views

Prallel processing using multiprocessing pool outside __main__

I have a requirement to extract data from BigQuery and post the data to an end point. End point can only accept one request at a time so if post one row at the time it is too slow so I decided to use ...
PravSajja's user avatar
0 votes
0 answers
13 views

SQL Query for Tracking User Path Flow in Looker Studio using Sankey Chart

I'm working on a SQL query to analyze user navigation paths in Google Analytics data, specifically for visualization in a Looker Studio Sankey chart. Looker Studio's Sankey chart requires the data to ...
ali izadi's user avatar
  • 143
0 votes
0 answers
33 views

SQL WHERE clause doesn't work with arrays

I currently using a tool named retool to query data from my bigquery instance. The problem is quite a few of the field names use arrays and I'm having bit of an issue filter field names with arrays in ...
TayOnTech's user avatar
0 votes
0 answers
38 views

Is there a SQL pattern to follow for historical tables that expire old records when new records come in?

id ... effect_date expire_date 1 ... 2023-10-23 9999-12-31 1 ... 2023-10-15 2023-10-22 1 ... 2023-10-10 2023-10-14 1 ... 2023-10-01 2023-10-09 I have a history table that has an id, bunch of data ...
Brian Jones's user avatar
0 votes
0 answers
24 views

Is it possible to backfill in bq data tranfer service with s3 as data source?

I need to schedule a backfill for my data using bigquery Data Tranfer Service, with s3 as source. But it doesn't seems to work following the doc from here I select run for specific date and set to ...
Pete's user avatar
  • 247
0 votes
1 answer
18 views

Convert Date data type on Bigquery string to date

Hi all I am trying to convert my data from string to date so I can do date_trunc for cohort analysis. My data is in string type and the formatting for date looks like this: 20221101 I tried parse_date ...
lyra chang's user avatar
0 votes
1 answer
47 views

BigQuery golang client not using storage API

In my bigquery client, I am enabling the storage read client, as per the documentation and examples at https://github.com/googleapis/google-cloud-go/blob/f2b13307a85e81e278476ea51a359cbb3974667a/...
stbenjam's user avatar
0 votes
1 answer
18 views

Bigquery and Google Data Studio - 1 min refresh

I have a google data studio dashboard where I switched data source from Google Sheets to BigQuery as BigQuery provided option of 1 minute data refresh. But when I started using BigQuery the data ...
dbeings's user avatar
  • 65
0 votes
0 answers
27 views

How to get employee's total hours worked per week relative to a start date on SQL Bigquery

I need to get total hours worked per employee. However, instead of getting just the overall total hours, I need to get the total hours worked per week relative to the employees' start date. For ...
ninop's user avatar
  • 23
0 votes
0 answers
9 views

BigQuery - Android Purchase Attribution

I'm trying to replicate GA4/Firebase traffic & user acquisition reports for mobile apps using BigQuery. I'm using this simple query to check the number of users, new users and purchasers for a ...
Nadia K's user avatar
  • 21
0 votes
1 answer
24 views

How can one unnest an array in BigQuery without using unnest?

I am trying to select hits.page.searchKeyword in a query, but I get the classical nested error. Do I have another option rather than unnest? When I unnest data from hits (GA3 - UA) all my other ...
Georgiana Costache's user avatar
0 votes
1 answer
29 views

Duplicate alias error with BigQuery connection to table with arrays

Inside Google Data Studio (now called Looker Studio) I'm getting BigQuery error: Query error: Duplicate alias **paid_per_stats** found at [line number] I am connecting to a table with nested arrays - ...
Geoff Coco's user avatar
0 votes
1 answer
64 views

What are the most recent StackOverflow datasets on BigQuery?

What are the most recent StackOverflow datasets on BigQuery? Two datasets I am aware of are very much out of date. Are there more recent ones? bigquery-public-data.stackoverflow referenced on https://...
Yulia V's user avatar
  • 3,517
0 votes
1 answer
21 views

Why is my SELECT Portion of my Syntax Showing Up as an Unrecognized Name Error on Big Query?

I've been practicing the big query exercises provided by Coursera. For this specific exercise, I wanted to collect tree sizes from the tree size databases that big query provided to me. My SELECT ...
Jordan Sandigo's user avatar
1 vote
1 answer
37 views

BigQuery: Using LAST_VALUE with an ARRAY

I'm hoping to use LAST_VALUE to fill missing data in a table, but the issue is, the missing data I need to fill consists of ARRAY fields. For example: SELECT id_field, timestamp_field, ...
justin's user avatar
  • 21
-2 votes
0 answers
30 views

One column is a string type the other is an integer type, how to inner join? [closed]

SELECT station_id, name, number_of_rides AS number_of_rides_starting_at_station FROM ( SELECT start_station_id, COUNT (*) number_of_rides FROM bigquery-...
Diana Thorson's user avatar
0 votes
1 answer
28 views

Execute DDL to create a native table in BigQuery from files stored on GCS

I would like to create a native table in BigQuery from a CSV file with DDL statements in a way that replicates the "create table from..." feature in BigQuery's UI as shown in the screenshot ...
Downforu's user avatar
  • 329
0 votes
1 answer
47 views

SQL, backward fill only for first nulls, forward fill only for last nulls, linear interpolation for other nulls

How do I fill backward, forward, and linear interpolation of a column in a table that has column timestamp in bigquery? I have this table: timestamp mycol 1 null 2 null 3 69 4 null 5 71 6 72 7 ...
Muhammad Ikhwan Perwira's user avatar
0 votes
0 answers
10 views

Using AWS Appflow to pull data from Google Bigquery results in a response too large error

I've established a workflow in Appflow that relies on a connection to BigQuery. However, when I execute the workflow, I come across an issue outlined below: The request failed because the service ...
Ashish Padave's user avatar
0 votes
0 answers
15 views

BigQuery There is a page view, but no product displays within 10 sec of starting the page view

I am working with BigQuery and I need to get data from Google Analytics. I am trying to make a query for: There is a page view, but no product displays within 10 sec of starting the page view The ...
junior young's user avatar
0 votes
0 answers
27 views

How to execute SQL from Dataflow Streaming job

I need to create a dataflow streaming pipeline in GCP with these requirements. Read data from Kafka and do streaming insert to a table say table1 in bigquery. Do an SQL operation in bigquery from the ...
Ananth's user avatar
  • 99
0 votes
1 answer
21 views

How to convert STRUCT to STRING in BigQuery

How to convert STRUCT into a STRING representation in BigQuery select struct("John" as name, 23 as age,"IT" as department) as mystruct A JSON representation would be ok. Something ...
Sergey Geron's user avatar
  • 9,412
0 votes
1 answer
48 views

MIN() in JOIN based of dates

I would like to join two tables based on two common fields. Assume the date is 2024 I want to join the sale table on dictionary table which hold only articles with dates 2022 and 2023. Because of the ...
refokaj's user avatar
  • 17
0 votes
0 answers
39 views

How to create a recursive query in BigQuery having the target column as a JSON array?

I have a table called area_groups which aims to represent a graph-like association in which there can have recursion. To illustrate my case, that table has following columns id (UUID) group_ids (...
Cristiano Mendonça's user avatar
0 votes
0 answers
7 views

Does the Simba JDBC Driver request compression from Big Query?

We are attempting to optimize extracts of data in Big Query to Tableau. Using the Simba JDBC driver, we are noticing that it appears to be pulling raw uncompressed data. Rather than leveraging the BQ ...
Phillip Overpeck's user avatar
0 votes
0 answers
8 views

Loading GeoJSON with complex "properties" object into BigQuery

I am attempting to load Daylight OpenStreetMap data into BigQuery. I extracted just the buildings using Osmium and produced a ~600GB newline-delimited GeoJSON file. I uploaded this file to Google ...
magneticMonster's user avatar
0 votes
0 answers
12 views

Custom definitions not populated in BigQuery, but are when previewed in Tag Manager

I have a setup where a Drupal site sets a data layer in the HTML that, as far as I have tested and debugged, correctly sets keys and values. I then have the site connected to GA4 through a Tag Manager ...
AlbionBrown's user avatar
0 votes
0 answers
27 views

BigQuery UDF Error: Correlated Subquery in Temp Function

I am currently creating a query with a UDF, that returns similar stores from a restaurant table. I want to utilize the UDF so that I can make life easier for myself when I need to reprint the data for ...
Moose's user avatar
  • 13
0 votes
0 answers
14 views

bigquery operation time out in europe west 3

All of our datasets have location europe-west3 in BigQuery and it's been going fine until last weekend. Suddenly all read queries fail to retrieve any data and hit operation timeout. When we changed ...
pa-nguyen's user avatar
  • 407
0 votes
0 answers
29 views

BigQuery SLA and service credits

Does anyone have experience measuring BigQuery's SLA and claiming service credits when it is out of SLA? The SLA is defined by API error rate (see full details here), and GCP only provide credits if ...
Henry Heath's user avatar
  • 1,072
0 votes
0 answers
11 views

Export from GCP BigQuery to parquet: How to get proper scale, precision of numeric type?

I have a columnA, of type NUMERIC(29) in tableA in BQ. I export it to parquet using BigQueryToGCSOperator: transferBigQuerryToBucket = BigQueryToGCSOperator( task_id='BigQuerryToBucket' ,...
PipelineSurfer's user avatar
0 votes
0 answers
24 views

Inquiry Regarding Unfamiliar Charges for "Dataplex" in GCP Billing

I am currently running queries for aggregations such as GA in GCP's BigQuery. The queries are straightforward, and the data volume is still relatively low, allowing me to perform aggregations within ...
Devatim's user avatar
  • 331
0 votes
0 answers
13 views

Bigquery query to find 2-week retention

I have made a query to find weekly retention of users. I want to change it to find 2-week retention of users. Below is the query to find weekly retention. First I bucket users into different cohort by ...
Pooja Dengla's user avatar
0 votes
0 answers
17 views

Failed to connect to BigQuery with Python - Deadline of 600.0s exceeded while calling target function

I'm connecting google bigquery via vpn, but the strange thing is that the following code can connect bq successfully when run on the cloud shell, but it always shows a timeout when run locally. The ...
Toby's user avatar
  • 1
0 votes
1 answer
42 views

Transpose JSON in Big Query SQL

We have a column in BigQuery that stored the JSON data below and I need to dynamically parse it. It's my first time to work in a JSON format and We need to transpose it (for reference please see the ...
Dave's user avatar
  • 21
0 votes
0 answers
37 views

Can I install gcloud Auth in a python environment for a dbt project connecting to BigQuery?

I have a DBT project on cli/core I need to connect to my BigQuery account. I ran dbt debug and got an error on the connection test with my dbt-bigquery adaptor saying: "dbt was unable to connect ...
Joshua Smart-Olufemi's user avatar
0 votes
1 answer
17 views

List tables queried by Power BI in Bigquery

I usually use SCHEMA_METADATA to track what is happening in my companies dataset. We use Power BI's connector to update datasets based on our Bigquery Tables, once a day. If I refresh my reports ...
Ivan Cereghetti's user avatar
0 votes
1 answer
44 views

Select all columns with match prefix column name in bigquery

I have this query: SELECT charttime, id, gender, lm_*, vs_* FROM `project-id.dataset.table`; How do I select column with match prefix lm_ and vs_?
Muhammad Ikhwan Perwira's user avatar
0 votes
0 answers
37 views

How to export BigQuery table to parquet using SQL and keep columns mode

I'm trying to export BQ table to GCS in the parquet format. Can't use API because of the size limits, so it's gotta be SQL query. The problem is when using SQL query, parquet ignores table columns ...
AlienDeg's user avatar
  • 1,290
0 votes
0 answers
8 views

where to get tutorial to create a Big Query Project?

Want to learn about path from where to start and finish everything to create my first Big Query project I Searched for tutorial but got nothing useful, so please guide me how can I learn from beginner ...
Manthan Kushwaha's user avatar
0 votes
0 answers
9 views

Bigquery, counts distinct of all columns

I have a table in bigquery that has arouns 500 columns. I want to get a distinct count of all columns. It's not optimal to do count(distinct col1) individually. Can anyone recommend a more optimal way ...
Sam's user avatar
  • 1
0 votes
1 answer
32 views

How to Stream data with Airflow

I have a very large csv files. I want to stream the rows of data either "one by one" or "batch by batch" and put (WRITE_APPEND) it into BigQuery using AirFlow. How can i achieve ...
Jooo21's user avatar
  • 120
0 votes
1 answer
35 views

what data type a pandas column should be for BQ to detect it as a json type?

I have a pandas data frame with a column that has the result of the "json.dumps()" function in python. When I'm trying to append this to a table in BQ using "pandas_gbq.to_gbq()" ...
VIJU's user avatar
  • 156
0 votes
1 answer
18 views

How to remove multiple symbols from column in Biq Query?

How to replace the symbols from below string? "- , KALIRHAT, - , Bagribari, Jalpaiguri, West Bengal Kalirhat Kalirhat Bagribari DHUPGURI WEST BENGAL 735210" Desired Output: "KALIRHAT ...
Kshitij Tandel's user avatar
0 votes
0 answers
53 views

Extend the Core method chain in DataForm

We are developing a model in DataForm using Javascript. https://cloud.google.com/dataform/docs/develop-workflows-js?hl=ja We would like to extend some behaviors, for the domain we want to use. More ...
naoki's user avatar
  • 19
0 votes
0 answers
35 views

dplyr::tbl() returning 0 rows from BigQuery external tables

I'm using bigrquery and dplyr/dbplyr to connect and interact with tables on Google BigQuery. I've noticed that calling dplyr::tbl() or DBI::dbReadTable() on a managed table correctly returns a preview ...
Darren Wong's user avatar
-1 votes
0 answers
44 views

How to separate first name and last name [closed]

I have written the code below SELECT visitor_nick , split(visitor_nick, ' ') AS spilit_test FROM bigquery-405008.Kernel_chat.Kernel_chat Click here But I want to separate First_name &...
Rahul's user avatar
  • 7
-1 votes
1 answer
28 views

BigQuery - Transpose JSON data stored in repeated string field type to a table

I have the following JSON data stored in repeated string field type. "cart_items":["{\"item_id\":\"186\",\"item_name\":\"abc\"},"{\"...
ronencozen's user avatar
  • 2,051

1
2 3 4 5
510