12

I am trying to figure out the number of slots used by every big query query. Is there a way to find it out?

4 Answers 4

8

As per google docs, this way we can calculate number of slot used(average).

Number of slot = total_slot_ms / TIMESTAMP_DIFF(end_time,start_time, MILLISECOND)

select job_id
       ,total_slot_ms / TIMESTAMP_DIFF(end_time,start_time,MILLISECOND) as num_slot
from `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT

or manually using BQ UI execution details if you don't have access to the table above.

Number of slot = Slot time consumed (convert in MILLISECOND)/Elapses time (convert in MILLISECOND) 
3
  • Do you have a link to the docs that you mentioned?
    – jamiet
    Jul 27, 2022 at 13:46
  • I can't find any reference for this calculation... wondering how you use the total_slot_ms (which is a processing time) to calculate the number of slots (which is a CPUs)?
    – Soliman
    Dec 20, 2022 at 3:04
  • @Soliman it's mentioned here: cloud.google.com/bigquery/docs/…
    – Deem
    Feb 21, 2023 at 20:55
5

This information is indeed available in job.statistics.query.timeline that forms parts of the Jobs API of BigQuery (https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#resource). When you get this information it comes in an array like this:

timeline: 
    [ '{"elapsedMs":"750","totalSlotMs":"2795","pendingUnits":"8","completedUnits":"66","activeUnits":"9"}',
      '{"elapsedMs":"1252","totalSlotMs":"3617","pendingUnits":"1","completedUnits":"73","activeUnits":"1"}',
      '{"elapsedMs":"2944","totalSlotMs":"5643","pendingUnits":"0","completedUnits":"78","activeUnits":"0"}' ],

So what you can do depends on your actual question:

1) If your question is "What is the total amount of slots utilised by the query over its running time?", then look at the final value of completedUnits

2) If your question is "How are the slots utilised during the query's running time?", then you can build an average of completedUnits over the elapsedMs per timeslice.

8
  • If the max slots for a on-demand project is 2000, how is it that my timeline shows ` "activeUnits": "7457"` for query I recently ran? I don't think "units" == "slots". Oct 24, 2019 at 11:58
  • Hi @GrahamPolley, I can offer you this thread with the BigQuery people on Twitter: twitter.com/eebsidian/status/1097960643498598408 Oct 25, 2019 at 14:22
  • 1
    I still don't understand it. I don't think anyone really does TBH. You normally shouldn't care, but if you're analysing if you need to move to flat-rate then it's important. If "activeUnits" are the number of slots the query is currently using to execute the query, and I'm using on-demand, then how can I see numbers greater than 2000 e.g. 7457?! Oct 26, 2019 at 0:11
  • I suggest two things: Take a look at how many projects you have. I'm guessing you have more than 1 and the other projects are also busy taking up slots (the quote is 2000 per project). Second thing is take a look at Stackdriver to see the breakdown of slot usage. Oct 26, 2019 at 7:30
  • is it possible to do this using the "bq" cli ? Nov 11, 2020 at 8:40
3

There is a Slot Utilization Chart in Stackdriver Monitoring For BigQuery
It shows Allocated and Available Slot for selected Project

Unfortunatelly, I don't think such stats is available on per each query basis

9
  • I am checking the stack driver. I could see that zero slots are allocated in specific time period. But I had a query running during that time period. So I am confused.
    – passionate
    Nov 22, 2016 at 21:13
  • I am running a query from X project but using data sources of Y project. Will it utilize the slots from project X or project Y?
    – passionate
    Nov 22, 2016 at 21:14
  • project X will be your project to check within Nov 22, 2016 at 21:15
  • I see zero slots for project X. So only I was trying to figure out the slots per query.
    – passionate
    Nov 22, 2016 at 21:18
  • can you confirm that project X is your billing project? Nov 22, 2016 at 21:20
1

You can get per-query slot utilisation using the INFORMATION_SCHEMA tables for jobs.

Example query to get the slot utilisation of today's queries of the current project:

SELECT
  project_id,
  job_id,
  start_time,
  end_time,
  query,
  total_slot_ms,
  total_bytes_processed/1e9 AS gbs_processed,
  destination_table.table_id AS destination_table
FROM
`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE DATE(creation_time)=CURRENT_DATE

The total_slot_ms field is what you're looking for, I guess.

In Google's words, it expresses the "Slot-milliseconds for the job over its entire duration." (from the schema documentation).

There are equivalent INFORMATION_SCHEMA tables for individual users (INFORMATION_SCHEMA.JOBS_BY_USER) and for the whole organisation (INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION).

2
  • I don't think total_slot_ms is the same as the number of slots used which is what the OP was asking about.
    – jamiet
    Jul 27, 2022 at 13:45
  • Yeah, the number of slots used is a quantity that changes from time to time. To get the average slot used during the query, one can sum total_slot_ms and divide by the query duration in milliseconds.
    – pietrodn
    Jul 28, 2022 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.