1

I am loading a New Line Delimited JSON to bigQuery using the following code snippet in Python 2.7:

from google.cloud import bigquery
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials

bigquery_client = bigquery.Client()
dataset = bigquery_client.dataset('testGAData')
table_ref = dataset.table('gaData')
table = bigquery.Table(table_ref)

with open('gaData.json', 'rb') as source_file:
    job_config = bigquery.LoadJobConfig()
    job_config.source_format = 'NEWLINE_DELIMITED_JSON'
    job = bigquery_client.load_table_from_file(
        source_file, table, job_config=job_config)

It returns me the following error:

File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 897, in load_table_from_file
    raise exceptions.from_http_response(exc.response)
google.api_core.exceptions.BadRequest: 400 POST https://www.googleapis.com/upload/bigquery/v2/projects/test-project-for-experiments/jobs?uploadType=resumable: Required parameter is missing

Why am I getting this error? How can I fix this? Has anyone else faced a similar issue? Thanks in advance. Edit: Added last para, included python imports and corrected the indents.

6
  • Please fix your indentation. Also, where is the bigquery_client initialized? Also, take a look at the docs. It seems that you're missing lots of non-optional variables.
    – kingJulian
    Aug 21, 2018 at 6:25
  • Already done so @kingJulian Aug 21, 2018 at 6:32
  • @kingJulian specified the project Id and the location, still giving the same error. Aug 21, 2018 at 6:42
  • Have you tried adding a full file path?
    – Bobbylank
    Aug 21, 2018 at 12:42
  • 1
    Adding to @elithrar's info, check this, under Table Operations->Upload table data from a file.
    – Iñigo
    Aug 21, 2018 at 13:19

1 Answer 1

8

Issues observed with the initial code

  • You are missing the schema for your table. You can either use job_config.autodetect = True or job_config.schema = [bigquery.SchemaField("FIELD NAME", "FIELD TYPE")].

  • From the documentation, you should set job_config.source_format = `bigquery.SourceFormat.NEWLINE_DELIMITED_JSON` for a JSON file source

  • You should pass yourtable_ref variable as an argument instead your table variable in bigquery_client.load_table_from_file(source_file, table, job_config=job_config)

Link to the documentation

Working Code

The below code works for me. I am using python 3 and google-cloud-bigquery v1.5

from google.cloud import bigquery

client = bigquery.Client()
dataset_id, table_id = "TEST_DATASET", "TEST_TABLE"
data_ref = client.dataset(dataset_id)
table_ref = data_ref.table(table_id)
file_path = "path/to/test.json"

job_config = bigquery.LoadJobConfig()
job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
#job_config.autodetect = True
job_config.schema = [bigquery.SchemaField("Name", "STRING"), bigquery.SchemaField("Age", "INTEGER")]

with open(file_path, 'rb') as source_file:
    job = client.load_table_from_file(source_file, table_ref, location='US', job_config=job_config)

job.result()

print('Loaded {} rows into {}:{}.'.format(job.output_rows, dataset_id, table_id))

Output

>> Loaded 2 rows into TEST_DATASET:TEST_TABLE.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.