12

In the example given below, the last line is not uploaded. I get an error:

Data between close double quote (") and field separator: 

This looks like a bug since all the data between pipe symbol should be treated as a single field.

Schema: one:string,two:string,three:string,four:string

Upload file:

This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | "start quote" and | a word after quotes

The first and second line above is processed. But not the third.


Update:

Can some please explain why does the following work except the third line?

This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | "start quote" and | a word after quotes
forth line | enclosed | {"GPRS","MCC_DETECTED":false,"MNC_DETECTED":false} | how does this work?
fifth line | with | {"start quote"} and | a word after quotes

There can be some fancy explanation to this. From the end user perspective this is absurd.

3
  • 2
    I just tried your update, and the example you gave does not actually work. Like most CSV parsing applications, BigQuery by default considers double quotes(") as a field enclosing character. The 3rd line is not escaped properly (see my answer below) and the ingestion will throw an error. Your options are: 1. Change the "configuration.load.quote" parameter in your ingestion requests to something other than double-quote (a character that does not appear in your data). 2. Escape your data as recommended below... for example, Python's csv class does this automatically. – Michael Manoochehri Sep 14 '12 at 18:44
  • While there may be merit in considering being able to do things like set "configuration.load.quote" to NULL, I think that it is important to be very explicit with delimiters and field enclosing characters when working with large amounts of CSV data. It helps to catch errors in the underlying data, and to ensure that the data you are ingesting is valid. – Michael Manoochehri Sep 14 '12 at 18:47
  • Try going thru Cloud SQL as an alternative: medium.com/google-cloud/… – Felipe Hoffa Sep 10 '19 at 1:25
12

From the CSV RFC4180 page: "If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote."

You probably want to do this:

This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | " ""start quote"" and " | a word after quotes

More about our CSV input format here.

3
  • 4
    Note you can change the quote character by setting the 'quote' field in the load job configuration to something you'll never use. – Jordan Tigani Sep 14 '12 at 14:48
  • 1
    How to set quote field in load job? if I use bq load --quote '^' ... then I get an error FATAL Flags parsing error: Unknown command line flag 'quote' # if I add quote parameter in ~/.bigqueryrc then I get error : Unknown flag configuration.load.quote found in bigqueryrc file – shantanuo Sep 15 '12 at 3:29
  • I guess "quote" parameter will be added to bq command line tool in the next release. This issue is so important to me that I will wait for the new version! – shantanuo Sep 16 '12 at 1:22
11

Using --quote worked perfectly.

bq load 
--source_format CSV --quote "" 
--field_delimiter \t 
--max_bad_records 10 
-E UTF-8   
destination table
Source files 
1
  • To be clear, the point here is that you're passing an empty string to the --quote flag – kjmerf Jun 11 '20 at 20:58
1

You can use the other flags also while uploading the data. I used the bq tool with following flags

bq load -F , --source_format CSV --skip_leading_rows 1 --max_bad_records 1 --format csv -E UTF-8 yourdatset gs://datalocation.
1
1

Try this as an alternative:

  • Load the MySQL backup files into a Cloud SQL instance.
  • Read the data in BigQuery straight out of MySQL.

Longer how-to:

0

Try loading every time with bq shell.

I had to load 1100 columns. While trying with the console with all the error options, it threw lot many errors. Ignoring the errors in the console means loosing records.

Hence tried with the shell and succeeded loading all the records.

Try the following:

bq load --source_format CSV --quote "" --field_delimiter \t --allow_jagged_rows --ignore_unknown_values --allow_quoted_newlines --max_bad_records 10 -E UTF-8 {dataset_name}.{table_name} gs://{google_cloud_storage_location}/* {col_1}:{data_type1},{col_2}:{data_type2}, ....

References:

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#bigquery_load_table_gcs_csv-cli

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#csv-options

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.