1

I am trying to import a CSV file into BQ from GS.

The cmd I use is:

$ bq load --field_delimiter=^ --quote='"' --allow_quoted_newlines --allow_jagged_rows --ignore_unknown_values wr_dev.drupal_user_profile gs://fls_csv_files/user_profileA.csv uid:string,first_name:string,last_name:string,category_id:string,logo_type:string,country_id:string,phone:string,phone_2:string,address:string,address_2:string,city:string,state:string,zip:string,company_name:string,created:string,updated:string,subscription:string

the reported error is

File: 0 / Line:1409 / Field:14, Data between close double quote (")
and field separator: field starts with: <Moreno L>

sample data is:

$ sed -n '1409,1409p' user_profileA.csv
$ 1893^"Moreno"^"Jackson"^17^0^1^"517-977-1133"^"517-303-3717"^""^""^""^""^""^"Moreno L Jackson \"THE MOTIVATOR!\" "^0^1282240785^1

which was generated from MySQL with:

SELECT * INTO OUTFILE '/opt/mysql_exports/user_profileA.csv' 
FIELDS TERMINATED BY '^' 
OPTIONALLY ENCLOSED BY '"' 
LINES TERMINATED BY '\n' 
FROM p;

Why I get the error message in BQ? How to properly export from MySQL CSV files that have newlines (CR and LF mixed, as it was user input from Windows or Mac)

Couple of job IDs:
Job ID: aerobic-forge-504:bqjob_r75d28c332a179207_0000014710c6969d_1
Job ID: aerobic-forge-504:bqjob_r732cb544f96e3d8d_0000014710f8ffe1_1

Update

Apparently it's more to this. I used 5.5.34-MariaDB-wsrep-log INTO OUTFILE, and either is a bug or something wrong, but I get invalid CSV exports. I had to use other tool to export proper CSV. (tool: SQLYog)

it has problems with double quotes, for example Field 14 here has error:

3819^Ron ^Wolbert^6^0^1^6123103169^^^^^^^""Lil"" Ron's^0^1282689026^1
1
4

UPDATE 2019:

Try this as an alternative:

  • Load the MySQL backup files into a Cloud SQL instance.
  • Read the data in BigQuery straight out of MySQL.

Longer how-to:


The proper way to encode a double quote in CSV is to put another double quote in front of it.

So instead of:

"Moreno L Jackson \"THE MOTIVATOR!\"...

Have:

"Moreno L Jackson ""THE MOTIVATOR!""...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.