3

I work with gzipped log files which contain url-encoded columns. (a space character is encoded as "%20", etc).

My plan was to import these files directly from Google Cloud Storage into BigQuery.

I did not find any option in the Load config to automatically decode values during the import.

I guess you wouldn't advice using a series of REGEXP_REPLACE in all my queries.

Any idea which would avoid parsing all the logs and escape all these characters before importing them to BigQuery (which would be dangerous if one of them is the separator) ?

2

2 Answers 2

4

The accepted answer if for Legacy SQL.

For Standard SQL:

#standardSQL

CREATE TEMPORARY FUNCTION DECODE_URI_COMPONENT(path STRING)
RETURNS STRING
LANGUAGE js AS """
if (path == null) return null;
try {
  return decodeURIComponent(path);
} catch (e) {
  return path;
}
""";

WITH source AS (SELECT "/work.json?myfield=R%C3%A9gions%2CSport" AS path)
SELECT DECODE_URI_COMPONENT(REGEXP_EXTRACT(path, r"[?&]myfield=([^&]+)")) AS myfield FROM source

This returns:

 myfield
---------------
 Régions,Sport
1

Most likely you already ended up with something like below :o)

SELECT url FROM
js(
  (SELECT url FROM 
    (SELECT 'http://example.com/query?q=my%20query%20string' AS url), 
    (SELECT 'http://example.com/query?q=your%20query%20string' AS url), 
    (SELECT 'http://example.com/query?q=his%20query%20string' AS url)
  ),
  // Input columns.
  url,
  // Output schema.
  "[
  {name: 'url', type:'string'}]",
  // The function.
  "function(r, emit) { 
    var url  = decodeURI(r.url);
    emit({
      url: url
    });
  }"
)

https://cloud.google.com/bigquery/user-defined-functions

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.