Google BigQuery - Loader Configuration

For GBQ we have tested and supported using “bq” command line tool which is available when Google Cloud SDK is installed (Version 267.0.0), using below link you can download this.

https://cloud.google.com/sdk/docs/#linux

Make sure that your system has Python 2 with a release number of Python 2.7.9 or higher.

Sample Loader script:

export PATH=<PYTHON2.7_HOME>/bin:$PATH
#For Example
#export PATH=/usr/local/python2.7/bin:$PATH
 “<GOOGLE_CLOUD_SDK_HOME>/bin/bq" show --schema "<PROJECT_NAME>:<DATASET_NAME>.$1" > <CAMPAIGN_HOME>/partitions/partition1/tmp/schema.csv.$$
"<GOOGLE_CLOUD_SDK_HOME>/bin/bq" load --field_delimiter="|"  "<DATASET>.$1" "$2" <CAMPAIGN_HOME>/partitions/partition1/tmp/schema.csv.$$
exit 0
#Replace <CAMPAIGN_HOME> , <GOOGLE_CLOUD_SDK_HOME> , <PROJECT_NAME> , <DATASET_HOME> with the actual values
Note: BigQuery has lot many syntax and can be configured in different ways to load data In GBQ, we have configured and tested as above but this can be modified and used according to your need.

Name above loader script as bqload.sh (or anything relevant) and copy into <CAMPAIGN_HOME>/partitions/partition1 directory.

Table 1. Google BigQuery user data source configuration for loader
LoaderCommand <CAMPAIGN_HOME>/partitions/partition1/bqload.sh <TABLE> <DATAFILE>
LoaderCommandForAppend <CAMPAIGN_HOME>/partitions/partition1/bqload.sh <TABLE> <DATAFILE>
LoaderControlFileSpecifiesFields FALSE
LoaderControlFileTemplate
LoaderControlFileTemplateForAppend
LoaderDelimiter
LoaderDelimiterAtEnd FALSE
LoaderDelimiterAtEndForAppend FALSE
LoaderDelimiterForAppend
LoaderNULLValueInDelimitedData Null
LoaderUseLocaleDP FALSE