Google BigQuery - Loader Configuration
For GBQ we have tested and supported using “bq” command line tool which is available when Google Cloud SDK is installed (Version 267.0.0), using below link you can download this.
https://cloud.google.com/sdk/docs/#linux
Make sure that your system has Python 2 with a release number of Python 2.7.9 or higher.
Sample Loader script:
export PATH=<PYTHON2.7_HOME>/bin:$PATH
#For Example
#export PATH=/usr/local/python2.7/bin:$PATH
“<GOOGLE_CLOUD_SDK_HOME>/bin/bq" show --schema "<PROJECT_NAME>:<DATASET_NAME>.$1" > <CAMPAIGN_HOME>/partitions/partition1/tmp/schema.csv.$$
"<GOOGLE_CLOUD_SDK_HOME>/bin/bq" load --field_delimiter="|" "<DATASET>.$1" "$2" <CAMPAIGN_HOME>/partitions/partition1/tmp/schema.csv.$$
exit 0
#Replace <CAMPAIGN_HOME> , <GOOGLE_CLOUD_SDK_HOME> , <PROJECT_NAME> , <DATASET_HOME> with the actual values
Note: BigQuery has lot many syntax and can be configured in different ways to load data In
GBQ, we have configured and tested as above but this can be modified and used according
to your need.
Name above loader script as bqload.sh (or anything relevant) and copy into <CAMPAIGN_HOME>/partitions/partition1 directory.
LoaderCommand | <CAMPAIGN_HOME>/partitions/partition1/bqload.sh <TABLE> <DATAFILE> |
LoaderCommandForAppend | <CAMPAIGN_HOME>/partitions/partition1/bqload.sh <TABLE> <DATAFILE> |
LoaderControlFileSpecifiesFields | FALSE |
LoaderControlFileTemplate | |
LoaderControlFileTemplateForAppend | |
LoaderDelimiter | |
LoaderDelimiterAtEnd | FALSE |
LoaderDelimiterAtEndForAppend | FALSE |
LoaderDelimiterForAppend | |
LoaderNULLValueInDelimitedData | Null |
LoaderUseLocaleDP | FALSE |