The target-bigquery loader loads extracted data into a BigQuery data warehouse.

Getting Started

Prerequisites

If you haven’t already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Add an extractor to pull data from a source

Then, follow the steps in the “Activate the Google BigQuery API” section of the repository’s README.

Installation and configuration

Using the Command Line Interface

  1. Add the target-bigquery loader to your project using meltano add:

     meltano add loader target-bigquery
    
  2. Configure the settings below using meltano config.

Using Meltano UI

  1. Start Meltano UI using meltano ui:

     meltano ui
    
  2. Open the Loaders interface at http://localhost:5000/loaders.
  3. Click the “Add to project” button for “BigQuery”.
  4. Configure the settings below in the “Configuration” interface that opens automatically.

Next steps

Follow the remaining step of the Getting Started guide:

  1. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Settings

target-bigquery requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you’re looking for, use the Table of Contents in the sidebar.

Minimal configuration

A minimal configuration of target-bigquery in your meltano.yml project file will look like this:

plugins:
  loaders:
  - name: target-bigquery
    variant: adswerve
    config:
      project_id: my-project-id
      # dataset_id: my-dataset-id   # override if default (see below) is not appropriate
      location: EU

Project ID

BigQuery project

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set project_id <id>

export TARGET_BIGQUERY_PROJECT_ID=<id>

Dataset ID

BigQuery dataset

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set dataset_id <id>

export TARGET_BIGQUERY_DATASET_ID=<id>

Location

Dataset location

See https://cloud.google.com/bigquery/docs/locations.

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set location EU

export TARGET_BIGQUERY_LOCATION=EU

Credentials Path

  • Name: credentials_path
  • Environment variable: TARGET_BIGQUERY_CREDENTIALS_PATH, alias: GOOGLE_APPLICATION_CREDENTIALS
  • Default: $MELTANO_PROJECT_ROOT/client_secrets.json

Fully qualified path to client_secrets.json for your service account.

See the “Activate the Google BigQuery API” section of the repository’s README and https://cloud.google.com/docs/authentication/production.

By default, this file is expected to be at the root of your project directory.

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set credentials_path /home/user/Downloads/client_secrets.json

export TARGET_BIGQUERY_CREDENTIALS_PATH=/home/user/Downloads/client_secrets.json

Validate Records

Validate records

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set validate_records true

export TARGET_BIGQUERY_VALIDATE_RECORDS=true

Add Metadata Columns

  • Name: add_metadata_columns
  • Environment variable: TARGET_BIGQUERY_ADD_METADATA_COLUMNS
  • Default: false

Add _time_extracted and _time_loaded metadata columns

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set add_metadata_columns true

export TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true

Replication Method

  • Name: replication_method
  • Environment variable: TARGET_BIGQUERY_REPLICATION_METHOD
  • Options: append, truncate
  • Default: append

Replication method, append or truncate

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set replication_method truncate

export TARGET_BIGQUERY_REPLICATION_METHOD=truncate

Table Prefix

Add prefix to table name

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set table_prefix <prefix>

export TARGET_BIGQUERY_TABLE_PREFIX=<prefix>

Table Suffix

Add suffix to table name

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set table_suffix <suffix>

export TARGET_BIGQUERY_TABLE_SUFFIX=<suffix>

Max Cache

Maximum cache size in MB

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set max_cache 100

export TARGET_BIGQUERY_MAX_CACHE=100