The target-bigquery Meltano loader sends data into BigQuery after it was pulled from a source using an extractor.

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Add an extractor to pull data from a source

Installation and configuration #

Using the Command Line Interface #

  1. Add the target-bigquery loader to your project using meltano add : 

    meltano add loader target-bigquery --variant adswerve
  2. Configure the settings below using meltano config .

Using Meltano UI #

  1. Start Meltano UI using meltano ui: 

    meltano ui
  2. Open the Loaders interface at http://localhost:5000/loaders.
  3. Click the “Add to project” button for “BigQuery”.
  4. Choose "Add variant 'adswerve'".
  5. Configure the settings below in the “Configuration” interface that opens automatically.

Next steps #

Follow the remaining steps of the Getting Started guide:

  1. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Settings #

target-bigquery requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Project Id (project_id) #

BigQuery project

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set project_id <project_id>

export TARGET_BIGQUERY_PROJECT_ID=<project_id>

Dataset Id (dataset_id) #

BigQuery dataset

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set dataset_id <dataset_id>

export TARGET_BIGQUERY_DATASET_ID=<dataset_id>

Location (location) #

Dataset location

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set location <location>

export TARGET_BIGQUERY_LOCATION=<location>

Credentials Path (credentials_path) #

  • Environment variable: TARGET_BIGQUERY_CREDENTIALS_PATH, alias: GOOGLE_APPLICATION_CREDENTIALS
  • Default: $MELTANO_PROJECT_ROOT/client_secrets.json

Fully qualified path to client_secrets.json for your service account.

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set credentials_path <credentials_path>

export TARGET_BIGQUERY_CREDENTIALS_PATH=<credentials_path>

Validate Records (validate_records) #

Validate records

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set validate_records true

export TARGET_BIGQUERY_VALIDATE_RECORDS=true

Add Metadata Columns (add_metadata_columns) #

Add _time_extracted and _time_loaded metadata columns

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set add_metadata_columns true

export TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true

Replication Method (replication_method) #

Replication method, append or truncate

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set replication_method append

export TARGET_BIGQUERY_REPLICATION_METHOD=append

Table Prefix (table_prefix) #

Add prefix to table name

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set table_prefix <table_prefix>

export TARGET_BIGQUERY_TABLE_PREFIX=<table_prefix>

Table Suffix (table_suffix) #

Add suffix to table name

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set table_suffix <table_suffix>

export TARGET_BIGQUERY_TABLE_SUFFIX=<table_suffix>

Max Cache (max_cache) #

Maximum cache size in MB

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set max_cache <max_cache>

export TARGET_BIGQUERY_MAX_CACHE=<max_cache>

Merge State Messages (merge_state_messages) #

Whether to merge multiple state messages from the tap into the state file or uses the last state message as the state file.

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set merge_state_messages true

export TARGET_BIGQUERY_MERGE_STATE_MESSAGES=true

Table Config (table_config) #

A path to a file containing the definition of partitioning and clustering.

How to use #

Manage this setting using Meltano UI, meltano config, or an environment variable: 

meltano config target-bigquery set table_config <table_config>

export TARGET_BIGQUERY_TABLE_CONFIG=<table_config>

Looking for help? #

If you're having trouble getting the target-bigquery loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to!