dbt BigQuery (EDK preview)

dbt-bigquery from dbt-labs

The dbt-bigquery utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse.

EDK Preview

This utility is in a preview stage of readiness. It is based upon our Extension Developer Kit (EDK) which is the new way to build and add plugins to Meltano Hub. This preview of dbt will replace the native transformer once it comes out of preview. In the meantime, we would appreciate any feedback in the EDK repo and in the dbt Extension repo.

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

  1. Add the dbt-bigquery utility to your project using
    meltano add
    :
  2. meltano add utility dbt-bigquery
  3. Configure the dbt-bigquery settings using
    meltano config
    :
  4. meltano config dbt-bigquery set --interactive

Next steps

  1. If you're running dbt for the first time in a new environment:
# create a starter dbt_project.yml file, a profiles.yml file, and models directory
meltano invoke dbt-bigquery:initialize

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project. dbt also has adapter-specific documentation for BigQuery.

The dbt-bigquery settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Skip Pre-invoke (skip_pre_invoke)

  • Environment variable: DBT_BIGQUERY_SKIP_PRE_INVOKE
  • Default Value: false

Whether to skip pre-invoke hooks which automatically run dbt clean and deps

dbt Profile Type (type)

  • Environment variable: DBT_BIGQUERY_TYPE
  • Default Value: bigquery
[No description provided.]

Project Directory (project_dir)

  • Environment variable: DBT_BIGQUERY_PROJECT_DIR
  • Default Value: $MELTANO_PROJECT_ROOT/transform
[No description provided.]

Profiles Directory (profiles_dir)

  • Environment variable: DBT_BIGQUERY_PROFILES_DIR
  • Default Value: $MELTANO_PROJECT_ROOT/transform/profiles/bigquery
[No description provided.]

Authentication Method (auth_method)

  • Environment variable: DBT_BIGQUERY_AUTH_METHOD

The auth method to use. One of: "oauth", "oauth-secrets", or "service-account"

Project (project)

  • Environment variable: DBT_BIGQUERY_PROJECT

The BigQuery project ID.

Dataset (dataset)

  • Environment variable: DBT_BIGQUERY_DATASET

The dataset to use.

Refresh Token (refresh_token)

  • Environment variable: DBT_BIGQUERY_REFRESH_TOKEN

The refresh token, if authenticating via oauth-secrets method.

Client ID (client_id)

  • Environment variable: DBT_BIGQUERY_CLIENT_ID

The client id to use, if authenticating via oauth-secrets method.

Client Secret (client_secret)

  • Environment variable: DBT_BIGQUERY_CLIENT_SECRET

The client secret to use, if authenticating via oauth-secrets method.

Token URI (token_uri)

  • Environment variable: DBT_BIGQUERY_TOKEN_URI

The token redirect URI

Keyfile (keyfile)

  • Environment variable: DBT_BIGQUERY_KEYFILE

The path to the `keyfile.json`` to use, if authenticating via service-account method.

Commands

The dbt-bigquery utility supports the following commands that can be used with
meltano invoke
:

build

  • Equivalent to: build

Will run your models, tests, snapshots and seeds in DAG order.

meltano invoke dbt-bigquery:build [args...]

clean

  • Equivalent to: clean

Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)

meltano invoke dbt-bigquery:clean [args...]

compile

  • Equivalent to: compile

Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.

meltano invoke dbt-bigquery:compile [args...]

debug

  • Equivalent to: debug

Debug your DBT project and warehouse connection.

meltano invoke dbt-bigquery:debug [args...]

deps

  • Equivalent to: deps

Pull the most recent version of the dependencies listed in packages.yml

meltano invoke dbt-bigquery:deps [args...]

describe

  • Equivalent to: describe

Describe the

meltano invoke dbt-bigquery:describe [args...]

docs-generate

  • Equivalent to: docs generate

Generate documentation for your project.

meltano invoke dbt-bigquery:docs-generate [args...]

docs-serve

  • Equivalent to: docs serve

Serve documentation for your project. Make sure you ran `docs-generate` first.

meltano invoke dbt-bigquery:docs-serve [args...]

freshness

  • Equivalent to: source freshness

Check the freshness of your source data.

meltano invoke dbt-bigquery:freshness [args...]

initialize

  • Equivalent to: initialize

Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory.

meltano invoke dbt-bigquery:initialize [args...]

run

  • Equivalent to: run

Compile SQL and execute against the current target database.

meltano invoke dbt-bigquery:run [args...]

seed

  • Equivalent to: seed

Load data from csv files into your data warehouse.

meltano invoke dbt-bigquery:seed [args...]

snapshot

  • Equivalent to: snapshot

Execute snapshots defined in your project.

meltano invoke dbt-bigquery:snapshot [args...]

test

  • Equivalent to: test

Runs tests on data in deployed models.

meltano invoke dbt-bigquery:test [args...]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the dbt-bigquery utility to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the
#plugins-general
channel.

Install

meltano add utility dbt-bigquery

Maintenance Status

  • Maintenance Status
  • Stars
  • Forks
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • dbt Labs

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

Keywords

  • meltano_edk