The dbt transformer uses SQL to transform data stored in your warehouse.

Getting Started

Prerequisites

If you haven’t already, follow the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Load data into your warehouse by following the rest of the Getting Started guide.

Installation and configuration

  1. Add the dbt transformer to your project using meltano add:

     meltano add transformer dbt
    
  2. Configure the settings below using meltano config.

Next steps

  1. Transform loaded data for analysis

If you run into any issues, learn how to get help.

Settings

Meltano automatically sets default values for all dbt settings that can be overridden if needed. These settings are documented below. To quickly find the setting you’re looking for, use the Table of Contents at the top of the page.

Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project.

Minimal configuration

A minimal configuration of dbt in your meltano.yml project file will look like this:

plugins:
  transformers:
  - name: dbt
    pip_url: dbt==0.21.0

Project Directory

The directory where the dbt project is stored. The default value is the /transform directory in the Meltano project root directory.

How to use

Manage this setting using meltano config or an environment variable:

meltano config dbt set project_dir <project_dir>

export DBT_PROJECT_DIR=<project_dir>

Profiles Directory

  • Name: profiles_dir
  • Environment variable: DBT_PROFILES_DIR
  • Default: $MELTANO_PROJECT_ROOT/transform/profile

The directory where the dbt profiles.yml file is stored.

This setting corresponds to dbt’s --profiles-dir option.

How to use

Manage this setting using meltano config or an environment variable:

meltano config dbt set profiles_dir <profiles_dir>

export DBT_PROFILES_DIR=<profiles_dir>

Target

  • Name: target
  • Environment variable: DBT_TARGET
  • Default: $MELTANO_LOAD__DIALECT, which will expand to the value of the dialect extra for the loader used in the pipeline, e.g. postgres for target-postgres and snowflake for target-snowflake.

This is the dialect of your warehouse where data is stored. It maps to the target: value in the dbt profiles.yml file.

How to use

This setting is managed by the loader and is not recommended to be changed via the transformer. You can, if needed, overwrite this setting using meltano config or an environment variable:

meltano config dbt set target <target>

export DBT_TARGET=<target>

Source Schema

  • Name: source_schema
  • Environment variable: DBT_SOURCE_SCHEMA
  • Default: $MELTANO_LOAD__TARGET_SCHEMA, which will expand to the value of the schema setting for the loader used in the pipeline.

This defines the schema were dbt will read data from.

How to use

This setting is managed by the loader and is not recommended to be changed via the transformer. You can, if needed, overwrite this setting using meltano config or an environment variable:

meltano config dbt set source_schema <source_schema>

export DBT_SOURCE_SCHEMA=<source_schema>

Target Schema

This is the schema dbt writes transformation results to.

How to use

Manage this setting using meltano config or an environment variable:

meltano config dbt set target_schema <schema>

export DBT_TARGET_SCHEMA=<schema>

Models

  • Name: models
  • Environment variable: DBT_MODELS
  • Default: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project, which will expand to the value of the package_name extra for any transform used in the pipeline, followed by the namespace of the extractor used in the pipeline (e.g. tap_gitlab for tap-gitlab, followed by my_meltano_project (referring to all models local to your dbt project).

This defines the list of models which dbt will run during a transformation.

This setting corresponds to dbt’s --models option.

How to use

Manage this setting using meltano config or an environment variable:

meltano config dbt set models <models>

export DBT_MODELS=<models>

Commands

Meltano commands are shortcuts for combinations of arguments used with meltano invoke. The command specified will use the defined arguments and pass along any specified environment variables. The commands for dbt are documented below.

Clean

  • Command: clean
  • Argument: clean
  • Reference: dbt clean

Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)

How to use

meltano invoke dbt:clean

Compile

  • Command: compile
  • Argument: compile --models $DBT_MODELS
  • Reference: dbt compile

Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory. View the Models documentation for details on the $DBT_MODELS environment variable.

How to use

meltano invoke dbt:compile

Dependencies

  • Command: deps
  • Argument: deps
  • Reference: dbt deps

Pull the most recent version of the dependencies listed in packages.yml.

How to use

meltano invoke dbt:deps

Run

  • Command: run
  • Argument: run --models $DBT_MODELS
  • Reference: dbt run

Compile SQL and execute against the current target database. View the Models documentation for details on the $DBT_MODELS environment variable.

How to use

meltano invoke dbt:run

Seed

  • Command: seed
  • Argument: seed
  • Reference: dbt seed

Load data from csv files into your data warehouse.

How to use

meltano invoke dbt:seed

Snapshot

  • Command: snapshot
  • Argument: snapshot
  • Reference: dbt snapshot

Execute snapshots defined in your project.

How to use

meltano invoke dbt:snapshot

Test

  • Command: test
  • Argument: test
  • Reference: dbt test

Runs tests on data in deployed models.

How to use

meltano invoke dbt:test

Troubleshooting

No Models Running

If no models are running, consider overriding the models setting using a pipeline-specific configuration.

You can also add a custom extra to your extractor definition in your meltano.yml project file, that can then be referenced from dbt’s models setting using an environment variable:

# meltano.yml
plugins:
  extractors:
  - name: tap-foo
    model_name: my_dbt_models # Specify any models here
# ...
  transformers:
  - name: dbt
    config:
      models: $MELTANO_EXTRACT__MODEL_NAME # Refers to the extractor's `model_name`