dbt-redshift - Meltano Hub

dbt Redshift

dbt-redshift (dbt-labs variant)

The dbt-redshift utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse. This utility plugin is meant to be used in favor of the dbt-redshift transformer plugin type. Note that this plugin can only be run as part of an ELT pipeline with the meltano run command. If you are using meltano elt you should use the transformer plugins. We do recommend migrating to meltano run as the transformer plugin type will be deprecated in a future major Meltano release.

EDK Based Plugin

This utility is based on the Meltano Extension Developer Kit (EDK) which is the preferred way to build and add non-Singer plugins to Meltano Hub. For more information about the EDK, please read this section of the Meltano docs. If you have any feedback or suggestions, add them to the EDK repo.

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the dbt-redshift utility to your project using
```
meltano add
```
:

meltano add utility dbt-redshift

Configure the dbt-redshift settings using

meltano config

:

meltano config dbt-redshift set --interactive

Next steps

If you're running dbt for the first time in a new environment:

# create a starter dbt_project.yml file, a profiles.yml file, and models directory
meltano invoke dbt-redshift:initialize

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project. dbt also has adapter-specific documentation for Redshift.

The dbt-redshift settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

auth_method
autocreate
cluster_id
db_groups
dbname
host
iam_duration_seconds
iam_profile
keepalives_idle
password
port
profiles_dir
project_dir
ra3_node
schema
search_path
skip_pre_invoke
sslmode
target_path
type
user

You can also list these settings using

meltano config

with the list subcommand:

meltano config dbt-redshift list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Authentication Method (auth_method)

Environment variable: DBT_REDSHIFT_AUTH_METHOD

The auth method to use (to use iam, set to "iam". Omit to use password-based auth.)

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set auth_method [value]

Autocreate (autocreate)

Environment variable: DBT_REDSHIFT_AUTOCREATE

Whether or not to automatically create entities.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set autocreate [value]

Cluster ID (cluster_id)

Environment variable: DBT_REDSHIFT_CLUSTER_ID

The cluster id.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set cluster_id [value]

Databse Groups (db_groups)

Environment variable: DBT_REDSHIFT_DB_GROUPS

Database groups to use.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set db_groups [value]

Database (dbname)

Environment variable: DBT_REDSHIFT_DBNAME

The name of the db to connect to.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set dbname [value]

Host (host)

Environment variable: DBT_REDSHIFT_HOST

The host for the cluster.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set host [value]

IAM Duration Seconds (iam_duration_seconds)

Environment variable: DBT_REDSHIFT_IAM_DURATION_SECONDS

Duration of the IAM session.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set iam_duration_seconds [value]

IAM Profile (iam_profile)

Environment variable: DBT_REDSHIFT_IAM_PROFILE

The IAM profile to use.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set iam_profile [value]

Keep Alives Idle (keepalives_idle)

Environment variable: DBT_REDSHIFT_KEEPALIVES_IDLE

Seconds between TCP keepalive packets

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set keepalives_idle [value]

Password (password)

Environment variable: DBT_REDSHIFT_PASSWORD

The password, if using password-based auth.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set password [value]

Port (port)

Environment variable: DBT_REDSHIFT_PORT

The port to connect to.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set port [value]

Profiles Directory (profiles_dir)

Environment variable: DBT_REDSHIFT_PROFILES_DIR
Default Value: $MELTANO_PROJECT_ROOT/transform/profiles/redshift

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set profiles_dir [value]

Project Directory (project_dir)

Environment variable: DBT_REDSHIFT_PROJECT_DIR
Default Value: $MELTANO_PROJECT_ROOT/transform

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set project_dir [value]

RA3 Node (ra3_node)

Environment variable: DBT_REDSHIFT_RA3_NODE

Enables cross-database sources.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set ra3_node [value]

Schema (schema)

Environment variable: DBT_REDSHIFT_SCHEMA

The schema to use.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set schema [value]

Search Path (search_path)

Environment variable: DBT_REDSHIFT_SEARCH_PATH

The search path to use (use of this setting is not recommended)

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set search_path [value]

Skip Pre-invoke (skip_pre_invoke)

Environment variable: DBT_REDSHIFT_SKIP_PRE_INVOKE
Default Value: false

Whether to skip pre-invoke hooks which automatically run dbt clean and deps

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set skip_pre_invoke [value]

SSL Mode (sslmode)

Environment variable: DBT_REDSHIFT_SSLMODE

SSL MOde used to connect to Redshift.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set sslmode [value]

Target Path (target_path)

Environment variable: DBT_REDSHIFT_TARGET_PATH
Default Value: $MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set target_path [value]

dbt Profile type (type)

Environment variable: DBT_REDSHIFT_TYPE
Default Value: redshift

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set type [value]

User (user)

Environment variable: DBT_REDSHIFT_USER

The user to connect as.

Configure this setting directly using the following Meltano command:

meltano config dbt-redshift set user [value]

Commands

The dbt-redshift utility supports the following commands that can be used with

meltano invoke

:

build

Equivalent to: build

Will run your models, tests, snapshots and seeds in DAG order.

meltano invoke dbt-redshift:build [args...]

clean

Equivalent to: clean

Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)

meltano invoke dbt-redshift:clean [args...]

compile

Equivalent to: compile

Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.

meltano invoke dbt-redshift:compile [args...]

debug

Equivalent to: debug

Debug your DBT project and warehouse connection.

meltano invoke dbt-redshift:debug [args...]

deps

Equivalent to: deps

Pull the most recent version of the dependencies listed in packages.yml

meltano invoke dbt-redshift:deps [args...]

describe

Equivalent to: describe

Describe the

meltano invoke dbt-redshift:describe [args...]

docs-generate

Equivalent to: docs generate

Generate documentation for your project.

meltano invoke dbt-redshift:docs-generate [args...]

docs-serve

Equivalent to: docs serve

Serve documentation for your project. Make sure you ran `docs-generate` first.

meltano invoke dbt-redshift:docs-serve [args...]

freshness

Equivalent to: source freshness

Check the freshness of your source data.

meltano invoke dbt-redshift:freshness [args...]

initialize

Equivalent to: initialize

Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory.

meltano invoke dbt-redshift:initialize [args...]

run

Equivalent to: run

Compile SQL and execute against the current target database.

meltano invoke dbt-redshift:run [args...]

seed

Equivalent to: seed

Load data from csv files into your data warehouse.

meltano invoke dbt-redshift:seed [args...]

snapshot

Equivalent to: snapshot

Execute snapshots defined in your project.

meltano invoke dbt-redshift:snapshot [args...]

test

Equivalent to: test

Runs tests on data in deployed models.

meltano invoke dbt-redshift:test [args...]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the dbt-redshift utility to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Install

meltano add utility dbt-redshift

Maintenance Status

Repo

https://github.com/dbt-labs/dbt-redshift

EDK Extension Repo

https://github.com/meltano/dbt-ext

Maintainer

Meltano Stats

Keywords