The dbt-redshift
utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse.
This utility plugin is meant to be used in favor of the dbt-redshift transformer plugin type.
EDK Based Plugin
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the dbt-redshift utility to your project
using
:meltano add
-
Configure the dbt-redshift settings using
:meltano config
meltano add utility dbt-redshift
meltano config dbt-redshift set --interactive
Next steps
- If you're running dbt for the first time in a new environment:
# create a starter dbt_project.yml file, a profiles.yml file, and models directory meltano invoke dbt-redshift:initialize
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project. dbt also has adapter-specific documentation for Redshift.
The
dbt-redshift
settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
skip_pre_invoke
type
project_dir
profiles_dir
auth_method
cluster_id
host
password
user
port
dbname
schema
iam_profile
iam_duration_seconds
autocreate
db_groups
keepalives_idle
search_path
sslmode
ra3_node
You can also list these settings using
with the meltano config
list
subcommand:
meltano config dbt-redshift list
You can
override these settings or specify additional ones
in your meltano.yml
by adding the settings
key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Skip Pre-invoke (skip_pre_invoke)
-
Environment variable:
DBT_REDSHIFT_SKIP_PRE_INVOKE
-
Default Value:
false
Whether to skip pre-invoke hooks which automatically run dbt clean and deps
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set skip_pre_invoke [value]
dbt Profile type (type)
-
Environment variable:
DBT_REDSHIFT_TYPE
-
Default Value:
redshift
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set type [value]
Project Directory (project_dir)
-
Environment variable:
DBT_REDSHIFT_PROJECT_DIR
-
Default Value:
$MELTANO_PROJECT_ROOT/transform
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set project_dir [value]
Profiles Directory (profiles_dir)
-
Environment variable:
DBT_REDSHIFT_PROFILES_DIR
-
Default Value:
$MELTANO_PROJECT_ROOT/transform/profiles/redshift
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set profiles_dir [value]
Authentication Method (auth_method)
-
Environment variable:
DBT_REDSHIFT_AUTH_METHOD
The auth method to use (to use iam, set to "iam". Omit to use password-based auth.)
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set auth_method [value]
Cluster ID (cluster_id)
-
Environment variable:
DBT_REDSHIFT_CLUSTER_ID
The cluster id.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set cluster_id [value]
Host (host)
-
Environment variable:
DBT_REDSHIFT_HOST
The host for the cluster.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set host [value]
Password (password)
-
Environment variable:
DBT_REDSHIFT_PASSWORD
The password, if using password-based auth.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set password [value]
User (user)
-
Environment variable:
DBT_REDSHIFT_USER
The user to connect as.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set user [value]
Port (port)
-
Environment variable:
DBT_REDSHIFT_PORT
The port to connect to.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set port [value]
Database (dbname)
-
Environment variable:
DBT_REDSHIFT_DBNAME
The name of the db to connect to.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set dbname [value]
Schema (schema)
-
Environment variable:
DBT_REDSHIFT_SCHEMA
The schema to use.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set schema [value]
IAM Profile (iam_profile)
-
Environment variable:
DBT_REDSHIFT_IAM_PROFILE
The IAM profile to use.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set iam_profile [value]
IAM Duration Seconds (iam_duration_seconds)
-
Environment variable:
DBT_REDSHIFT_IAM_DURATION_SECONDS
Duration of the IAM session.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set iam_duration_seconds [value]
Autocreate (autocreate)
-
Environment variable:
DBT_REDSHIFT_AUTOCREATE
Whether or not to automatically create entities.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set autocreate [value]
Databse Groups (db_groups)
-
Environment variable:
DBT_REDSHIFT_DB_GROUPS
Database groups to use.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set db_groups [value]
Keep Alives Idle (keepalives_idle)
-
Environment variable:
DBT_REDSHIFT_KEEPALIVES_IDLE
Seconds between TCP keepalive packets
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set keepalives_idle [value]
Search Path (search_path)
-
Environment variable:
DBT_REDSHIFT_SEARCH_PATH
The search path to use (use of this setting is not recommended)
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set search_path [value]
SSL Mode (sslmode)
-
Environment variable:
DBT_REDSHIFT_SSLMODE
SSL MOde used to connect to Redshift.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set sslmode [value]
RA3 Node (ra3_node)
-
Environment variable:
DBT_REDSHIFT_RA3_NODE
Enables cross-database sources.
Configure this setting directly using the following Meltano command:
meltano config dbt-redshift set ra3_node [value]
Commands
The dbt-redshift utility supports the following commands that can be used withmeltano invoke
:build
-
Equivalent to:
build
Will run your models, tests, snapshots and seeds in DAG order.
meltano invoke dbt-redshift:build [args...]
clean
-
Equivalent to:
clean
Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
meltano invoke dbt-redshift:clean [args...]
compile
-
Equivalent to:
compile
Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
meltano invoke dbt-redshift:compile [args...]
debug
-
Equivalent to:
debug
Debug your DBT project and warehouse connection.
meltano invoke dbt-redshift:debug [args...]
deps
-
Equivalent to:
deps
Pull the most recent version of the dependencies listed in packages.yml
meltano invoke dbt-redshift:deps [args...]
describe
-
Equivalent to:
describe
Describe the
meltano invoke dbt-redshift:describe [args...]
docs-generate
-
Equivalent to:
docs generate
Generate documentation for your project.
meltano invoke dbt-redshift:docs-generate [args...]
docs-serve
-
Equivalent to:
docs serve
Serve documentation for your project. Make sure you ran `docs-generate` first.
meltano invoke dbt-redshift:docs-serve [args...]
freshness
-
Equivalent to:
source freshness
Check the freshness of your source data.
meltano invoke dbt-redshift:freshness [args...]
initialize
-
Equivalent to:
initialize
Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory.
meltano invoke dbt-redshift:initialize [args...]
run
-
Equivalent to:
run
Compile SQL and execute against the current target database.
meltano invoke dbt-redshift:run [args...]
seed
-
Equivalent to:
seed
Load data from csv files into your data warehouse.
meltano invoke dbt-redshift:seed [args...]
snapshot
-
Equivalent to:
snapshot
Execute snapshots defined in your project.
meltano invoke dbt-redshift:snapshot [args...]
test
-
Equivalent to:
test
Runs tests on data in deployed models.
meltano invoke dbt-redshift:test [args...]
Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general
channel.