The dbt-duckdb
utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse.
This utility plugin is meant to be used in favor of the dbt-duckdb transformer plugin type.
Note that this plugin can only be run as part of an ELT pipeline with the
EDK Based Plugin
meltano run command. If you are using meltano elt you should use the transformer plugins. We do recommend migrating to meltano run as the transformer plugin type will be deprecated in a future major Meltano release.
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the dbt-duckdb utility to your
project using
:meltano add -
Configure the dbt-duckdb
settings using
:meltano config
meltano add utility dbt-duckdbmeltano config dbt-duckdb set --interactiveNext steps
- If you're running dbt for the first time in a new environment:
# create a starter dbt_project.yml file, a profiles.yml file, and models directory meltano invoke dbt-duckdb:initialize
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project.
The
dbt-duckdb settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
You can also list these settings using
with the meltano configlist
subcommand:
meltano config dbt-duckdb list
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Path (path)
-
Environment variable:
DBT_DUCKDB_PATH -
Default Value:
${MELTANO_PROJECT_ROOT}/output/warehouse.duckdb
The path on your local filesystem where you would like the DuckDB database file and it's associated write-ahead log to be written.
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set path [value]Profiles Directory (profiles_dir)
-
Environment variable:
DBT_DUCKDB_PROFILES_DIR -
Default Value:
$MELTANO_PROJECT_ROOT/transform/profiles/duckdb
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set profiles_dir [value]Projects Directory (project_dir)
-
Environment variable:
DBT_DUCKDB_PROJECT_DIR -
Default Value:
$MELTANO_PROJECT_ROOT/transform
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set project_dir [value]Schema (schema)
-
Environment variable:
DBT_DUCKDB_SCHEMA -
Default Value:
main
Specify the schema to write into.
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set schema [value]Skip Pre-invoke (skip_pre_invoke)
-
Environment variable:
DBT_DUCKDB_SKIP_PRE_INVOKE -
Default Value:
false
Whether to skip pre-invoke hooks which automatically run dbt clean and deps
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set skip_pre_invoke [value]Target Path (target_path)
-
Environment variable:
DBT_DUCKDB_TARGET_PATH -
Default Value:
$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set target_path [value]dbt Profile type (type)
-
Environment variable:
DBT_DUCKDB_TYPE -
Default Value:
duckdb
Configure this setting directly using the following Meltano command:
meltano config dbt-duckdb set type [value]Commands
The dbt-duckdb utility supports the following commands that can be used withmeltano invokebuild
-
Equivalent to:
build
Will run your models, tests, snapshots and seeds in DAG order.
meltano invoke dbt-duckdb:build [args...]clean
-
Equivalent to:
clean
Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
meltano invoke dbt-duckdb:clean [args...]compile
-
Equivalent to:
compile
Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
meltano invoke dbt-duckdb:compile [args...]debug
-
Equivalent to:
debug
Debug your DBT project and warehouse connection.
meltano invoke dbt-duckdb:debug [args...]deps
-
Equivalent to:
deps
Pull the most recent version of the dependencies listed in packages.yml
meltano invoke dbt-duckdb:deps [args...]describe
-
Equivalent to:
describe
Describe the
meltano invoke dbt-duckdb:describe [args...]docs-generate
-
Equivalent to:
docs generate
Generate documentation for your project.
meltano invoke dbt-duckdb:docs-generate [args...]docs-serve
-
Equivalent to:
docs serve
Serve documentation for your project. Make sure you ran `docs-generate` first.
meltano invoke dbt-duckdb:docs-serve [args...]freshness
-
Equivalent to:
source freshness
Check the freshness of your source data.
meltano invoke dbt-duckdb:freshness [args...]initialize
-
Equivalent to:
initialize
Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory.
meltano invoke dbt-duckdb:initialize [args...]run
-
Equivalent to:
run
Compile SQL and execute against the current target database.
meltano invoke dbt-duckdb:run [args...]seed
-
Equivalent to:
seed
Load data from csv files into your data warehouse.
meltano invoke dbt-duckdb:seed [args...]snapshot
-
Equivalent to:
snapshot
Execute snapshots defined in your project.
meltano invoke dbt-duckdb:snapshot [args...]test
-
Equivalent to:
test
Runs tests on data in deployed models.
meltano invoke dbt-duckdb:test [args...]Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general-brightgreen){kind=icon}
-24,666-c0c0c4){kind=icon}
-47-c0c0c4){kind=icon}
