Snowflake (meltano
variant)
Table of Contents
The target-snowflake
Meltano loader sends data into Snowflake after it was pulled from a source using an extractor.
Alternative variants #
Multiple
variants
of target-snowflake
are available.
This document describes the meltano
variant.
Alternative variants are:
-
datamill-co
-
transferwise
(default)
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
target-snowflake
loader to your project usingmeltano add
:meltano add loader target-snowflake --variant meltano
-
Configure the settings below using
meltano config
.
Next steps #
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.Capabilities #
These capabilities can also be overriden
by specifying the capabilities
key in your meltano.yml
file.
Settings #
target-snowflake
requires the
configuration
of the following settings:
The settings for loader target-snowflake
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
You can override these
settings or specify additional
ones in your meltano.yml
by adding the settings
key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull
request to the
YAML file
that defines the settings for this
loader.
Account (account
)
#
-
Environment variable:
TARGET_SNOWFLAKE_ACCOUNT
Account Name in Snowflake (https://XXXXX.snowflakecomputing.com)
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set account <account>
export TARGET_SNOWFLAKE_ACCOUNT=<account>
Username (username
)
#
-
Environment variable:
TARGET_SNOWFLAKE_USERNAME
The username you use for logging in
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set username <username>
export TARGET_SNOWFLAKE_USERNAME=<username>
Password (password
)
#
-
Environment variable:
TARGET_SNOWFLAKE_PASSWORD
The password you use for logging in
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set password <password>
export TARGET_SNOWFLAKE_PASSWORD=<password>
Role (role
)
#
-
Environment variable:
TARGET_SNOWFLAKE_ROLE
Role to be used for loading the data, e.g. LOADER
. Also this role is GRANTed usage to all tables and schemas created
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set role <role>
export TARGET_SNOWFLAKE_ROLE=<role>
Database (database
)
#
-
Environment variable:
TARGET_SNOWFLAKE_DATABASE
The name of the Snowflake database you want to use
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set database <database>
export TARGET_SNOWFLAKE_DATABASE=<database>
Warehouse (warehouse
)
#
-
Environment variable:
TARGET_SNOWFLAKE_WAREHOUSE
The name of the Snowflake warehouse you want to use
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set warehouse <warehouse>
export TARGET_SNOWFLAKE_WAREHOUSE=<warehouse>
Schema (schema
)
#
-
Environment variable:
TARGET_SNOWFLAKE_SCHEMA
- Default:
$MELTANO_EXTRACT__LOAD_SCHEMA
Note $MELTANO_EXTRACT__LOAD_SCHEMA
will expand to the value of the load_schema
extra for the extractor used in the pipeline, which defaults to the extractor’s namespace, e.g. tap_gitlab
for tap-gitlab
. Values are automatically converted to uppercase before they’re passed on to the plugin, so tap_gitlab
becomes TAP_GITLAB
.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set schema <schema>
export TARGET_SNOWFLAKE_SCHEMA=<schema>
Batch Size (batch_size
)
#
-
Environment variable:
TARGET_SNOWFLAKE_BATCH_SIZE
- Default:
5000
The number of records sent to Snowflake at a time.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set batch_size 5000
export TARGET_SNOWFLAKE_BATCH_SIZE=5000
Timestamp Column (timestamp_column
)
#
-
Environment variable:
TARGET_SNOWFLAKE_TIMESTAMP_COLUMN
- Default:
__loaded_at
Name of the column used for recording the timestamp when Data are uploaded to Snowflake.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-snowflake set timestamp_column <timestamp_column>
export TARGET_SNOWFLAKE_TIMESTAMP_COLUMN=<timestamp_column>
Looking for help? #
If you're having trouble getting the
target-snowflake
loader to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!