target-redshift - Meltano Hub

Amazon Redshift

target-redshift (ticketswap variant)🥈

Amazon Redshift loader

The target-redshift loader sends data into Amazon Redshift after it was pulled from a source using an extractor

Alternate Implementations

TicketSwap (default)🥈
Wise
🥈

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the target-redshift loader to your project using
```
meltano add
```
:

meltano add target-redshift

Configure the target-redshift settings using

meltano config

:

meltano config target-redshift set --interactive

Next steps

Follow the remaining steps of the Getting Started guide:

Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for target-redshift may have been automatically set when originally added to the Hub. Please review the capabilities when using this loader. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this loader.

This plugin has the following capabilities:

about
activate-version
hard-delete
schema-flattening
stream-maps

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

This target uses boto3 to connect to AWS and perform copy and possibly credential operations. The agent running this target should be connected to AWS to be able to open a boto3 client.

The target-redshift settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

activate_version
aws_redshift_copy_role_arn
batch_size_rows
cluster_identifier
dbname
enable_iam_authentication
grants
host
password
port
process_activate_version_messages
remove_s3_files
s3_bucket
s3_key_prefix
s3_region
ssl_enable
ssl_mode
temp_dir
user

Expand To Show SDK Settings

add_record_metadata
default_target_schema
faker_config.locale
faker_config.seed
flattening_enabled
flattening_max_depth
hard_delete
load_method
stream_map_config
stream_maps
validate_records

You can also list these settings using

meltano config

with the list subcommand:

meltano config target-redshift list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Activate Version (activate_version)

Environment variable: TARGET_REDSHIFT_ACTIVATE_VERSION
Default Value: false

If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set activate_version [value]

AWS Redshift Copy Role ARN (aws_redshift_copy_role_arn)

Environment variable: TARGET_REDSHIFT_AWS_REDSHIFT_COPY_ROLE_ARN

Redshift copy role arn to use for the COPY command from s3

Configure this setting directly using the following Meltano command:

meltano config target-redshift set aws_redshift_copy_role_arn [value]

Batch Size Rows (batch_size_rows)

Environment variable: TARGET_REDSHIFT_BATCH_SIZE_ROWS

Maximum number of rows in each batch.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set batch_size_rows [value]

Cluster IDentifier (cluster_identifier)

Environment variable: TARGET_REDSHIFT_CLUSTER_IDENTIFIER

Redshift cluster identifier. Note if sqlalchemy_url is set or enable_iam_authentication is false this will be ignored.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set cluster_identifier [value]

Database Name (dbname)

Environment variable: TARGET_REDSHIFT_DBNAME

Database name. Note if sqlalchemy_url is set this will be ignored.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set dbname [value]

Enable IAM Authentication (enable_iam_authentication)

Environment variable: TARGET_REDSHIFT_ENABLE_IAM_AUTHENTICATION

If true, use temporary credentials (https://docs.aws.amazon.com/redshift/latest/mgmt/generating-iam-credentials-cli-api.html).

Configure this setting directly using the following Meltano command:

meltano config target-redshift set enable_iam_authentication [value]

Grants (grants)

Environment variable: TARGET_REDSHIFT_GRANTS

List of users/roles/groups that will have select permissions on the tables

Configure this setting directly using the following Meltano command:

meltano config target-redshift set grants [value]

Host (host)

Environment variable: TARGET_REDSHIFT_HOST

Hostname for redshift instance.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set host [value]

Password (password)

Environment variable: TARGET_REDSHIFT_PASSWORD

Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set password [value]

Port (port)

Environment variable: TARGET_REDSHIFT_PORT
Default Value: 5432

The port on which redshift is awaiting connection.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set port [value]

Process `ACTIVATE_VERSION` messages (process_activate_version_messages)

Environment variable: TARGET_REDSHIFT_PROCESS_ACTIVATE_VERSION_MESSAGES
Default Value: true

Whether to process ACTIVATE_VERSION messages.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set process_activate_version_messages [value]

Remove S3 Files (remove_s3_files)

Environment variable: TARGET_REDSHIFT_REMOVE_S3_FILES
Default Value: false

If you want to remove staging files in S3

Configure this setting directly using the following Meltano command:

meltano config target-redshift set remove_s3_files [value]

S3 Bucket (s3_bucket)

Environment variable: TARGET_REDSHIFT_S3_BUCKET

S3 bucket to save staging files before using COPY command

Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_bucket [value]

S3 Key Prefix (s3_key_prefix)

Environment variable: TARGET_REDSHIFT_S3_KEY_PREFIX

S3 key prefix to save staging files before using COPY command

Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_key_prefix [value]

S3 Region (s3_region)

Environment variable: TARGET_REDSHIFT_S3_REGION

AWS region for S3 bucket. If not specified, region will be detected by boto config resolution. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_region [value]

SSL Enable (ssl_enable)

Environment variable: TARGET_REDSHIFT_SSL_ENABLE
Default Value: false

Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set ssl_enable [value]

SSL Mode (ssl_mode)

Environment variable: TARGET_REDSHIFT_SSL_MODE
Default Value: verify-full

SSL Protection method, see [redshift documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set ssl_mode [value]

Temp Dir (temp_dir)

Environment variable: TARGET_REDSHIFT_TEMP_DIR
Default Value: temp

Where you want to store your temp data files.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set temp_dir [value]

User (user)

Environment variable: TARGET_REDSHIFT_USER

User name used to authenticate. Note if sqlalchemy_url is set this will be ignored.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set user [value]

Expand To Show SDK Settings

Add Record Metadata (add_record_metadata)

Environment variable: TARGET_REDSHIFT_ADD_RECORD_METADATA
Default Value: false

Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.htmlfor more information.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set add_record_metadata [value]

Default Target Schema (default_target_schema)

Environment variable: TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA
Default Value: $MELTANO_EXTRACT__LOAD_SCHEMA

Redshift schema to send data to, example: tap-clickup

Configure this setting directly using the following Meltano command:

meltano config target-redshift set default_target_schema [value]

Faker Locale (faker_config.locale)

Environment variable: TARGET_REDSHIFT_FAKER_CONFIG_LOCALE

One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization

Configure this setting directly using the following Meltano command:

meltano config target-redshift set faker_config locale [value]

Faker Seed (faker_config.seed)

Environment variable: TARGET_REDSHIFT_FAKER_CONFIG_SEED

Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator

Configure this setting directly using the following Meltano command:

meltano config target-redshift set faker_config seed [value]

Enable Schema Flattening (flattening_enabled)

Environment variable: TARGET_REDSHIFT_FLATTENING_ENABLED

'True' to enable schema flattening and automatically expand nested properties.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set flattening_enabled [value]

Max Flattening Depth (flattening_max_depth)

Environment variable: TARGET_REDSHIFT_FLATTENING_MAX_DEPTH

The max depth to flatten schemas.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set flattening_max_depth [value]

Hard Delete (hard_delete)

Environment variable: TARGET_REDSHIFT_HARD_DELETE
Default Value: false

When activate version is sent from a tap this specefies if we should delete the records that don't match, or mark them with a date in the _sdc_deleted_at column. This config option is ignored if activate_version is set to false.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set hard_delete [value]

Load Method (load_method)

Environment variable: TARGET_REDSHIFT_LOAD_METHOD
Default Value: append-only

The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set load_method [value]

User Stream Map Configuration (stream_map_config)

Environment variable: TARGET_REDSHIFT_STREAM_MAP_CONFIG

User-defined config values to be used within map expressions.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set stream_map_config [value]

Stream Maps (stream_maps)

Environment variable: TARGET_REDSHIFT_STREAM_MAPS

Config object for stream maps capability. For more information check out Stream Maps.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set stream_maps [value]

Validate Records (validate_records)

Environment variable: TARGET_REDSHIFT_VALIDATE_RECORDS
Default Value: true

Whether to validate the schema of the incoming streams.

Configure this setting directly using the following Meltano command:

meltano config target-redshift set validate_records [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-redshift loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Install

meltano add
              target-redshift

Maintenance Status

Repo

https://github.com/TicketSwap/target-redshift

Maintainer

Meltano Stats

Keywords