Amazon Redshift

target-redshift (ticketswap variant)🥈

Amazon Redshift loader

The target-redshift loader sends data into Amazon Redshift after it was pulled from a source using an extractor

Alternate Implementations

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

  1. Add the target-redshift loader to your project using 
    meltano add
     : 
    2. meltano add loader target-redshift
  2. Configure the target-redshift settings using 
    meltano config
     : 
    3. meltano config target-redshift set --interactive

Next steps

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for target-redshift may have been automatically set when originally added to the Hub. Please review the capabilities when using this loader. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this loader.

This plugin has the following capabilities:

  • about
  • hard-delete
  • schema-flattening
  • stream-maps

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

This target uses boto3 to connect to AWS and perform copy and possibly credential operations. The agent running this target should be connected to AWS to be able to open a boto3 client.

The target-redshift settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using 

meltano config
with the list subcommand: 

meltano config target-redshift list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Activate Version (activate_version)

  • Environment variable: TARGET_REDSHIFT_ACTIVATE_VERSION
  • Default Value: false

If set to false, the tap will ignore activate version messages. If set to true, add_record_metadata must be set to true as well.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set activate_version [value]

AWS Redshift Copy Role Arn (aws_redshift_copy_role_arn)

  • Environment variable: TARGET_REDSHIFT_AWS_REDSHIFT_COPY_ROLE_ARN

Redshift copy role arn to use for the COPY command from s3


Configure this setting directly using the following Meltano command:

meltano config target-redshift set aws_redshift_copy_role_arn [value]

Batch Size Rows (batch_size_rows)

  • Environment variable: TARGET_REDSHIFT_BATCH_SIZE_ROWS

Maximum number of rows in each batch.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set batch_size_rows [value]

Cluster IDentifier (cluster_identifier)

  • Environment variable: TARGET_REDSHIFT_CLUSTER_IDENTIFIER

Redshift cluster identifier. Note if sqlalchemy_url is set or enable_iam_authentication is false this will be ignored.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set cluster_identifier [value]

Database name (dbname)

  • Environment variable: TARGET_REDSHIFT_DBNAME

Database name. Note if sqlalchemy_url is set this will be ignored.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set dbname [value]

Enable Iam Authentication (enable_iam_authentication)

  • Environment variable: TARGET_REDSHIFT_ENABLE_IAM_AUTHENTICATION

If true, use temporary credentials (https://docs.aws.amazon.com/redshift/latest/mgmt/generating-iam-credentials-cli-api.html).


Configure this setting directly using the following Meltano command:

meltano config target-redshift set enable_iam_authentication [value]

Host (host)

  • Environment variable: TARGET_REDSHIFT_HOST

Hostname for redshift instance.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set host [value]

Password (password)

  • Environment variable: TARGET_REDSHIFT_PASSWORD

Password used to authenticate. Note if sqlalchemy_url is set this will be ignored.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set password [value]

Port (port)

  • Environment variable: TARGET_REDSHIFT_PORT
  • Default Value: 5432

The port on which redshift is awaiting connection.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set port [value]

Remove S3 Files (remove_s3_files)

  • Environment variable: TARGET_REDSHIFT_REMOVE_S3_FILES
  • Default Value: false

If you want to remove staging files in S3


Configure this setting directly using the following Meltano command:

meltano config target-redshift set remove_s3_files [value]

S3 Bucket (s3_bucket)

  • Environment variable: TARGET_REDSHIFT_S3_BUCKET

S3 bucket to save staging files before using COPY command


Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_bucket [value]

S3 Key Prefix (s3_key_prefix)

  • Environment variable: TARGET_REDSHIFT_S3_KEY_PREFIX

S3 key prefix to save staging files before using COPY command


Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_key_prefix [value]

S3 Region (s3_region)

  • Environment variable: TARGET_REDSHIFT_S3_REGION

AWS region for S3 bucket. If not specified, region will be detected by boto config resolution. See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html


Configure this setting directly using the following Meltano command:

meltano config target-redshift set s3_region [value]

SSL Enable (ssl_enable)

  • Environment variable: TARGET_REDSHIFT_SSL_ENABLE
  • Default Value: false

Whether or not to use ssl to verify the server's identity. Use ssl_certificate_authority and ssl_mode for further customization. To use a client certificate to authenticate yourself to the server, use ssl_client_certificate_enable instead.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set ssl_enable [value]

SSL Mode (ssl_mode)

  • Environment variable: TARGET_REDSHIFT_SSL_MODE
  • Default Value: verify-full

SSL Protection method, see [redshift documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html for more information. Must be one of disable, allow, prefer, require, verify-ca, or verify-full.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set ssl_mode [value]

Temp Dir (temp_dir)

  • Environment variable: TARGET_REDSHIFT_TEMP_DIR
  • Default Value: temp

Where you want to store your temp data files.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set temp_dir [value]

User (user)

  • Environment variable: TARGET_REDSHIFT_USER

User name used to authenticate. Note if sqlalchemy_url is set this will be ignored.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set user [value]
Expand To Show SDK Settings

Add Record Metadata (add_record_metadata)

  • Environment variable: TARGET_REDSHIFT_ADD_RECORD_METADATA
  • Default Value: false

Note that this must be enabled for activate_version to work!This adds _sdc_extracted_at, _sdc_batched_at, and more to every table. See https://sdk.meltano.com/en/latest/implementation/record_metadata.html for more information.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set add_record_metadata [value]

Default Target Schema (default_target_schema)

  • Environment variable: TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA
  • Default Value: $MELTANO_EXTRACT__LOAD_SCHEMA

Redshift schema to send data to, example: tap-clickup


Configure this setting directly using the following Meltano command:

meltano config target-redshift set default_target_schema [value]

Faker Locale (faker_config.locale)

  • Environment variable: TARGET_REDSHIFT_FAKER_CONFIG_LOCALE

One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization


Configure this setting directly using the following Meltano command:

meltano config target-redshift set faker_config locale [value]

Faker Seed (faker_config.seed)

  • Environment variable: TARGET_REDSHIFT_FAKER_CONFIG_SEED

Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator


Configure this setting directly using the following Meltano command:

meltano config target-redshift set faker_config seed [value]

Enable Schema Flattening (flattening_enabled)

  • Environment variable: TARGET_REDSHIFT_FLATTENING_ENABLED

'True' to enable schema flattening and automatically expand nested properties.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set flattening_enabled [value]

Max Flattening Depth (flattening_max_depth)

  • Environment variable: TARGET_REDSHIFT_FLATTENING_MAX_DEPTH

The max depth to flatten schemas.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set flattening_max_depth [value]

Hard Delete (hard_delete)

  • Environment variable: TARGET_REDSHIFT_HARD_DELETE
  • Default Value: false

When activate version is sent from a tap this specefies if we should delete the records that don't match, or mark them with a date in the _sdc_deleted_at column. This config option is ignored if activate_version is set to false.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set hard_delete [value]

Load Method (load_method)

  • Environment variable: TARGET_REDSHIFT_LOAD_METHOD
  • Default Value: append-only

The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set load_method [value]

User Stream Map Configuration (stream_map_config)

  • Environment variable: TARGET_REDSHIFT_STREAM_MAP_CONFIG

User-defined config values to be used within map expressions.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set stream_map_config [value]

Stream Maps (stream_maps)

  • Environment variable: TARGET_REDSHIFT_STREAM_MAPS

Config object for stream maps capability. For more information check out Stream Maps.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set stream_maps [value]

Validate Records (validate_records)

  • Environment variable: TARGET_REDSHIFT_VALIDATE_RECORDS
  • Default Value: true

Whether to validate the schema of the incoming streams.


Configure this setting directly using the following Meltano command:

meltano config target-redshift set validate_records [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-redshift loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the 
#plugins-general
channel.

Install

meltano add loader target-redshift

Maintenance Status

  • Maintenance Status
  • Built with the Meltano SDK

Repo

https://github.com/TicketSwap/target-redshift
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • TicketSwap

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

Keywords

  • meltano_sdkdatabase