The target-kinesis Meltano loader sends data into Kinesis after it was pulled from a source using an extractor.

Alternative variants #

Multiple variants of target-kinesis are available. This document describes the default prontopro variant, which is recommended for new users.

Alternative variants are:

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Add an extractor to pull data from a source

Installation and configuration #

  1. Add the target-kinesis loader to your project using meltano add : 

    meltano add loader target-kinesis
  2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

  1. Run a data integration (EL) pipeline
If you run into any issues, learn how to get help.

Capabilities #

Settings #

target-kinesis requires the configuration of the following settings:

The settings for loader target-kinesis that are known to Meltano are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Stream Name (stream_name) #

The name of the Kinesis stream to write to.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set stream_name <stream_name>

export TARGET_KINESIS_STREAM_NAME=<stream_name>

AWS S3 Access Key ID (aws_access_key_id) #

S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set aws_access_key_id <aws_access_key_id>

export TARGET_KINESIS_AWS_ACCESS_KEY_ID=<aws_access_key_id>

AWS S3 Secret Access Key (aws_secret_access_key) #

S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set aws_secret_access_key <aws_secret_access_key>

export TARGET_KINESIS_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

AWS Region (region) #

The AWS region to use i.e. us-east-2.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set region <region>

export TARGET_KINESIS_REGION=<region>

Is Firehouse (is_firehose) #

(Default False) Whether to user a Firehouse client or not.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set is_firehose true

export TARGET_KINESIS_IS_FIREHOSE=true

Record Chunks (record_chunks) #

(Default - 10) The max number of records per chunk.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set record_chunks 1234

export TARGET_KINESIS_RECORD_CHUNKS=1234

Data Chunks (data_chunks) #

(Default - 1000) The max size of data per chunk.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set data_chunks 1234

export TARGET_KINESIS_DATA_CHUNKS=1234

Partition Key (partition_key) #

(Default - id) The partition key to use when writing to Kinesis.

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config target-kinesis set partition_key <partition_key>

export TARGET_KINESIS_PARTITION_KEY=<partition_key>

Looking for help? #

If you're having trouble getting the target-kinesis loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!