The target-kinesis Singer target sends data into Kinesis after it was pulled from a source using a Singer tap.

Alternative variants

Multiple variants of target-kinesis are available. This document describes the default prontopro variant, which is recommended for new users.

Alternative variants are:

Standalone usage

Install the package using pip: 

pip install git+https://github.com/ProntoPro/target-kinesis.git

For additional instructions, refer to the README in the repository.

Usage with Meltano

Install Meltano, create your Meltano project, and add the target to your project as a custom loader: 

meltano add --custom loader target-kinesis

Then, configure the loader, add any Singer tap as an extractor to pull data from a source and run a data integration (EL) pipeline.

Capabilities

Settings

Stream Name (stream_name)

The name of the Kinesis stream to write to.

AWS S3 Access Key ID (aws_access_key_id)

S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

AWS S3 Secret Access Key (aws_secret_access_key)

S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

AWS Region (region)

The AWS region to use i.e. us-east-2.

Is Firehouse (is_firehose)

(Default False) Whether to user a Firehouse client or not.

Record Chunks (record_chunks)

(Default - 10) The max number of records per chunk.

Data Chunks (data_chunks)

(Default - 1000) The max size of data per chunk.

Partition Key (partition_key)

(Default - id) The partition key to use when writing to Kinesis.

Looking for help?

If you're having trouble getting target-kinesis to work by itself or with Meltano, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page?

This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.


Edit this page on GitLab!