Kinesis
Table of Contents
The target-kinesis
Meltano loader sends data into Kinesis after it was pulled from a source using an extractor.
-
- Repository: https://github.com/ProntoPro/target-kinesis
-
-
-
-
-
-
-
- Maintainer:
Alternative variants #
Multiple
variants
of target-kinesis
are available.
This document describes the default prontopro
variant,
which is recommended for new users.
Alternative variants are:
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
target-kinesis
loader to your project usingmeltano add
:meltano add loader target-kinesis
-
Configure the settings below using
meltano config
.
Next steps #
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.Capabilities #
Settings #
target-kinesis
requires the
configuration
of the following settings:
The settings for loader target-kinesis
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
Stream Name (stream_name
)
#
-
Environment variable:
TARGET_KINESIS_STREAM_NAME
The name of the Kinesis stream to write to.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set stream_name <stream_name>
export TARGET_KINESIS_STREAM_NAME=<stream_name>
AWS S3 Access Key ID (aws_access_key_id
)
#
-
Environment variable:
TARGET_KINESIS_AWS_ACCESS_KEY_ID
S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set aws_access_key_id <aws_access_key_id>
export TARGET_KINESIS_AWS_ACCESS_KEY_ID=<aws_access_key_id>
AWS S3 Secret Access Key (aws_secret_access_key
)
#
-
Environment variable:
TARGET_KINESIS_AWS_SECRET_ACCESS_KEY
S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set aws_secret_access_key <aws_secret_access_key>
export TARGET_KINESIS_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
AWS Region (region
)
#
-
Environment variable:
TARGET_KINESIS_REGION
The AWS region to use i.e. us-east-2.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set region <region>
export TARGET_KINESIS_REGION=<region>
Is Firehouse (is_firehose
)
#
-
Environment variable:
TARGET_KINESIS_IS_FIREHOSE
(Default False) Whether to user a Firehouse client or not.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set is_firehose true
export TARGET_KINESIS_IS_FIREHOSE=true
Record Chunks (record_chunks
)
#
-
Environment variable:
TARGET_KINESIS_RECORD_CHUNKS
(Default - 10) The max number of records per chunk.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set record_chunks 1234
export TARGET_KINESIS_RECORD_CHUNKS=1234
Data Chunks (data_chunks
)
#
-
Environment variable:
TARGET_KINESIS_DATA_CHUNKS
(Default - 1000) The max size of data per chunk.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set data_chunks 1234
export TARGET_KINESIS_DATA_CHUNKS=1234
Partition Key (partition_key
)
#
-
Environment variable:
TARGET_KINESIS_PARTITION_KEY
(Default - id
) The partition key to use when writing to Kinesis.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-kinesis set partition_key <partition_key>
export TARGET_KINESIS_PARTITION_KEY=<partition_key>
Looking for help? #
If you're having trouble getting the
target-kinesis
loader to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!