The target-s3-avro loader sends data into S3 Avro after it was pulled from a source using an extractor
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the target-s3-avro loader to your
project using
:meltano add -
Configure the target-s3-avro
settings using
:meltano config
meltano add loader target-s3-avromeltano config target-s3-avro set --interactiveNext steps
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
The
target-s3-avro settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
api_versionaws_access_key_idaws_secret_access_keyaws_session_tokenconfigendpoint_urlflatten_delimiterregion_nametarget_bucket_keytarget_schema_bucket_keytmp_diruse_sslverify
You can also list these settings using
with the meltano configlist
subcommand:
meltano config target-s3-avro list
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Api Version (api_version)
-
Environment variable:
TARGET_S3_AVRO_API_VERSION
The API version to use. By default, botocore will use the latest API version when creating a client. You only need to specify this parameter if you want to use a previous API version of the client.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set api_version [value]Aws Access Key Id (aws_access_key_id)
-
Environment variable:
TARGET_S3_AVRO_AWS_ACCESS_KEY_ID
The access key to use when creating the client. This is entirely optional, and if not provided, the credentials configured for the session will automatically be used. You only need to provide this argument if you want to override the credentials used for this specific client.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set aws_access_key_id [value]Aws Secret Access Key (aws_secret_access_key)
-
Environment variable:
TARGET_S3_AVRO_AWS_SECRET_ACCESS_KEY
The secret key to use when creating the client. Same semantics as aws_access_key_id above.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set aws_secret_access_key [value]Aws Session Token (aws_session_token)
-
Environment variable:
TARGET_S3_AVRO_AWS_SESSION_TOKEN
The session token to use when creating the client. Same semantics as aws_access_key_id.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set aws_session_token [value]Config (config)
-
Environment variable:
TARGET_S3_AVRO_CONFIG
Advanced client configuration options. If region_name is specified in the client config, its value will take precedence over environment variables and configuration values, but not over a region_name value passed explicitly to the method. See botocore config documentation for more details. See boto Docs for more details.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set config [value]Endpoint Url (endpoint_url)
-
Environment variable:
TARGET_S3_AVRO_ENDPOINT_URL
The complete URL to use for the constructed client. Normally, botocore will automatically construct the appropriate URL to use when communicating with a service. You can specify a complete URL (including the "http/https" scheme) to override this behavior. If this value is provided, then use_ssl is ignored.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set endpoint_url [value]Flatten Delimiter (flatten_delimiter)
-
Environment variable:
TARGET_S3_AVRO_FLATTEN_DELIMITER
The delimiter to use when flatttening records.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set flatten_delimiter [value]Region Name (region_name)
-
Environment variable:
TARGET_S3_AVRO_REGION_NAME
The name of the region associated with the client. A client is associated with a single region.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set region_name [value]Target Bucket Key (target_bucket_key)
-
Environment variable:
TARGET_S3_AVRO_TARGET_BUCKET_KEY
Destination bucket and key folder in the format of {bucket}/{key}.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set target_bucket_key [value]Target Schema Bucket Key (target_schema_bucket_key)
-
Environment variable:
TARGET_S3_AVRO_TARGET_SCHEMA_BUCKET_KEY
Schema destination bucket and key folder in the format of {bucket}/{key}
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set target_schema_bucket_key [value]Tmp Dir (tmp_dir)
-
Environment variable:
TARGET_S3_AVRO_TMP_DIR
The directory path for where to write temp files. Default to current working directory.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set tmp_dir [value]Use SSL (use_ssl)
-
Environment variable:
TARGET_S3_AVRO_USE_SSL
Whether or not to use SSL. By default, SSL is used. Note that not all services support non-ssl connections.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set use_ssl [value]Verify (verify)
-
Environment variable:
TARGET_S3_AVRO_VERIFY
Whether or not to verify SSL certificates.
Configure this setting directly using the following Meltano command:
meltano config target-s3-avro set verify [value]Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general
