S3 Avro

target-s3-avro (faumel variant)

Avro serialization format written to AWS S3.

The target-s3-avro loader sends data into S3 Avro after it was pulled from a source using an extractor

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

  1. Add the target-s3-avro loader to your project using
    meltano add
    :
  2. meltano add loader target-s3-avro
  3. Configure the target-s3-avro settings using
    meltano config
    :
  4. meltano config target-s3-avro set --interactive

Next steps

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

The target-s3-avro settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using

meltano config
with the list subcommand:

meltano config target-s3-avro list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Verify (verify)

  • Environment variable: TARGET_S3_AVRO_VERIFY

Whether or not to verify SSL certificates.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set verify [value]

Aws Session Token (aws_session_token)

  • Environment variable: TARGET_S3_AVRO_AWS_SESSION_TOKEN

The session token to use when creating the client. Same semantics as aws_access_key_id.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set aws_session_token [value]

Api Version (api_version)

  • Environment variable: TARGET_S3_AVRO_API_VERSION

The API version to use. By default, botocore will use the latest API version when creating a client. You only need to specify this parameter if you want to use a previous API version of the client.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set api_version [value]

Endpoint Url (endpoint_url)

  • Environment variable: TARGET_S3_AVRO_ENDPOINT_URL

The complete URL to use for the constructed client. Normally, botocore will automatically construct the appropriate URL to use when communicating with a service. You can specify a complete URL (including the "http/https" scheme) to override this behavior. If this value is provided, then use_ssl is ignored.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set endpoint_url [value]

Aws Secret Access Key (aws_secret_access_key)

  • Environment variable: TARGET_S3_AVRO_AWS_SECRET_ACCESS_KEY

The secret key to use when creating the client. Same semantics as aws_access_key_id above.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set aws_secret_access_key [value]

Aws Access Key Id (aws_access_key_id)

  • Environment variable: TARGET_S3_AVRO_AWS_ACCESS_KEY_ID

The access key to use when creating the client. This is entirely optional, and if not provided, the credentials configured for the session will automatically be used. You only need to provide this argument if you want to override the credentials used for this specific client.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set aws_access_key_id [value]

Flatten Delimiter (flatten_delimiter)

  • Environment variable: TARGET_S3_AVRO_FLATTEN_DELIMITER

The delimiter to use when flatttening records.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set flatten_delimiter [value]

Region Name (region_name)

  • Environment variable: TARGET_S3_AVRO_REGION_NAME

The name of the region associated with the client. A client is associated with a single region.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set region_name [value]

Tmp Dir (tmp_dir)

  • Environment variable: TARGET_S3_AVRO_TMP_DIR

The directory path for where to write temp files. Default to current working directory.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set tmp_dir [value]

Use SSL (use_ssl)

  • Environment variable: TARGET_S3_AVRO_USE_SSL

Whether or not to use SSL. By default, SSL is used. Note that not all services support non-ssl connections.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set use_ssl [value]

Target Schema Bucket Key (target_schema_bucket_key)

  • Environment variable: TARGET_S3_AVRO_TARGET_SCHEMA_BUCKET_KEY

Schema destination bucket and key folder in the format of {bucket}/{key}


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set target_schema_bucket_key [value]

Config (config)

  • Environment variable: TARGET_S3_AVRO_CONFIG

Advanced client configuration options. If region_name is specified in the client config, its value will take precedence over environment variables and configuration values, but not over a region_name value passed explicitly to the method. See botocore config documentation for more details. See boto Docs for more details.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set config [value]

Target Bucket Key (target_bucket_key)

  • Environment variable: TARGET_S3_AVRO_TARGET_BUCKET_KEY

Destination bucket and key folder in the format of {bucket}/{key}.


Configure this setting directly using the following Meltano command:

meltano config target-s3-avro set target_bucket_key [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-s3-avro loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the
#plugins-general
channel.

Install

meltano add loader target-s3-avro

Maintenance Status

  • Maintenance Status

Repo

https://github.com/faumel/target-s3-avro
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • faumel

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

Keywords

  • awsfile