target-s3 - Meltano Hub

S3 (Multi Format)

Meltano Community

target-s3 (plinxore variant)

Open source connector supported by the Meltano Community, a growing community of over 5500+ data engineers.

Open source Join Slack community

The target-s3 loader sends data into S3 (Multi Format) after it was pulled from a source using an extractor

Alternate Implementations

crowemi
(default)🥇
plinxore

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the target-s3 loader to your project using
```
meltano add
```
:

meltano add target-s3 --variant plinxore

Configure the target-s3 settings using

meltano config

:

meltano config target-s3 set --interactive

Next steps

Follow the remaining steps of the Getting Started guide:

Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for target-s3 may have been automatically set when originally added to the Hub. Please review the capabilities when using this loader. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this loader.

This plugin has the following capabilities:

about
schema-flattening
stream-maps

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The target-s3 settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

Expand To Show SDK Settings

add_record_metadata
flattening_enabled
flattening_max_depth
load_method
stream_map_config
stream_maps

You can also list these settings using

meltano config

with the list subcommand:

meltano config target-s3 list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Append Date To Filename (append_date_to_filename)

Environment variable: TARGET_S3_APPEND_DATE_TO_FILENAME
Default Value: true

A flag to append the date to the key filename, for readability. Every filename already carries a monotonic '-part-NNNNN' batch counter regardless of this setting, so this is purely cosmetic -- it is not what guarantees batch-to-batch uniqueness.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set append_date_to_filename [value]

Append Date To Filename Grain (append_date_to_filename_grain)

Environment variable: TARGET_S3_APPEND_DATE_TO_FILENAME_GRAIN
Default Value: day

The grain of the date to append to the filename.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set append_date_to_filename_grain [value]

Append Date To Prefix (append_date_to_prefix)

Environment variable: TARGET_S3_APPEND_DATE_TO_PREFIX
Default Value: true

A flag to append the date to the key prefix.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set append_date_to_prefix [value]

Append Date To Prefix Grain (append_date_to_prefix_grain)

Environment variable: TARGET_S3_APPEND_DATE_TO_PREFIX_GRAIN
Default Value: day

The grain of the date to append to the prefix.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set append_date_to_prefix_grain [value]

Append Uuid (append_uuid)

Environment variable: TARGET_S3_APPEND_UUID
Default Value: true

Whether to append a random UUID to the key filename, after the '-part-NNNNN' batch counter (which is always present and unaffected by this setting). 'true' (default) guarantees no two runs can ever collide on the same key -- a same-day retry gets fresh UUIDs and its objects accumulate alongside the previous run's instead of overwriting them, which is the safe default for a published target. Set to 'false' to make filenames reusable across runs with the same batch layout, so a rerun overwrites the previous run's objects on matching keys instead of accumulating -- this is one of three things a real full-refresh overwrite needs, not the whole mechanism: it must be combined with purging the key prefix before the run (orchestration) and reading from a fixed path (the consumer). This target only controls naming; it does not purge, deduplicate, or know whether a given run is a full refresh.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set append_uuid [value]

Cloud Provider AWS AWS Access Key ID (cloud_provider.aws.aws_access_key_id)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_ACCESS_KEY_ID

Your AWS access key ID.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_access_key_id [value]

Cloud Provider AWS AWS Bucket (cloud_provider.aws.aws_bucket)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_BUCKET

The name of the S3 bucket to write to.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_bucket [value]

Cloud Provider AWS AWS Endpoint Override (cloud_provider.aws.aws_endpoint_override)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_ENDPOINT_OVERRIDE

S3-compatible endpoint URL override, e.g. for MinIO or other non-AWS S3-compatible storage.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_endpoint_override [value]

Cloud Provider AWS AWS Profile Name (cloud_provider.aws.aws_profile_name)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_PROFILE_NAME

Your AWS profile name to optionally use for auth.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_profile_name [value]

Cloud Provider AWS AWS Region (cloud_provider.aws.aws_region)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_REGION

Your AWS region.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_region [value]

Cloud Provider AWS AWS Secret Access Key (cloud_provider.aws.aws_secret_access_key)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_SECRET_ACCESS_KEY

Your AWS secret access key.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_secret_access_key [value]

Cloud Provider AWS AWS Session Token (cloud_provider.aws.aws_session_token)

Environment variable: TARGET_S3_CLOUD_PROVIDER_AWS_AWS_SESSION_TOKEN

Optional AWS session token for temporary credentials.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider aws.aws_session_token [value]

Cloud Provider Cloud Provider Type (cloud_provider.cloud_provider_type)

Environment variable: TARGET_S3_CLOUD_PROVIDER_CLOUD_PROVIDER_TYPE

The cloud provider type to use.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set cloud_provider cloud_provider_type [value]

Compression (compression)

Environment variable: TARGET_S3_COMPRESSION
Default Value: gzip

Compression to apply to written files. 'gzip' appends a .gz suffix and produces real gzip output; 'none' disables compression.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set compression [value]

Datetime Error Treatment (datetime_error_treatment)

Environment variable: TARGET_S3_DATETIME_ERROR_TREATMENT
Default Value: null

How to handle date/date-time values the SDK cannot parse (e.g. MySQL/MyISAM zero-dates like '0000-00-00 00:00:00'). 'null' replaces the value with null, 'max' replaces it with the max representable timestamp, 'error' aborts the run.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set datetime_error_treatment [value]

Format Format Csv (format.format_csv)

Environment variable: TARGET_S3_FORMAT_FORMAT_CSV

Configuration object for the csv format type (not yet implemented).

Configure this setting directly using the following Meltano command:

meltano config target-s3 set format format_csv [value]

Format Format Json (format.format_json)

Environment variable: TARGET_S3_FORMAT_FORMAT_JSON

Configuration object for the json format type.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set format format_json [value]

Format Format Parquet Get Schema From Tap (format.format_parquet.get_schema_from_tap)

Environment variable: TARGET_S3_FORMAT_FORMAT_PARQUET_GET_SCHEMA_FROM_TAP
Default Value: true

Derive the Parquet schema once from the stream's Singer SCHEMA message and reuse it for every batch of that stream, instead of letting pyarrow infer a schema independently per batch. Per-batch inference (set this to false to use it) is what causes schema drift between Parquet files of the same stream -- a batch where a column happens to be all-null infers that column as pyarrow's null type, while a later batch with real values infers a real type, producing files that can't be read together. Doesn't work with 'anyOf' types or when complex data is not defined at element level. Doesn't work with the validate option (validate only applies to the per-batch-inference path).

Configure this setting directly using the following Meltano command:

meltano config target-s3 set format format_parquet.get_schema_from_tap [value]

Format Format Parquet Validate (format.format_parquet.validate)

Environment variable: TARGET_S3_FORMAT_FORMAT_PARQUET_VALIDATE
Default Value: false

This flag determines whether the data types of incoming data elements should be validated. When set to true, a schema is created from the first record and all subsequent records that don't match that data type are cast.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set format format_parquet.validate [value]

Format Format Type (format.format_type)

Environment variable: TARGET_S3_FORMAT_FORMAT_TYPE

The format type of the output files.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set format format_type [value]

Include Process Date (include_process_date)

Environment variable: TARGET_S3_INCLUDE_PROCESS_DATE
Default Value: false

A flag indicating whether to append _process_date to record.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set include_process_date [value]

Max Batch Age (max_batch_age)

Environment variable: TARGET_S3_MAX_BATCH_AGE
Default Value: 5

Maximum time in minutes between state messages when records are streamed in.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set max_batch_age [value]

Max Batch Size (max_batch_size)

Environment variable: TARGET_S3_MAX_BATCH_SIZE
Default Value: 10000

Maximum size of batches when records are streamed in.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set max_batch_size [value]

Partition By (partition_by)

Environment variable: TARGET_S3_PARTITION_BY

List of key-value strings (e.g., 'tenant=${TENANT}') to prepend as partitions in the S3 key path after the stream name.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set partition_by [value]

Partition Name Enabled (partition_name_enabled)

Environment variable: TARGET_S3_PARTITION_NAME_ENABLED
Default Value: false

A flag (only works if append_date_to_prefix is enabled) to have partitioning name formatted e.g. 'year=2023/month=01/day=01'.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set partition_name_enabled [value]

Prefix (prefix)

Environment variable: TARGET_S3_PREFIX

The prefix for the key.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set prefix [value]

Stream Name Path Override (stream_name_path_override)

Environment variable: TARGET_S3_STREAM_NAME_PATH_OVERRIDE

The S3 key stream name override.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set stream_name_path_override [value]

Use Raw Stream Name (use_raw_stream_name)

Environment variable: TARGET_S3_USE_RAW_STREAM_NAME
Default Value: false

A flag to force the filename to be identical to the stream name.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set use_raw_stream_name [value]

Expand To Show SDK Settings

Add Record Metadata (add_record_metadata)

Environment variable: TARGET_S3_ADD_RECORD_METADATA

Add metadata to records.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set add_record_metadata [value]

Flattening Enabled (flattening_enabled)

Environment variable: TARGET_S3_FLATTENING_ENABLED

'True' to enable schema flattening and automatically expand nested properties.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set flattening_enabled [value]

Flattening Max Depth (flattening_max_depth)

Environment variable: TARGET_S3_FLATTENING_MAX_DEPTH

The max depth to flatten schemas.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set flattening_max_depth [value]

Load Method (load_method)

Environment variable: TARGET_S3_LOAD_METHOD
Default Value: append-only

The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set load_method [value]

Stream Map Config (stream_map_config)

Environment variable: TARGET_S3_STREAM_MAP_CONFIG

User-defined config values to be used within map expressions.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set stream_map_config [value]

Stream Maps (stream_maps)

Environment variable: TARGET_S3_STREAM_MAPS

Config object for stream maps capability. For more information check out Stream Maps.

Configure this setting directly using the following Meltano command:

meltano config target-s3 set stream_maps [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-s3 loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Meltano Cloud

For ambitious analytics engineering teams that want an efficient, fully managed solution with personalised expert support, explore Meltano Cloud

No maintenance overhead
Rigorously tested
Expert support from Meltano

Learn more Book a demo

Install

meltano add
                target-s3 --variant plinxore

Maintenance Status

Repo

https://github.com/plinxore/target-s3

Maintainer

PyPI Stats

Keywords