target-athena - Meltano Hub

Amazon Athena

target-athena (meltanolabs variant)🥈

Serverless S3 Database

The target-athena loader sends data into Amazon Athena after it was pulled from a source using an extractor

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the target-athena loader to your project using
```
meltano add
```
:

meltano add target-athena

Configure the target-athena settings using

meltano config

:

meltano config target-athena set --interactive

Next steps

Follow the remaining steps of the Getting Started guide:

Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for target-athena may have been automatically set when originally added to the Hub. Please review the capabilities when using this loader. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this loader.

This plugin has the following capabilities:

about
stream-maps

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The target-athena settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

add_record_metadata
athena_database
aws_access_key_id
aws_profile
aws_region
aws_secret_access_key
aws_session_token
compression
delimiter
encryption_key
encryption_type
flatten_records
naming_convention
object_format
quotechar
s3_bucket
s3_key_prefix
s3_staging_dir
temp_dir

You can also list these settings using

meltano config

with the list subcommand:

meltano config target-athena list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Add Record Metadata (add_record_metadata)

Environment variable: TARGET_ATHENA_ADD_RECORD_METADATA

(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix sdc. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _sdc_deleted_at metadata column. Without the add_record_metadata option the deleted rows from singer taps will not be recongisable in Snowflake.

Configure this setting directly using the following Meltano command:

meltano config target-athena set add_record_metadata [value]

Athena Database (athena_database)

Environment variable: TARGET_ATHENA_ATHENA_DATABASE

The name of the Athena database to connect to.

Configure this setting directly using the following Meltano command:

meltano config target-athena set athena_database [value]

AWS S3 Access Key ID (aws_access_key_id)

Environment variable: TARGET_ATHENA_AWS_ACCESS_KEY_ID

S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

Configure this setting directly using the following Meltano command:

meltano config target-athena set aws_access_key_id [value]

AWS profile name (aws_profile)

Environment variable: TARGET_ATHENA_AWS_PROFILE

Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.

Configure this setting directly using the following Meltano command:

meltano config target-athena set aws_profile [value]

AWS Region (aws_region)

Environment variable: TARGET_ATHENA_AWS_REGION

The AWS region to use i.e. us-east-2.

Configure this setting directly using the following Meltano command:

meltano config target-athena set aws_region [value]

AWS S3 Secret Access Key (aws_secret_access_key)

Environment variable: TARGET_ATHENA_AWS_SECRET_ACCESS_KEY

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config target-athena set aws_secret_access_key [value]

AWS S3 Session Token (aws_session_token)

Environment variable: TARGET_ATHENA_AWS_SESSION_TOKEN

Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.

Configure this setting directly using the following Meltano command:

meltano config target-athena set aws_session_token [value]

Compression (compression)

Environment variable: TARGET_ATHENA_COMPRESSION

The type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files.

Configure this setting directly using the following Meltano command:

meltano config target-athena set compression [value]

Delimiter (delimiter)

Environment variable: TARGET_ATHENA_DELIMITER

(Default - ',') A one-character string used to separate fields.

Configure this setting directly using the following Meltano command:

meltano config target-athena set delimiter [value]

Encryption Key (encryption_key)

Environment variable: TARGET_ATHENA_ENCRYPTION_KEY

A reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. '1234abcd-1234-1234-1234-1234abcd1234'). This field is ignored if 'encryption_type' is none or blank.

Configure this setting directly using the following Meltano command:

meltano config target-athena set encryption_key [value]

Encryption Type (encryption_type)

Environment variable: TARGET_ATHENA_ENCRYPTION_TYPE

(Default - 'none') The type of encryption to use. Current supported options are - 'none' and 'KMS'.

Configure this setting directly using the following Meltano command:

meltano config target-athena set encryption_type [value]

Flattten Records (flatten_records)

Environment variable: TARGET_ATHENA_FLATTEN_RECORDS

Whether to flatten nested import data before loading it to Athena.

Configure this setting directly using the following Meltano command:

meltano config target-athena set flatten_records [value]

Naming Convention (naming_convention)

Environment variable: TARGET_ATHENA_NAMING_CONVENTION

(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports "folders" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the "filename". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv

Configure this setting directly using the following Meltano command:

meltano config target-athena set naming_convention [value]

Object Format (object_format)

Environment variable: TARGET_ATHENA_OBJECT_FORMAT

(Default - jsonl) The format to write the S3 in - jsonl or csv

Configure this setting directly using the following Meltano command:

meltano config target-athena set object_format [value]

Quote Character (quotechar)

Environment variable: TARGET_ATHENA_QUOTECHAR

(Default - '"') A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

Configure this setting directly using the following Meltano command:

meltano config target-athena set quotechar [value]

Bucket (s3_bucket)

Environment variable: TARGET_ATHENA_S3_BUCKET

AWS S3 bucket name

Configure this setting directly using the following Meltano command:

meltano config target-athena set s3_bucket [value]

S3 Key Prefix (s3_key_prefix)

Environment variable: TARGET_ATHENA_S3_KEY_PREFIX

A static prefix before the generated S3 key names. Using prefixes you can upload files into specific directories in the S3 bucket. Default(None)

Configure this setting directly using the following Meltano command:

meltano config target-athena set s3_key_prefix [value]

S3 Staging Directory (s3_staging_dir)

Environment variable: TARGET_ATHENA_S3_STAGING_DIR

S3 location to stage files. Example - s3://YOUR_S3_BUCKET/path/to/

Configure this setting directly using the following Meltano command:

meltano config target-athena set s3_staging_dir [value]

Temp Directory (temp_dir)

Environment variable: TARGET_ATHENA_TEMP_DIR

(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.

Configure this setting directly using the following Meltano command:

meltano config target-athena set temp_dir [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-athena loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Install

meltano add
              target-athena

Maintenance Status

Repo

https://github.com/MeltanoLabs/target-athena

Maintainer

Meltano Stats

Keywords