Athena
Table of Contents
- Getting Started
- Capabilities
- Settings
-
Athena Database (
athena_database
) -
AWS Region (
aws_region
) -
AWS S3 Access Key ID (
aws_access_key_id
) -
AWS S3 Secret Access Key (
aws_secret_access_key
) -
AWS S3 Session Token (
aws_session_token
) -
AWS profile name (
aws_profile
) -
Bucket (
s3_bucket
) -
S3 Key Prefix (
s3_key_prefix
) -
S3 Staging Directory (
s3_staging_dir
) -
Delimiter (
delimiter
) -
Quote Character (
quotechar
) -
Add Record Metadata (
add_record_metadata
) -
Encryption Type (
encryption_type
) -
Encryption Key (
encryption_key
) -
Compression (
compression
) -
Naming Convention (
naming_convention
) -
Temp Directory (
temp_dir
) -
Object Format (
object_format
) -
Flattten Records (
flatten_records
)
-
Athena Database (
- Looking for help?
The target-athena
Meltano loader sends data into Athena after it was pulled from a source using an extractor.
-
- Repository: https://github.com/MeltanoLabs/target-athena
-
-
-
-
-
-
-
- Maintainer: Meltano Community
- Meltano Stats:
-
-
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
target-athena
loader to your project usingmeltano add
:meltano add loader target-athena
-
Configure the settings below using
meltano config
.
Next steps #
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.Capabilities #
Settings #
target-athena
requires the
configuration
of the following settings:
The settings for loader target-athena
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
Athena Database (athena_database
)
#
-
Environment variable:
TARGET_ATHENA_ATHENA_DATABASE
The name of the Athena database to connect to.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set athena_database <athena_database>
export TARGET_ATHENA_ATHENA_DATABASE=<athena_database>
AWS Region (aws_region
)
#
-
Environment variable:
TARGET_ATHENA_AWS_REGION
The AWS region to use i.e. us-east-2.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set aws_region <aws_region>
export TARGET_ATHENA_AWS_REGION=<aws_region>
AWS S3 Access Key ID (aws_access_key_id
)
#
-
Environment variable:
TARGET_ATHENA_AWS_ACCESS_KEY_ID
S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set aws_access_key_id <aws_access_key_id>
export TARGET_ATHENA_AWS_ACCESS_KEY_ID=<aws_access_key_id>
AWS S3 Secret Access Key (aws_secret_access_key
)
#
-
Environment variable:
TARGET_ATHENA_AWS_SECRET_ACCESS_KEY
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set aws_secret_access_key <aws_secret_access_key>
export TARGET_ATHENA_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
AWS S3 Session Token (aws_session_token
)
#
-
Environment variable:
TARGET_ATHENA_AWS_SESSION_TOKEN
Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set aws_session_token <aws_session_token>
export TARGET_ATHENA_AWS_SESSION_TOKEN=<aws_session_token>
AWS profile name (aws_profile
)
#
-
Environment variable:
TARGET_ATHENA_AWS_PROFILE
Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set aws_profile <aws_profile>
export TARGET_ATHENA_AWS_PROFILE=<aws_profile>
Bucket (s3_bucket
)
#
-
Environment variable:
TARGET_ATHENA_S3_BUCKET
AWS S3 bucket name
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set s3_bucket <s3_bucket>
export TARGET_ATHENA_S3_BUCKET=<s3_bucket>
S3 Key Prefix (s3_key_prefix
)
#
-
Environment variable:
TARGET_ATHENA_S3_KEY_PREFIX
A static prefix before the generated S3 key names. Using prefixes you can upload files into specific directories in the S3 bucket. Default(None)
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set s3_key_prefix <s3_key_prefix>
export TARGET_ATHENA_S3_KEY_PREFIX=<s3_key_prefix>
S3 Staging Directory (s3_staging_dir
)
#
-
Environment variable:
TARGET_ATHENA_S3_STAGING_DIR
S3 location to stage files. Example - s3://YOUR_S3_BUCKET/path/to/
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set s3_staging_dir <s3_staging_dir>
export TARGET_ATHENA_S3_STAGING_DIR=<s3_staging_dir>
Delimiter (delimiter
)
#
-
Environment variable:
TARGET_ATHENA_DELIMITER
(Default - ‘,’) A one-character string used to separate fields.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set delimiter <delimiter>
export TARGET_ATHENA_DELIMITER=<delimiter>
Quote Character (quotechar
)
#
-
Environment variable:
TARGET_ATHENA_QUOTECHAR
(Default - ‘”’) A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set quotechar <quotechar>
export TARGET_ATHENA_QUOTECHAR=<quotechar>
Add Record Metadata (add_record_metadata
)
#
-
Environment variable:
TARGET_ATHENA_ADD_RECORD_METADATA
(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix sdc. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _sdc_deleted_at metadata column. Without the add_record_metadata option the deleted rows from singer taps will not be recongisable in Snowflake.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set add_record_metadata true
export TARGET_ATHENA_ADD_RECORD_METADATA=true
Encryption Type (encryption_type
)
#
-
Environment variable:
TARGET_ATHENA_ENCRYPTION_TYPE
(Default - ‘none’) The type of encryption to use. Current supported options are - ‘none’ and ‘KMS’.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set encryption_type <encryption_type>
export TARGET_ATHENA_ENCRYPTION_TYPE=<encryption_type>
Encryption Key (encryption_key
)
#
-
Environment variable:
TARGET_ATHENA_ENCRYPTION_KEY
A reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. ‘1234abcd-1234-1234-1234-1234abcd1234’). This field is ignored if ‘encryption_type’ is none or blank.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set encryption_key <encryption_key>
export TARGET_ATHENA_ENCRYPTION_KEY=<encryption_key>
Compression (compression
)
#
-
Environment variable:
TARGET_ATHENA_COMPRESSION
The type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set compression <compression>
export TARGET_ATHENA_COMPRESSION=<compression>
Naming Convention (naming_convention
)
#
-
Environment variable:
TARGET_ATHENA_NAMING_CONVENTION
(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports “folders” in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3key_prefix, if set, by prepending the “filename”. E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix results in folder1/prefix_my_file.csv
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set naming_convention <naming_convention>
export TARGET_ATHENA_NAMING_CONVENTION=<naming_convention>
Temp Directory (temp_dir
)
#
-
Environment variable:
TARGET_ATHENA_TEMP_DIR
(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set temp_dir <temp_dir>
export TARGET_ATHENA_TEMP_DIR=<temp_dir>
Object Format (object_format
)
#
-
Environment variable:
TARGET_ATHENA_OBJECT_FORMAT
(Default - jsonl) The format to write the S3 in - jsonl or csv
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set object_format <object_format>
export TARGET_ATHENA_OBJECT_FORMAT=<object_format>
Flattten Records (flatten_records
)
#
-
Environment variable:
TARGET_ATHENA_FLATTEN_RECORDS
Whether to flatten nested import data before loading it to Athena.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config target-athena set flatten_records true
export TARGET_ATHENA_FLATTEN_RECORDS=true
Looking for help? #
If you're having trouble getting the
target-athena
loader to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!