Athena (meltanolabs
variant)
Table of Contents
- Standalone usage
- Usage with Meltano
- Capabilities
- Settings
-
Athena Database (
athena_database
) -
AWS Region (
aws_region
) -
AWS S3 Access Key ID (
aws_access_key_id
) -
AWS S3 Secret Access Key (
aws_secret_access_key
) -
AWS S3 Session Token (
aws_session_token
) -
AWS profile name (
aws_profile
) -
Bucket (
s3_bucket
) -
S3 Key Prefix (
s3_key_prefix
) -
S3 Staging Directory (
s3_staging_dir
) -
Delimiter (
delimiter
) -
Quote Character (
quotechar
) -
Add Record Metadata (
add_record_metadata
) -
Encryption Type (
encryption_type
) -
Encryption Key (
encryption_key
) -
Compression (
compression
) -
Naming Convention (
naming_convention
) -
Temp Directory (
temp_dir
) -
Object Format (
object_format
) -
Flattten Records (
flatten_records
)
-
Athena Database (
- Looking for help?
The
target-athena
Singer target
sends data into
Athena
after it was pulled from a source using a
Singer tap.
-
- Repository: https://github.com/MeltanoLabs/target-athena
-
-
-
-
-
-
-
- Maintainer: Meltano Community
- Meltano Stats:
-
-
Standalone usage #
Install the package using pip:
pip install git+https://github.com/MeltanoLabs/target-athena.git
For additional instructions, refer to the README in the repository.
Usage with Meltano #
Install Meltano, create your Meltano project, and add the target to your project as a custom loader:
meltano add --custom loader target-athena --variant meltanolabs
Then, configure the loader, add any Singer tap as an extractor to pull data from a source and run a data integration (EL) pipeline.
Capabilities #
Settings #
Athena Database (athena_database
)
#
The name of the Athena database to connect to.
AWS Region (aws_region
)
#
The AWS region to use i.e. us-east-2.
AWS S3 Access Key ID (aws_access_key_id
)
#
S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
AWS S3 Secret Access Key (aws_secret_access_key
)
#
AWS S3 Session Token (aws_session_token
)
#
Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.
AWS profile name (aws_profile
)
#
Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.
Bucket (s3_bucket
)
#
AWS S3 bucket name
S3 Key Prefix (s3_key_prefix
)
#
A static prefix before the generated S3 key names. Using prefixes you can upload files into specific directories in the S3 bucket. Default(None)
S3 Staging Directory (s3_staging_dir
)
#
S3 location to stage files. Example - s3://YOUR_S3_BUCKET/path/to/
Delimiter (delimiter
)
#
(Default - ‘,’) A one-character string used to separate fields.
Quote Character (quotechar
)
#
(Default - ‘”’) A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
Add Record Metadata (add_record_metadata
)
#
(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix sdc. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _sdc_deleted_at metadata column. Without the add_record_metadata option the deleted rows from singer taps will not be recongisable in Snowflake.
Encryption Type (encryption_type
)
#
(Default - ‘none’) The type of encryption to use. Current supported options are - ‘none’ and ‘KMS’.
Encryption Key (encryption_key
)
#
A reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. ‘1234abcd-1234-1234-1234-1234abcd1234’). This field is ignored if ‘encryption_type’ is none or blank.
Compression (compression
)
#
The type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files.
Naming Convention (naming_convention
)
#
(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports “folders” in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3key_prefix, if set, by prepending the “filename”. E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix results in folder1/prefix_my_file.csv
Temp Directory (temp_dir
)
#
(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.
Object Format (object_format
)
#
(Default - jsonl) The format to write the S3 in - jsonl or csv
Flattten Records (flatten_records
)
#
Whether to flatten nested import data before loading it to Athena.
Looking for help? #
If you're having trouble getting
target-athena
to work by itself or with
Meltano, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.