AWS S3 CSV
Table of Contents
The tap-s3-csv
Meltano extractor pulls data from AWS S3 CSV that can then be sent to a destination using a loader.
Alternative variants #
Multiple
variants
of tap-s3-csv
are available.
This document describes the default transferwise
variant,
which is recommended for new users.
Alternative variants are:
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
tap-s3-csv
extractor to your project usingmeltano add
:meltano add extractor tap-s3-csv
-
Configure the settings below using
meltano config
.
Next steps #
Follow the remaining steps of the Getting Started guide:
- Select entities and attributes to extract
- Add a loader to send data to a destination
- Run a data integration (EL) pipeline
Capabilities #
Settings #
tap-s3-csv
requires the
configuration
of the following settings:
The settings for extractor tap-s3-csv
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
AWS S3 Access Key ID (aws_access_key_id
)
#
-
Environment variable:
TAP_S3_CSV_AWS_ACCESS_KEY_ID
S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set aws_access_key_id <aws_access_key_id>
export TAP_S3_CSV_AWS_ACCESS_KEY_ID=<aws_access_key_id>
AWS S3 Secret Access Key (aws_secret_access_key
)
#
-
Environment variable:
TAP_S3_CSV_AWS_SECRET_ACCESS_KEY
S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set aws_secret_access_key <aws_secret_access_key>
export TAP_S3_CSV_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
Start Date (start_date
)
#
-
Environment variable:
TAP_S3_CSV_START_DATE
Determines how much historical data will be extracted. Please be aware that the larger the time period and amount of data, the longer the initial extraction can be expected to take.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set start_date YYYY-MM-DDTHH:MM:SSZ
export TAP_S3_CSV_START_DATE=YYYY-MM-DDTHH:MM:SSZ
Bucket (bucket
)
#
-
Environment variable:
TAP_S3_CSV_BUCKET
AWS S3 bucket name
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set bucket <bucket>
export TAP_S3_CSV_BUCKET=<bucket>
Tables (tables
)
#
-
Environment variable:
TAP_S3_CSV_TABLES
An array that consists of one or more objects that describe how to find files and emit records. Required - table_name
and search_pattern
. Optional - key_properties
, search_prefix
, date_overrides
, delimiter
.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set tables '[...]'
export TAP_S3_CSV_TABLES='[...]'
AWS Endpoint URL (aws_endpoint_url
)
#
-
Environment variable:
TAP_S3_CSV_AWS_ENDPOINT_URL
The AWS endpoint URL.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set aws_endpoint_url <aws_endpoint_url>
export TAP_S3_CSV_AWS_ENDPOINT_URL=<aws_endpoint_url>
AWS profile name (aws_profile
)
#
-
Environment variable:
TAP_S3_CSV_AWS_PROFILE
Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set aws_profile <aws_profile>
export TAP_S3_CSV_AWS_PROFILE=<aws_profile>
AWS S3 Session Token (aws_session_token
)
#
-
Environment variable:
TAP_S3_CSV_AWS_SESSION_TOKEN
Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-s3-csv set aws_session_token <aws_session_token>
export TAP_S3_CSV_AWS_SESSION_TOKEN=<aws_session_token>
Looking for help? #
If you're having trouble getting the
tap-s3-csv
extractor to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!