AWS S3 CSV
Table of Contents
The tap-s3-csv
Singer tap
pulls data from
AWS S3 CSV
that can then be sent to a destination using a
Singer target.
Alternative variants
Multiple
variants
of tap-s3-csv
are available. This document describes the
default transferwise
variant, which is recommended for new
users.
Alternative variants are:
Standalone usage
Install the package using pip:
pip install pipelinewise-tap-s3-csv
For additional instructions, refer to the README in the repository.
Usage with Meltano
Install Meltano, create your Meltano project, and add the tap to your project as a custom extractor:
meltano add --custom extractor tap-s3-csv
Then, configure the extractor, add any Singer target as a loader to send data to a destination and run a data integration (EL) pipeline.
Capabilities
Settings
AWS S3 Access Key ID (aws_access_key_id
)
S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
AWS S3 Secret Access Key (aws_secret_access_key
)
S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.
Start Date (start_date
)
Determines how much historical data will be extracted. Please be aware that the larger the time period and amount of data, the longer the initial extraction can be expected to take.
Bucket (bucket
)
AWS S3 bucket name
Tables (tables
)
An array that consists of one or more objects that describe how to find files and emit records. Required - table_name
and search_pattern
. Optional - key_properties
, search_prefix
, date_overrides
, delimiter
.
AWS Endpoint URL (aws_endpoint_url
)
The AWS endpoint URL.
AWS profile name (aws_profile
)
Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.
AWS S3 Session Token (aws_session_token
)
Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.
Looking for help?
If you're having trouble getting
tap-s3-csv
to work by itself or with
Meltano, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page?
This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.