S3 Access Key Id. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

The AWS endpoint URL.

Optional - AWS profile name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.

S3 Secret Access Key. If not provided, aws_profile or AWS_ACCESS_KEY_ID environment variable will be used.

Optional - S3 AWS STS token for temporary credentials. If not provided, AWS_SESSION_TOKEN environment variable will be used.

AWS S3 bucket name

Optional - A dict of proxies settings for use of a proxy server. Set to {} to avoid using a proxy server for s3 traffic.

Optional - When set true will emit null (the JSON equivalent of None) instead of an empty string.

Determines how much historical data will be extracted. Please be aware that the larger the time period and amount of data, the longer the initial extraction can be expected to take.

Optional - If set will append a suffix on each of the tables to provide some uniqueness e.g. a date or supplier identifier.

An array that consists of one or more objects that describe how to find files and emit records. Required - table_name and search_pattern. Optional - key_properties, search_prefix, datatype_overrides, date_overrides, delimiter, remove_character, string_overrides, guess_types, encoding.

• search_prefix - This is a prefix to apply after the bucket, but before the file search pattern, to allow you to find files in "directories" below the bucket.
• search_pattern - This is an escaped regular expression that the tap will use to find files in the bucket + prefix. It's a bit strange, since this is an escaped string inside of an escaped string, any backslashes in the RegEx will need to be double-escaped.
• table_name - This value is a string of your choosing, and will be used to name the stream that records are emitted under for files matching content.
• key_properties - These are the "primary keys" of the CSV files, to be used by the target for deduplication and primary key definitions downstream in the destination.
• datatype_overrides - A object / dictionary of header names in the file and any override datatype other than string you with to set as the datatype. Example config: "datatype_overrides":{"administration_number":"integer","percentage": "number","grade":"integer"}.
• date_overrides - Specifies field names in the files that are supposed to be parsed as a datetime. The tap doesn't attempt to automatically determine if a field is a datetime, so this will make it explicit in the discovered schema.
• delimiter - This allows you to specify a custom delimiter, such as \t or |, if that applies to your files.
• string_overrides - DEPRECATED: Specifies field names in the files that should be parsed as a string regardless of what was discovered.
• guess_types - DEPRECATED: (default True) By default, column data types will be determined via scanning the first file in a table_spec. Set this to False to disable this and set all columns to string.
• remove_character - Specifies a character which can be removed from each line in the the file e.g. "\"" will remove all double-quotes.
• encoding - The encoding to use to read these files from codecs -> Standard Encodings

Optional - Will attempt to log a warning rather than error if there are no files found for the search criteria if the setting is set to true.