The target-hdfs loader sends data into HDFS Parquet after it was pulled from a source using an extractor
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the target-hdfs loader to your
project using
:meltano add -
Configure the target-hdfs
settings using
:meltano config
meltano add loader target-hdfsmeltano config target-hdfs set --interactiveNext steps
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
The
target-hdfs settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
compression_extensioncompression_methodfile_prefixfile_size_mbfilename_separatorhdfs_destination_pathmax_queue_sizepartitionsrows_per_filestreams_in_separate_folder
You can also list these settings using
with the meltano configlist
subcommand:
meltano config target-hdfs list
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Compression Extension (compression_extension)
-
Environment variable:
TARGET_HDFS_COMPRESSION_EXTENSION
The compression extension to include in the file names.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set compression_extension [value]Compression Method (compression_method)
-
Environment variable:
TARGET_HDFS_COMPRESSION_METHOD
The compression method to use.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set compression_method [value]File Prefix (file_prefix)
-
Environment variable:
TARGET_HDFS_FILE_PREFIX
The prefix to add to each output file.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set file_prefix [value]File Size Mb (file_size_mb)
-
Environment variable:
TARGET_HDFS_FILE_SIZE_MB
The size of the output files.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set file_size_mb [value]Filename Separator (filename_separator)
-
Environment variable:
TARGET_HDFS_FILENAME_SEPARATOR
The separator to use in the file name.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set filename_separator [value]Hdfs Destination Path (hdfs_destination_path)
-
Environment variable:
TARGET_HDFS_HDFS_DESTINATION_PATH
The destination path in HDFS.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set hdfs_destination_path [value]Max Queue Size (max_queue_size)
-
Environment variable:
TARGET_HDFS_MAX_QUEUE_SIZE
The max size of the queue.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set max_queue_size [value]Partitions (partitions)
-
Environment variable:
TARGET_HDFS_PARTITIONS
A partition value to use for separating output files.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set partitions [value]Rows Per File (rows_per_file)
-
Environment variable:
TARGET_HDFS_ROWS_PER_FILE
The number of rows per file.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set rows_per_file [value]Streams In Separate Folder (streams_in_separate_folder)
-
Environment variable:
TARGET_HDFS_STREAMS_IN_SEPARATE_FOLDER
Whether to put stream data in separate folders or not.
Configure this setting directly using the following Meltano command:
meltano config target-hdfs set streams_in_separate_folder [value]Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general-brightgreen){kind=icon}
-4-c0c0c4){kind=icon}
-2-c0c0c4){kind=icon}
