The target-parquet Singer target sends data into Parquet after it was pulled from a source using a Singer tap.

Alternative variants

Multiple variants of target-parquet are available. This document describes the default estrategiahq variant, which is recommended for new users.

Alternative variants are:

Standalone usage

Install the package using pip:

pip install git+https://github.com/estrategiahq/target-parquet.git

For additional instructions, refer to the README in the repository.

Usage with Meltano

Install Meltano, create your Meltano project, and add the target to your project as a custom loader:

meltano add --custom loader target-parquet

Then, configure the loader, add any Singer tap as an extractor to pull data from a source and run a data integration (EL) pipeline.

Capabilities

Check the README and code in the repository for more information on capabilities for this target.

Settings

Disable Collection (disable_collection)

A boolean of whether to disable Singer anonymous tracking.

Logging Level (logging_level)

(Default - INFO) The log level. Can also be set using environment variables.

Destination Path (destination_path)

(Default - ‘.’) The path to write files out to.

Compression Method (compression_method)

Compression methods have to be supported by Pyarrow, and currently the compression modes available are - snappy (recommended), zstd, brotli and gzip.

Streams In Separate Folder (streams_in_separate_folder)

(Default - False) The option to create each stream in a different folder, as these are expected to come in different schema.

File Size (file_size)

The number of rows to write per file. The default is to write to a single file.

Looking for help?

If you're having trouble getting target-parquet to work by itself or with Meltano, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page?

This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.


Edit this page on GitLab!