PostgreSQL (datamill-co
variant)
Table of Contents
- Alternative variants
- Standalone usage
- Usage with Meltano
- Capabilities
- Settings
-
postgres_host
-
postgres_port
-
postgres_database
-
postgres_username
-
postgres_password
-
postgres_schema
-
postgres_sslmode
-
postgres_sslcert
-
postgres_sslkey
-
postgres_sslrootcert
-
postgres_sslcrl
-
invalid_records_detect
-
invalid_records_threshold
-
disable_collection
-
logging_level
-
persist_empty_tables
-
max_batch_rows
-
max_buffer_size
-
batch_detection_threshold
-
state_support
-
add_upsert_indexes
-
before_run_sql
-
after_run_sql
-
- Looking for help?
The
target-postgres
Singer target
sends data into
PostgreSQL
after it was pulled from a source using a
Singer tap.
Alternative variants #
Multiple
variants
of target-postgres
are available. This document describes the
datamill-co
variant.
Alternative variants are:
-
transferwise
(default) -
meltano
Standalone usage #
Install the package using pip:
pip install singer-target-postgres
For additional instructions, refer to the README in the repository.
Usage with Meltano #
Install Meltano, create your Meltano project, and add the target to your project as a loader:
meltano add loader target-postgres --variant datamill-co
For additional instructions, refer to the
Meltano-specific documentation for target-postgres
.
Capabilities #
Settings #
postgres_host
#
- Default:
localhost
postgres_port
#
- Default:
5432
postgres_database
#
postgres_username
#
postgres_password
#
postgres_schema
#
postgres_sslmode
#
- Default:
prefer
Refer to the libpq docs for more information about SSL: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
postgres_sslcert
#
- Default:
~/.postgresql/postgresql.crt
Only used if a SSL request w/ a client certificate is being made
postgres_sslkey
#
- Default:
~/.postgresql/postgresql.key
Only used if a SSL request w/ a client certificate is being made
postgres_sslrootcert
#
- Default:
~/.postgresql/root.crt
Used for authentication of a server SSL certificate
postgres_sslcrl
#
- Default:
~/.postgresql/root.crl
Used for authentication of a server SSL certificate
invalid_records_detect
#
- Default:
true
Include false
in your config to disable target-postgres
from crashing on invalid records
invalid_records_threshold
#
- Default:
0
Include a positive value n
in your config to allow for target-postgres
to encounter at most n
invalid records per stream before giving up.
disable_collection
#
- Default:
false
Include true
in your config to disable Singer Usage Logging: https://github.com/datamill-co/target-postgres#usage-logging
logging_level
#
- Default:
INFO
The level for logging. Set to DEBUG
to get things like queries executed, timing of those queries, etc.
persist_empty_tables
#
- Default:
false
Whether the Target should create tables which have no records present in Remote.
max_batch_rows
#
- Default:
200000
The maximum number of rows to buffer in memory before writing to the destination table in Postgres
max_buffer_size
#
- Default:
104857600
The maximum number of bytes to buffer in memory before writing to the destination table in Postgres. Default: 100MB in bytes
batch_detection_threshold
#
How often, in rows received, to count the buffered rows and bytes to check if a flush is necessary. There’s a slight performance penalty to checking the buffered records count or bytesize, so this controls how often this is polled in order to mitigate the penalty. This value is usually not necessary to set as the default is dynamically adjusted to check reasonably often.
state_support
#
- Default:
true
Whether the Target should emit STATE
messages to stdout for further consumption. In this mode, which is on by default, STATE messages are buffered in memory until all the records that occurred before them are flushed according to the batch flushing schedule the target is configured with.
add_upsert_indexes
#
- Default:
true
Whether the Target should create column indexes on the important columns used during data loading. These indexes will make data loading slightly slower but the deduplication phase much faster. Defaults to on for better baseline performance.
before_run_sql
#
Raw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup like SET ROLE
or other connection state that is important.
after_run_sql
#
Raw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup like SET ROLE
or other connection state that is important.
Looking for help? #
If you're having trouble getting
target-postgres
to work by itself or with
Meltano, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to! It is also validated against a JSON Schema used for taps and targets.