PostgreSQL
Table of Contents
- Alternative variants
- Getting Started
- Capabilities
- Settings
-
Host (
host
) -
Port (
port
) -
User (
user
) -
Password (
password
) -
Database Name (
dbname
) -
SSL (
ssl
) -
Filter Schemas (
filter_schemas
) -
Default Replication Method (
default_replication_method
) -
Max Run Seconds (
max_run_seconds
) -
Logical Poll Total Seconds (
logical_poll_total_seconds
) -
Break At End LSN (
break_at_end_lsn
)
-
Host (
- Troubleshooting
- Looking for help?
The tap-postgres
Meltano extractor pulls data from PostgreSQL that can then be sent to a destination using a loader.
Alternative variants #
Multiple
variants
of tap-postgres
are available.
This document describes the default transferwise
variant,
which is recommended for new users.
Alternative variants are:
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
tap-postgres
extractor to your project usingmeltano add
:meltano add extractor tap-postgres
-
Configure the settings below using
meltano config
.
Next steps #
Follow the remaining steps of the Getting Started guide:
- Select entities and attributes to extract
- Add a loader to send data to a destination
- Run a data integration (EL) pipeline
Capabilities #
Settings #
tap-postgres
requires the
configuration
of the following settings:
The settings for extractor tap-postgres
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
Host (host
)
#
-
Environment variable:
TAP_POSTGRES_HOST
- Default:
localhost
PostgreSQL host
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set host <host>
export TAP_POSTGRES_HOST=<host>
Port (port
)
#
-
Environment variable:
TAP_POSTGRES_PORT
- Default:
5432
PostgreSQL port
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set port 5432
export TAP_POSTGRES_PORT=5432
User (user
)
#
-
Environment variable:
TAP_POSTGRES_USER
PostgreSQL user
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set user <user>
export TAP_POSTGRES_USER=<user>
Password (password
)
#
-
Environment variable:
TAP_POSTGRES_PASSWORD
PostgreSQL password
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set password <password>
export TAP_POSTGRES_PASSWORD=<password>
Database Name (dbname
)
#
-
Environment variable:
TAP_POSTGRES_DBNAME
PostgreSQL database name
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set dbname <dbname>
export TAP_POSTGRES_DBNAME=<dbname>
SSL (ssl
)
#
-
Environment variable:
TAP_POSTGRES_SSL
- Default:
false
Using SSL via postgres sslmode='require'
option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set ssl true
export TAP_POSTGRES_SSL=true
Filter Schemas (filter_schemas
)
#
-
Environment variable:
TAP_POSTGRES_FILTER_SCHEMAS
Scan only the specified comma-separated schemas to improve the performance of data extraction
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set filter_schemas <filter_schemas>
export TAP_POSTGRES_FILTER_SCHEMAS=<filter_schemas>
Default Replication Method (default_replication_method
)
#
-
Environment variable:
TAP_POSTGRES_DEFAULT_REPLICATION_METHOD
-
Options:
LOG_BASED
INCREMENTAL
FULL_TABLE
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set default_replication_method LOG_BASED
export TAP_POSTGRES_DEFAULT_REPLICATION_METHOD=LOG_BASED
Max Run Seconds (max_run_seconds
)
#
-
Environment variable:
TAP_POSTGRES_MAX_RUN_SECONDS
- Default:
43200
Stop running the tap after certain number of seconds
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set max_run_seconds 43200
export TAP_POSTGRES_MAX_RUN_SECONDS=43200
Logical Poll Total Seconds (logical_poll_total_seconds
)
#
-
Environment variable:
TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS
- Default:
10800
Stop running the tap when no data received from wal after certain number of seconds
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set logical_poll_total_seconds 10800
export TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS=10800
Break At End LSN (break_at_end_lsn
)
#
-
Environment variable:
TAP_POSTGRES_BREAK_AT_END_LSN
- Default:
true
Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config tap-postgres set break_at_end_lsn false
export TAP_POSTGRES_BREAK_AT_END_LSN=false
Troubleshooting #
Error: pg_config executable not found
or libpq-fe.h: No such file or directory
#
This error message indicates that the libpq
dependency is missing.
To resolve this, refer to the “Dependencies” section above.
Looking for help? #
If you're having trouble getting the
tap-postgres
extractor to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!