The tap-postgres Meltano extractor pulls data from PostgreSQL that can then be sent to a destination using a loader.

Alternative variants #

Multiple variants of tap-postgres are available. This document describes the default transferwise variant, which is recommended for new users.

Alternative variants are:

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration #

  1. Add the tap-postgres extractor to your project using meltano add : 

    meltano add extractor tap-postgres
  2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

  1. Select entities and attributes to extract
  2. Add a loader to send data to a destination
  3. Run a data integration (EL) pipeline
If you run into any issues, learn how to get help.

Capabilities #

Settings #

tap-postgres requires the configuration of the following settings:

The settings for extractor tap-postgres that are known to Meltano are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

Host (host) #

PostgreSQL host

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set host <host>

export TAP_POSTGRES_HOST=<host>

Port (port) #

PostgreSQL port

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set port 5432

export TAP_POSTGRES_PORT=5432

User (user) #

PostgreSQL user

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set user <user>

export TAP_POSTGRES_USER=<user>

Password (password) #

PostgreSQL password

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set password <password>

export TAP_POSTGRES_PASSWORD=<password>

Database Name (dbname) #

PostgreSQL database name

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set dbname <dbname>

export TAP_POSTGRES_DBNAME=<dbname>

SSL (ssl) #

Using SSL via postgres sslmode='require' option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set ssl true

export TAP_POSTGRES_SSL=true

Filter Schemas (filter_schemas) #

Scan only the specified comma-separated schemas to improve the performance of data extraction

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set filter_schemas <filter_schemas>

export TAP_POSTGRES_FILTER_SCHEMAS=<filter_schemas>

Default Replication Method (default_replication_method) #

  • Environment variable: TAP_POSTGRES_DEFAULT_REPLICATION_METHOD
  • Options: LOG_BASED INCREMENTAL FULL_TABLE

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set default_replication_method LOG_BASED

export TAP_POSTGRES_DEFAULT_REPLICATION_METHOD=LOG_BASED

Max Run Seconds (max_run_seconds) #

Stop running the tap after certain number of seconds

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set max_run_seconds 43200

export TAP_POSTGRES_MAX_RUN_SECONDS=43200

Logical Poll Total Seconds (logical_poll_total_seconds) #

Stop running the tap when no data received from wal after certain number of seconds

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set logical_poll_total_seconds 10800

export TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS=10800

Break At End LSN (break_at_end_lsn) #

Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started

How to use #

Manage this setting using meltano config or an environment variable: 

meltano config tap-postgres set break_at_end_lsn false

export TAP_POSTGRES_BREAK_AT_END_LSN=false

Troubleshooting #

Error: pg_config executable not found or libpq-fe.h: No such file or directory #

This error message indicates that the libpq dependency is missing.

To resolve this, refer to the “Dependencies” section above.

Looking for help? #

If you're having trouble getting the tap-postgres extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!