PostgreSQL

The tap-postgres Meltano extractor pulls data from PostgreSQL that can then be sent to a destination using a loader.

• 
• Repository: https://github.com/transferwise/pipelinewise-tap-postgres
• 
• 
• 
• 
• 
• 
• 
• Maintainer: Wise
• Meltano Stats (across all plugin variants):
• 
• 
• 
• 

Alternative variants #

Multiple variants of tap-postgres are available. This document describes the default transferwise variant, which is recommended for new users.

Alternative variants are:

• singer-io

Getting Started #

Prerequisites #

If you haven't already, follow the initial steps of the Getting Started guide:

1. Install Meltano
2. Create your Meltano project

Installation and configuration #

1. Add the tap-postgres extractor to your project using meltano add :

   meltano add extractor tap-postgres

2. Configure the settings below using meltano config .

Next steps #

Follow the remaining steps of the Getting Started guide:

1. Select entities and attributes to extract
2. Add a loader to send data to a destination
3. Run a data integration (EL) pipeline

Capabilities #

• catalog
• discover
• state

These capabilities can also be overriden by specifying the capabilities key in your meltano.yml file.

Settings #

tap-postgres requires the configuration of the following settings:

• host
• port
• user
• password
• dbname

The settings for extractor tap-postgres that are known to Meltano are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

You can override these settings or specify additional ones in your meltano.yml by adding the settings key. Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this extractor.

Host (host) #

• Environment variable: TAP_POSTGRES_HOST
• Default: localhost

PostgreSQL host

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set host <host>

export TAP_POSTGRES_HOST=<host>

Port (port) #

• Environment variable: TAP_POSTGRES_PORT
• Default: 5432

PostgreSQL port

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set port 5432

export TAP_POSTGRES_PORT=5432

User (user) #

• Environment variable: TAP_POSTGRES_USER

PostgreSQL user

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set user <user>

export TAP_POSTGRES_USER=<user>

Password (password) #

• Environment variable: TAP_POSTGRES_PASSWORD

PostgreSQL password

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set password <password>

export TAP_POSTGRES_PASSWORD=<password>

Database Name (dbname) #

• Environment variable: TAP_POSTGRES_DBNAME

PostgreSQL database name

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set dbname <dbname>

export TAP_POSTGRES_DBNAME=<dbname>

SSL (ssl) #

• Environment variable: TAP_POSTGRES_SSL
• Default: false

Using SSL via postgres sslmode='require' option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set ssl true

export TAP_POSTGRES_SSL=true

Filter Schemas (filter_schemas) #

• Environment variable: TAP_POSTGRES_FILTER_SCHEMAS

Scan only the specified comma-separated schemas to improve the performance of data extraction

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set filter_schemas <filter_schemas>

export TAP_POSTGRES_FILTER_SCHEMAS=<filter_schemas>

Default Replication Method (default_replication_method) #

• Environment variable: TAP_POSTGRES_DEFAULT_REPLICATION_METHOD
• Options: LOG_BASED INCREMENTAL FULL_TABLE

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set default_replication_method LOG_BASED

export TAP_POSTGRES_DEFAULT_REPLICATION_METHOD=LOG_BASED

Max Run Seconds (max_run_seconds) #

• Environment variable: TAP_POSTGRES_MAX_RUN_SECONDS
• Default: 43200

Stop running the tap after certain number of seconds

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set max_run_seconds 43200

export TAP_POSTGRES_MAX_RUN_SECONDS=43200

Logical Poll Total Seconds (logical_poll_total_seconds) #

• Environment variable: TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS
• Default: 10800

Stop running the tap when no data received from wal after certain number of seconds

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set logical_poll_total_seconds 10800

export TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS=10800

Break At End LSN (break_at_end_lsn) #

• Environment variable: TAP_POSTGRES_BREAK_AT_END_LSN
• Default: true

Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started

How to use #

Manage this setting using meltano config or an environment variable:

meltano config tap-postgres set break_at_end_lsn false

export TAP_POSTGRES_BREAK_AT_END_LSN=false

Troubleshooting #

Error: pg_config executable not found or libpq-fe.h: No such file or directory #

This error message indicates that the libpq dependency is missing.

To resolve this, refer to the “Dependencies” section above.

Looking for help? #

If you're having trouble getting the tap-postgres extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page? #

This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!