The tap-postgres Meltano extractor pulls data from the PostgreSQL that can then be sent to a destination using a loader.

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

Using the Command Line Interface

  1. Add the tap-postgres extractor to your project using meltano add :

    meltano add extractor tap-postgres --variant transferwise
  2. Configure the settings below using meltano config .

Using Meltano UI

  1. Start Meltano UI using meltano ui:

    meltano ui
  2. Open the Extractors interface at http://localhost:5000/extractors.
  3. Click the “Add to project” button for “PostgreSQL”.
  4. Choose "Add variant 'transferwise'".
  5. Configure the settings below in the “Configuration” interface that opens automatically.

Next steps

Follow the remaining steps of the Getting Started guide:

  1. Select entities and attributes to extract
  2. Add a loader to send data to a destination
  3. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Settings

tap-postgres requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents at the top of the page.

host

PostgreSQL host

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set host <host>

export TAP_POSTGRES_HOST=<host>

port

PostgreSQL port

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set port 5432

export TAP_POSTGRES_PORT=5432

user

PostgreSQL user

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set user <user>

export TAP_POSTGRES_USER=<user>

password

PostgreSQL password

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set password <password>

export TAP_POSTGRES_PASSWORD=<password>

dbname

PostgreSQL database name

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set dbname <dbname>

export TAP_POSTGRES_DBNAME=<dbname>

ssl

Using SSL via postgres sslmode='require' option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set ssl true

export TAP_POSTGRES_SSL=true

filter_schemas

Scan only the specified comma-separated schemas to improve the performance of data extraction

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set filter_schemas <filter_schemas>

export TAP_POSTGRES_FILTER_SCHEMAS=<filter_schemas>

default_replication_method

  • Environment variable: TAP_POSTGRES_DEFAULT_REPLICATION_METHOD
  • Options: LOG_BASED INCREMENTAL FULL_TABLE

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set default_replication_method LOG_BASED

export TAP_POSTGRES_DEFAULT_REPLICATION_METHOD=LOG_BASED

max_run_seconds

Stop running the tap after certain number of seconds

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set max_run_seconds 43200

export TAP_POSTGRES_MAX_RUN_SECONDS=43200

logical_poll_total_seconds

Stop running the tap when no data received from wal after certain number of seconds

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set logical_poll_total_seconds 10800

export TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS=10800

break_at_end_lsn

Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-postgres set break_at_end_lsn false

export TAP_POSTGRES_BREAK_AT_END_LSN=false

Looking for help?

If you're having trouble getting the tap-postgres extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the #plugins-general channel.

Found an issue on this page?

This page is generated from a YAML file that you can contribute changes to!