tap-mysql - Meltano Hub

MySQL/MariaDB

tap-mysql (transferwise variant)🥈

MySQL/MariaDB database extractor

The tap-mysql extractor pulls data from MySQL/MariaDB that can then be sent to a destination using a loader.

Alternate Implementations

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Usage Notes

This tap is configured to use FULL_TABLE replication by default. See the setting metadata documentation for more details and instructions on using INCREMENTAL replication.

Installation and configuration

Add the tap-mysql extractor to your project using
```
meltano add
```
:

meltano add tap-mysql

Configure the tap-mysql settings using

meltano config

:

meltano config tap-mysql set --interactive

Test that extractor settings are valid using

meltano config

:

meltano config tap-mysql test

Next steps

Follow the remaining steps of the Getting Started guide:

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for tap-mysql may have been automatically set when originally added to the Hub. Please review the capabilities when using this extractor. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this extractor.

This plugin has the following capabilities:

catalog
discover
state

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

Setup

Incremental Replication

- name: tap-mysql
  variant: transferwise
  pip_url: git+https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/tap-mysql
  config:
    user: meltano
  select:
  - <my_schema>-<my_table>.*
  metadata:
    <my_schema>-<my_table>:
      replication-method: INCREMENTAL
      replication_key: key
      key_properties:
      - key

The tap-mysql settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

cursorclass
database
engine
filter_dbs
host
internal_hostname
password
port
server_id
session_sqls
ssl
ssl_ca
ssl_cert
ssl_key
use_gtid
user

You can also list these settings using

meltano config

with the list subcommand:

meltano config tap-mysql list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Cursor Class (cursorclass)

Environment variable: TAP_MYSQL_CURSORCLASS

Cursor class used by PyMYSQL.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set cursorclass [value]

Database (database)

Environment variable: TAP_MYSQL_DATABASE

The MySQL/MariaDB database name.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set database [value]

Engine (engine)

Environment variable: TAP_MYSQL_ENGINE
Default Value: mysql

Indicate which flavor the server is, used for LOG_BASED with GTID. Can be either 'mysql' or 'mariadb'.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set engine [value]

Filter DBs (filter_dbs)

Environment variable: TAP_MYSQL_FILTER_DBS

Comma separated list of schemas to extract tables only from particular schemas and to improve data extraction performance.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set filter_dbs [value]

Host (host)

Environment variable: TAP_MYSQL_HOST
Default Value: localhost

The MySQL/MariaDB hostname.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set host [value]

Internal Hostname (internal_hostname)

Environment variable: TAP_MYSQL_INTERNAL_HOSTNAME

Override match hostname for Google cloud.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set internal_hostname [value]

Password (password)

Environment variable: TAP_MYSQL_PASSWORD

The MySQL/MariaDB password.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set password [value]

Port (port)

Environment variable: TAP_MYSQL_PORT
Default Value: 3306

The MySQL/MariaDB port.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set port [value]

Server ID (server_id)

Environment variable: TAP_MYSQL_SERVER_ID

Used as the slave id when this tap is connecting to the server.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set server_id [value]

Session SQLs (session_sqls)

Environment variable: TAP_MYSQL_SESSION_SQLS
Default Value: ["SET @@session.time_zone='+0:00'","SET @@session.wait_timeout=28800","SET @@session.net_read_timeout=3600","SET @@session.innodb_lock_wait_timeout=3600"]

List of SQL commands to run when a connection made. This allows to set session variables dynamically, like timeouts.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set session_sqls [value]

SSL (ssl)

Environment variable: TAP_MYSQL_SSL
Default Value: false

Whether the MySQL/MariaDB connection should use SSL.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl [value]

SSL CA (ssl_ca)

Environment variable: TAP_MYSQL_SSL_CA

CA for self-signed SSL.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_ca [value]

SSL Certificate (ssl_cert)

Environment variable: TAP_MYSQL_SSL_CERT

Certificate for self-signed SSL.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_cert [value]

SSL Key (ssl_key)

Environment variable: TAP_MYSQL_SSL_KEY

Key for self-signed SSL.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_key [value]

Use GTID (use_gtid)

Environment variable: TAP_MYSQL_USE_GTID
Default Value: false

Flag to enable log based replication using GTID.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set use_gtid [value]

User (user)

Environment variable: TAP_MYSQL_USER

The MySQL/MariaDB username.

Configure this setting directly using the following Meltano command:

meltano config tap-mysql set user [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the tap-mysql extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Install

meltano add
              tap-mysql

Maintenance Status

Repo

https://github.com/transferwise/pipelinewise

Maintainer

Meltano Stats

Keywords