MySQL/MariaDB

tap-mysql (transferwise variant)🥈

MySQL/MariaDB database extractor

The tap-mysql extractor pulls data from MySQL/MariaDB that can then be sent to a destination using a loader.

Alternate Implementations

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Usage Notes

This tap is configured to use FULL_TABLE replication by default. See the setting metadata documentation for more details and instructions on using INCREMENTAL replication.

Installation and configuration

  1. Add the tap-mysql extractor to your project using
    meltano add
    :
  2. meltano add extractor tap-mysql
  3. Configure the tap-mysql settings using
    meltano config
    :
  4. meltano config tap-mysql set --interactive
  5. Test that extractor settings are valid using
    meltano config
    :
  6. meltano config tap-mysql test

Next steps

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for tap-mysql may have been automatically set when originally added to the Hub. Please review the capabilities when using this extractor. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this extractor.

This plugin has the following capabilities:

  • catalog
  • discover
  • state

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

Setup

Incremental Replication

- name: tap-mysql
  variant: transferwise
  pip_url: pipelinewise-tap-mysql
  config:
    user: meltano
  select:
  - <my_schema>-<my_table>.*
  metadata:
    <my_schema>-<my_table>:
      replication-method: INCREMENTAL
      replication_key: key
      key_properties:
      - key

The tap-mysql settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using

meltano config
with the list subcommand:

meltano config tap-mysql list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Cursor Class (cursorclass)

  • Environment variable: TAP_MYSQL_CURSORCLASS

Cursor class used by PyMYSQL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set cursorclass [value]

Database (database)

  • Environment variable: TAP_MYSQL_DATABASE

The MySQL/MariaDB database name.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set database [value]

Engine (engine)

  • Environment variable: TAP_MYSQL_ENGINE
  • Default Value: mysql

Indicate which flavor the server is, used for LOG_BASED with GTID. Can be either 'mysql' or 'mariadb'.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set engine [value]

Filter DBs (filter_dbs)

  • Environment variable: TAP_MYSQL_FILTER_DBS

Comma separated list of schemas to extract tables only from particular schemas and to improve data extraction performance.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set filter_dbs [value]

Host (host)

  • Environment variable: TAP_MYSQL_HOST
  • Default Value: localhost

The MySQL/MariaDB hostname.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set host [value]

Internal Hostname (internal_hostname)

  • Environment variable: TAP_MYSQL_INTERNAL_HOSTNAME

Override match hostname for Google cloud.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set internal_hostname [value]

Password (password)

  • Environment variable: TAP_MYSQL_PASSWORD

The MySQL/MariaDB password.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set password [value]

Port (port)

  • Environment variable: TAP_MYSQL_PORT
  • Default Value: 3306

The MySQL/MariaDB port.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set port [value]

Server ID (server_id)

  • Environment variable: TAP_MYSQL_SERVER_ID

Used as the slave id when this tap is connecting to the server.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set server_id [value]

Session SQLs (session_sqls)

  • Environment variable: TAP_MYSQL_SESSION_SQLS
  • Default Value: ["SET @@session.time_zone='+0:00'","SET @@session.wait_timeout=28800","SET @@session.net_read_timeout=3600","SET @@session.innodb_lock_wait_timeout=3600"]

List of SQL commands to run when a connection made. This allows to set session variables dynamically, like timeouts.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set session_sqls [value]

SSL (ssl)

  • Environment variable: TAP_MYSQL_SSL
  • Default Value: false

Whether the MySQL/MariaDB connection should use SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl [value]

SSL CA (ssl_ca)

  • Environment variable: TAP_MYSQL_SSL_CA

CA for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_ca [value]

SSL Certificate (ssl_cert)

  • Environment variable: TAP_MYSQL_SSL_CERT

Certificate for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_cert [value]

SSL Key (ssl_key)

  • Environment variable: TAP_MYSQL_SSL_KEY

Key for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_key [value]

Use GTID (use_gtid)

  • Environment variable: TAP_MYSQL_USE_GTID
  • Default Value: false

Flag to enable log based replication using GTID.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set use_gtid [value]

User (user)

  • Environment variable: TAP_MYSQL_USER

The MySQL/MariaDB username.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set user [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the tap-mysql extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the
#plugins-general
channel.

Install

meltano add extractor tap-mysql

Maintenance Status

  • Maintenance Status

Repo

https://github.com/transferwise/pipelinewise-tap-mysql
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • Wise

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

PyPI Stats

  • PyPI Downloads
  • PyPI Package Version

Keywords

  • database