MySQL/MariaDB

tap-mysql (transferwise variant)🥈

MySQL/MariaDB database extractor

The tap-mysql extractor pulls data from MySQL/MariaDB that can then be sent to a destination using a loader.

Alternate Implementations

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Known limitations

Note that this extractor is incompatible with the datamill-co variants of target-postgres and target-snowflake, because they don't support stream names that include the source schema in addition to the table name: <schema>-<table>, e.g. public-accounts.

Installation and configuration

  1. Add the tap-mysql extractor to your project using 
    meltano add
     : 
    2. meltano add extractor tap-mysql
  2. Configure the tap-mysql settings using 
    meltano config
     : 
    3. meltano config tap-mysql set --interactive
  3. Test that extractor settings are valid using 
    meltano config
     : 
    4. meltano config tap-mysql test

Next steps

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for tap-mysql may have been automatically set when originally added to the Hub. Please review the capabilities when using this extractor. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this extractor.

This plugin has the following capabilities:

  • catalog
  • discover
  • state

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The tap-mysql settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using 

meltano config
with the list subcommand: 

meltano config tap-mysql list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Host (host)

  • Environment variable: TAP_MYSQL_HOST
  • Default Value: localhost

The MySQL/MariaDB hostname.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set host [value]

Port (port)

  • Environment variable: TAP_MYSQL_PORT
  • Default Value: 3306

The MySQL/MariaDB port.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set port [value]

User (user)

  • Environment variable: TAP_MYSQL_USER

The MySQL/MariaDB username.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set user [value]

Password (password)

  • Environment variable: TAP_MYSQL_PASSWORD

The MySQL/MariaDB password.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set password [value]

Database (database)

  • Environment variable: TAP_MYSQL_DATABASE

The MySQL/MariaDB database name.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set database [value]

Cursor Class (cursorclass)

  • Environment variable: TAP_MYSQL_CURSORCLASS

Cursor class used by PyMYSQL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set cursorclass [value]

Server ID (server_id)

  • Environment variable: TAP_MYSQL_SERVER_ID

Used as the slave id when this tap is connecting to the server.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set server_id [value]

Filter DBs (filter_dbs)

  • Environment variable: TAP_MYSQL_FILTER_DBS

Comma separated list of schemas to extract tables only from particular schemas and to improve data extraction performance.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set filter_dbs [value]

Use GTID (use_gtid)

  • Environment variable: TAP_MYSQL_USE_GTID
  • Default Value: false

Flag to enable log based replication using GTID.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set use_gtid [value]

Engine (engine)

  • Environment variable: TAP_MYSQL_ENGINE
  • Default Value: mysql

Indicate which flavor the server is, used for LOG_BASED with GTID. Can be either 'mysql' or 'mariadb'.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set engine [value]

SSL (ssl)

  • Environment variable: TAP_MYSQL_SSL
  • Default Value: false

Whether the MySQL/MariaDB connection should use SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl [value]

SSL CA (ssl_ca)

  • Environment variable: TAP_MYSQL_SSL_CA

CA for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_ca [value]

SSL Certificate (ssl_cert)

  • Environment variable: TAP_MYSQL_SSL_CERT

Certificate for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_cert [value]

SSL Key (ssl_key)

  • Environment variable: TAP_MYSQL_SSL_KEY

Key for self-signed SSL.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set ssl_key [value]

Internal Hostname (internal_hostname)

  • Environment variable: TAP_MYSQL_INTERNAL_HOSTNAME

Override match hostname for Google cloud.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set internal_hostname [value]

Session SQLs (session_sqls)

  • Environment variable: TAP_MYSQL_SESSION_SQLS
  • Default Value: ["SET @@session.time_zone='+0:00'","SET @@session.wait_timeout=28800","SET @@session.net_read_timeout=3600","SET @@session.innodb_lock_wait_timeout=3600"]

List of SQL commands to run when a connection made. This allows to set session variables dynamically, like timeouts.


Configure this setting directly using the following Meltano command:

meltano config tap-mysql set session_sqls [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the tap-mysql extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the 
#plugins-general
channel.

Install

meltano add extractor tap-mysql

Maintenance Status

  • Maintenance Status

Repo

https://github.com/transferwise/pipelinewise-tap-mysql
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • Wise

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

PyPI Stats

  • PyPI Downloads
  • PyPI Package Version

Keywords

  • database