The tap-mongodb extractor pulls data from a MongoDB database.

Getting Started

Prerequisites

If you haven’t already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

Using the Command Line Interface

  1. Add the tap-mongodb extractor to your project using meltano add:

     meltano add extractor tap-mongodb
    
  2. Configure the settings below using meltano config.

Using Meltano UI

  1. Start Meltano UI using meltano ui:

     meltano ui
    
  2. Open the Extractors interface at http://localhost:5000/extractors.
  3. Click the “Add to project” button for “MongoDB”.
  4. Configure the settings below in the “Configuration” interface that opens automatically.

Next steps

Follow the remaining steps of the Getting Started guide:

  1. Select entities and attributes to extract
  2. Choose how to replicate each entity

    Supported replication methods: LOG_BASED, FULL_TABLE

  3. Add a loader to send data to a destination
  4. Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Settings

tap-mongodb requires the configuration of the following settings:

Minimal configuration

A minimal configuration of tap-mongodb in your meltano.yml project file will look like this:

plugins:
  extractors:
  - name: tap-mongodb
    variant: singer-io
    config:
      host: mongodb.example.com
      port: 27017
      user: my_user
      database: my_database

Sensitive values are most appropriately stored in the environment or your project’s .env file:

export TAP_MONGODB_PASSWORD=my_password

Host

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set host <host>

export TAP_MONGODB_HOST=<host>

Port

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set port 27018

export TAP_MONGODB_PORT=27018

User

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set user <user>

export TAP_MONGODB_USER=<user>

Password

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set password <password>

export TAP_MONGODB_PASSWORD=<password>

Database

This is the database used for authentication, not the database used for extraction. The data extracted is determined by following the selecting entities and attributes instructions.

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set database <database>

export TAP_MONGODB_DATABASE=<database>

Replica Set

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set replica_set <replica_set>

export TAP_MONGODB_REPLICA_SET=<replica_set>

SSL

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set ssl true

export TAP_MONGODB_SSL=true

Verify Mode

SSL verify mode

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set verify_mode false

export TAP_MONGODB_VERIFY_MODE=false

Include Schemas In Destination Stream Name

  • Name: include_schemas_in_destination_stream_name
  • Environment variable: TAP_MONGODB_INCLUDE_SCHEMAS_IN_DESTINATION_STREAM_NAME
  • Default: false

Forces the stream names to take the form <database_name>_<collection_name> instead of <collection_name>

How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config tap-mongodb set include_schemas_in_destination_stream_name true

export TAP_MONGODB_INCLUDE_SCHEMAS_IN_DESTINATION_STREAM_NAME=true