target-mssql - Meltano Hub

Microsoft SQL Server

Meltano Community

target-mssql (meltanolabs variant) 🥇

Open source connector supported by the Meltano Community, a growing community of over 5500+ data engineers.

Open source Join Slack community

The target-mssql loader sends data into Microsoft SQL Server after it was pulled from a source using an extractor

Alternate Implementations

Meltano (default)🥇
Storebrand
🥇

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the target-mssql loader to your project using
```
meltano add
```
:

meltano add target-mssql

Configure the target-mssql settings using

meltano config

:

meltano config target-mssql set --interactive

Next steps

Follow the remaining steps of the Getting Started guide:

Run a data integration (EL) pipeline

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for target-mssql may have been automatically set when originally added to the Hub. Please review the capabilities when using this loader. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this loader.

This plugin has the following capabilities:

about
activate-version
batch
hard-delete
schema-flattening
stream-maps
structured-logging
validate-records

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The target-mssql settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

Expand To Show SDK Settings

add_record_metadata
batch_config.encoding.compression
batch_config.encoding.format
batch_config.storage.prefix
batch_config.storage.root
default_target_schema
faker_config.locale
faker_config.seed
flattening_enabled
flattening_max_depth
hard_delete
load_method
stream_map_config
stream_maps
validate_records

You can also list these settings using

meltano config

with the list subcommand:

meltano config target-mssql list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Azure Blob Storage Account Name (azure_blob_storage.account_name)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_ACCOUNT_NAME

Azure Storage account name

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage account_name [value]

Azure Blob Storage Container (azure_blob_storage.container)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_CONTAINER

Blob container used as the staging area

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage container [value]

Azure Blob Storage Credential (azure_blob_storage.credential)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_CREDENTIAL

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage credential [value]

Azure Blob Storage Data Source (azure_blob_storage.data_source)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_DATA_SOURCE

[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage data_source [value]

Azure Blob Storage Path Prefix (azure_blob_storage.path_prefix)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_PATH_PREFIX
Default Value: target-mssql

Path prefix (virtual directory) inside the container

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage path_prefix [value]

Azure Blob Storage Sas Token (azure_blob_storage.sas_token)

Environment variable: TARGET_MSSQL_AZURE_BLOB_STORAGE_SAS_TOKEN

Shared Access Signature token for the storage account (without a leading '?'). Must grant read/write/delete on blobs.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set azure_blob_storage sas_token [value]

Batch Size Rows (batch_size_rows)

Environment variable: TARGET_MSSQL_BATCH_SIZE_ROWS

Maximum number of rows in each batch.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set batch_size_rows [value]

Database (database)

Environment variable: TARGET_MSSQL_DATABASE

SQL Server database

Configure this setting directly using the following Meltano command:

meltano config target-mssql set database [value]

Driver (driver)

Environment variable: TARGET_MSSQL_DRIVER
Default Value: pymssql

The driver to use for the database connection (pymssql or pyodbc)

Configure this setting directly using the following Meltano command:

meltano config target-mssql set driver [value]

Max Key Length (flattening_max_key_length)

Environment variable: TARGET_MSSQL_FLATTENING_MAX_KEY_LENGTH

The maximum length of a flattened key.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set flattening_max_key_length [value]

Flattening Separator (flattening_separator)

Environment variable: TARGET_MSSQL_FLATTENING_SEPARATOR

The separator to use when flattening keys.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set flattening_separator [value]

Host (host)

Environment variable: TARGET_MSSQL_HOST

SQL Server host

Configure this setting directly using the following Meltano command:

meltano config target-mssql set host [value]

Odbc Driver (odbc_driver)

Environment variable: TARGET_MSSQL_ODBC_DRIVER

The ODBC driver to use when driver=pyodbc (e.g. 'ODBC Driver 18 for SQL Server')

Configure this setting directly using the following Meltano command:

meltano config target-mssql set odbc_driver [value]

Password (password)

Environment variable: TARGET_MSSQL_PASSWORD

SQL Server password

Configure this setting directly using the following Meltano command:

meltano config target-mssql set password [value]

Port (port)

Environment variable: TARGET_MSSQL_PORT
Default Value: 1433

SQL Server port

Configure this setting directly using the following Meltano command:

meltano config target-mssql set port [value]

Prefer Float Over Numeric (prefer_float_over_numeric)

Environment variable: TARGET_MSSQL_PREFER_FLOAT_OVER_NUMERIC
Default Value: false

Use float data type for numbers (otherwise number type is used)

Configure this setting directly using the following Meltano command:

meltano config target-mssql set prefer_float_over_numeric [value]

Process `ACTIVATE_VERSION` messages (process_activate_version_messages)

Environment variable: TARGET_MSSQL_PROCESS_ACTIVATE_VERSION_MESSAGES
Default Value: true

Whether to process ACTIVATE_VERSION messages.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set process_activate_version_messages [value]

SQLAlchemy URL (sqlalchemy_url)

Environment variable: TARGET_MSSQL_SQLALCHEMY_URL

SQLAlchemy connection string

Configure this setting directly using the following Meltano command:

meltano config target-mssql set sqlalchemy_url [value]

Table Prefix (table_prefix)

Environment variable: TARGET_MSSQL_TABLE_PREFIX

Prefix to add to table name

Configure this setting directly using the following Meltano command:

meltano config target-mssql set table_prefix [value]

Trust Server Certificate (trust_server_certificate)

Environment variable: TARGET_MSSQL_TRUST_SERVER_CERTIFICATE
Default Value: false

Trust the server certificate without validation (useful for self-signed certs)

Configure this setting directly using the following Meltano command:

meltano config target-mssql set trust_server_certificate [value]

Username (username)

Environment variable: TARGET_MSSQL_USERNAME

SQL Server username

Configure this setting directly using the following Meltano command:

meltano config target-mssql set username [value]

Expand To Show SDK Settings

Add Record Metadata (add_record_metadata)

Environment variable: TARGET_MSSQL_ADD_RECORD_METADATA

Whether to add metadata fields to records.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set add_record_metadata [value]

Batch Compression Format (batch_config.encoding.compression)

Environment variable: TARGET_MSSQL_BATCH_CONFIG_ENCODING_COMPRESSION

Compression format to use for batch files.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set batch_config encoding.compression [value]

Batch Encoding Format (batch_config.encoding.format)

Environment variable: TARGET_MSSQL_BATCH_CONFIG_ENCODING_FORMAT

Format to use for batch files.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set batch_config encoding.format [value]

Batch Storage Prefix (batch_config.storage.prefix)

Environment variable: TARGET_MSSQL_BATCH_CONFIG_STORAGE_PREFIX

Prefix to use when writing batch files.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set batch_config storage.prefix [value]

Batch Storage Root (batch_config.storage.root)

Environment variable: TARGET_MSSQL_BATCH_CONFIG_STORAGE_ROOT
Default Value: file://

Root path to use when writing batch files.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set batch_config storage.root [value]

Default Target Schema (default_target_schema)

Environment variable: TARGET_MSSQL_DEFAULT_TARGET_SCHEMA

Default target schema to write to

Configure this setting directly using the following Meltano command:

meltano config target-mssql set default_target_schema [value]

Faker Locale (faker_config.locale)

Environment variable: TARGET_MSSQL_FAKER_CONFIG_LOCALE

One or more LCID locale strings to produce localized output for: https://faker.readthedocs.io/en/master/#localization

Configure this setting directly using the following Meltano command:

meltano config target-mssql set faker_config locale [value]

Faker Seed (faker_config.seed)

Environment variable: TARGET_MSSQL_FAKER_CONFIG_SEED

Value to seed the Faker generator for deterministic output: https://faker.readthedocs.io/en/master/#seeding-the-generator

Configure this setting directly using the following Meltano command:

meltano config target-mssql set faker_config seed [value]

Enable Schema Flattening (flattening_enabled)

Environment variable: TARGET_MSSQL_FLATTENING_ENABLED

'True' to enable schema flattening and automatically expand nested properties.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set flattening_enabled [value]

Max Flattening Depth (flattening_max_depth)

Environment variable: TARGET_MSSQL_FLATTENING_MAX_DEPTH

The max depth to flatten schemas.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set flattening_max_depth [value]

Hard Delete (hard_delete)

Environment variable: TARGET_MSSQL_HARD_DELETE
Default Value: false

Hard delete records.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set hard_delete [value]

Load Method (load_method)

Environment variable: TARGET_MSSQL_LOAD_METHOD
Default Value: append-only

The method to use when loading data into the destination. append-only will always write all input records whether that records already exists or not. upsert will update existing records and insert new records. overwrite will delete all existing records and insert all input records.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set load_method [value]

User Stream Map Configuration (stream_map_config)

Environment variable: TARGET_MSSQL_STREAM_MAP_CONFIG

User-defined config values to be used within map expressions.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set stream_map_config [value]

Stream Maps (stream_maps)

Environment variable: TARGET_MSSQL_STREAM_MAPS

Config object for stream maps capability. For more information check out Stream Maps.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set stream_maps [value]

Validate Records (validate_records)

Environment variable: TARGET_MSSQL_VALIDATE_RECORDS
Default Value: true

Whether to validate the schema of the incoming streams.

Configure this setting directly using the following Meltano command:

meltano config target-mssql set validate_records [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-mssql loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Meltano Cloud

For ambitious analytics engineering teams that want an efficient, fully managed solution with personalised expert support, explore Meltano Cloud

No maintenance overhead
Rigorously tested
Expert support from Meltano

Learn more Book a demo

Install

meltano add
                target-mssql

Maintenance Status

Repo

https://github.com/MeltanoLabs/target-mssql

Maintainer

Meltano Stats

Keywords