airflow - Meltano Hub

Airflow

airflow (apache variant)

The airflow orchestrator allows for workflows to be programmatically authored, scheduled, and monitored.

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the airflow orchestrator to your project using
```
meltano add
```
:

meltano add --plugin-type orchestrator airflow

Configure the airflow settings using

meltano config

meltano config airflow set --interactive

Next steps

The version of Airflow currently installed with Meltano (2.1.2) requires that Python be at version 3.9 or lower. Upgrading the Airflow version is tracked in this issue.

Use the meltano schedule command to create pipeline schedules in your project, to be run by Airflow.

If you're running Airflow for the first time in a new environment, create an admin user:

meltano invoke airflow:create-admin
# This is equivalent to `airflow users create` with some arguments in the Airflow documentation

Launch the Airflow UI and log in using the username/password you created:
```
meltano invoke airflow:ui
```
By default, the UI will be available at at http://localhost:8080. You can change this using the webserver.web_server_port setting documented below.
Start Scheduler or execute Airflow commands directly using the instructions in the Meltano docs.

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

Meltano centralizes the configuration of all of the plugins in your project, including Airflow's. This means that if the Airflow documentation tells you to put something in airflow.cfg, you can use meltano config, meltano.yml, or environment variables instead, and get the benefits of Meltano features like environments.

Any setting you can add to airflow.cfg can be added to meltano.yml, manually or using meltano config. For example, [core] executor = SequentialExecutor becomes meltano config airflow set core executor SequentialExecutor on the CLI, or core.executor: SequentialExecutor in meltano.yml. Config sections indicated by [section] in airflow.cfg become nested dictionaries in meltano.yml.

The airflow settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

core.dags_are_paused_at_creation
core.dags_folder
core.load_examples
core.plugins_folder
core.sql_alchemy_conn
webserver.web_server_port

You can also list these settings using

meltano config

with the list subcommand:

meltano config airflow list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Pause DAGs at Creation (core.dags_are_paused_at_creation)

Environment variable: AIRFLOW_CORE_DAGS_ARE_PAUSED_AT_CREATION
Default Value: false

[No description provided.]