The airflow orchestrator allows for workflows to be programmatically authored, scheduled, and monitored.
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the airflow orchestrator to your
project using
:meltano add
-
Configure the airflow
settings using
:meltano config
meltano add orchestrator airflow
meltano config airflow set --interactive
Next steps
The version of Airflow currently installed with Meltano (2.1.2) requires that Python be at version 3.9 or lower. Upgrading the Airflow version is tracked in this issue.
Use the meltano schedule command to create pipeline schedules in your project, to be run by Airflow.
If you're running Airflow for the first time in a new environment, create an admin user:
meltano invoke airflow:create-admin # This is equivalent to `airflow users create` with some arguments in the Airflow documentation
Launch the Airflow UI and log in using the username/password you created:
meltano invoke airflow:ui
By default, the UI will be available at at
http://localhost:8080
. You can change this using thewebserver.web_server_port
setting documented below.Start Scheduler or execute Airflow commands directly using the instructions in the Meltano docs.
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
Meltano centralizes the configuration of all of the plugins in your project, including Airflow's. This means that if the Airflow documentation tells you to put something in airflow.cfg
, you can use meltano config
, meltano.yml
, or environment variables instead, and get the benefits of Meltano features like environments.
Any setting you can add to airflow.cfg
can be added to meltano.yml
, manually or using meltano config
. For example, [core] executor = SequentialExecutor
becomes meltano config airflow set core executor SequentialExecutor
on the CLI, or core.executor: SequentialExecutor
in meltano.yml
. Config sections indicated by [section]
in airflow.cfg
become nested dictionaries in meltano.yml
.
The
airflow
settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
core.dags_are_paused_at_creation
core.dags_folder
core.load_examples
core.plugins_folder
core.sql_alchemy_conn
webserver.web_server_port
You can also list these settings using
with the meltano config
list
subcommand:
meltano config airflow list
You can
override these settings or specify additional ones
in your meltano.yml
by adding the settings
key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Pause DAGs at Creation (core.dags_are_paused_at_creation)
-
Environment variable:
AIRFLOW_CORE_DAGS_ARE_PAUSED_AT_CREATION
-
Default Value:
false
Configure this setting directly using the following Meltano command:
meltano config airflow set core dags_are_paused_at_creation [value]
DAGs Folder (core.dags_folder)
-
Environment variable:
AIRFLOW_CORE_DAGS_FOLDER
-
Default Value:
$MELTANO_PROJECT_ROOT/orchestrate/dags
Configure this setting directly using the following Meltano command:
meltano config airflow set core dags_folder [value]
Load Examples (core.load_examples)
-
Environment variable:
AIRFLOW_CORE_LOAD_EXAMPLES
-
Default Value:
false
Configure this setting directly using the following Meltano command:
meltano config airflow set core load_examples [value]
Plugins Folder (core.plugins_folder)
-
Environment variable:
AIRFLOW_CORE_PLUGINS_FOLDER
-
Default Value:
$MELTANO_PROJECT_ROOT/orchestrate/plugins
Configure this setting directly using the following Meltano command:
meltano config airflow set core plugins_folder [value]
SQL Alchemy Connection (core.sql_alchemy_conn)
-
Environment variable:
AIRFLOW_CORE_SQL_ALCHEMY_CONN
-
Default Value:
sqlite:///$MELTANO_PROJECT_ROOT/.meltano/orchestrators/airflow/airflow.db
Configure this setting directly using the following Meltano command:
meltano config airflow set core sql_alchemy_conn [value]
Webserver Port (webserver.web_server_port)
-
Environment variable:
AIRFLOW_WEBSERVER_WEB_SERVER_PORT
-
Default Value:
8080
Configure this setting directly using the following Meltano command:
meltano config airflow set webserver web_server_port [value]
Commands
The airflow orchestrator supports the following commands that can be used withmeltano invoke
:create-admin
-
Equivalent to:
users create --username admin --firstname FIRST_NAME --lastname LAST_NAME --role Admin --email admin@example.org
Create an admin user.
meltano invoke airflow:create-admin [args...]
ui
-
Equivalent to:
webserver
Start the Airflow webserver.
meltano invoke airflow:ui [args...]
Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general
channel.