Airflow
Table of Contents
The airflow
orchestrator allows for workflows to be programmatically authored, scheduled, and monitored.
Getting Started #
Prerequisites #
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration #
-
Add the
airflow
orchestrator to your project usingmeltano add
:meltano add orchestrator airflow
-
Configure the settings below using
meltano config
.
Next steps #
-
Use the meltano schedule command to create pipeline schedules in your project, to be run by Airflow.
-
If you’re running Airflow for the first time in a new environment, create an admin user:
meltano invoke airflow:create-admin # This is equivalent to `airflow users create` with some arguments in the Airflow documentation
-
Launch the Airflow UI and log in using the username/password you created:
meltano invoke airflow:ui
By default, the UI will be available at at
http://localhost:8080
. You can change this using thewebserver.web_server_port
setting documented below. -
Start Scheduler or execute Airflow commands directly using the instructions in the Meltano docs.
Settings #
Meltano centralizes the configuration of all of the plugins in your project, including Airflow’s. This means that if the Airflow documentation tells you to put something in airflow.cfg
, you can use meltano config
, meltano.yml
, or environment variables instead, and get the benefits of Meltano features like environments.
Any setting you can add to airflow.cfg
can be added to meltano.yml
, manually or using meltano config
. For example, [core] executor = SequentialExecutor
becomes meltano config airflow set core executor SequentialExecutor
on the CLI, or core.executor: SequentialExecutor
in meltano.yml
. Config sections indicated by [section]
in airflow.cfg
become nested dictionaries in meltano.yml
.
The settings for orchestrator airflow
that are known to Meltano are documented below.
To quickly find the
setting you're looking for, use the Table of Contents at
the top of the page.
DAGs Folder (core.dags_folder
)
#
-
Environment variable:
AIRFLOW__CORE__DAGS_FOLDER
, alias:AIRFLOW_CORE_DAGS_FOLDER
- Default:
$MELTANO_PROJECT_ROOT/orchestrate/dags
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set core.dags_folder <core.dags_folder>
export AIRFLOW__CORE__DAGS_FOLDER=<core.dags_folder>
Plugins Folder (core.plugins_folder
)
#
-
Environment variable:
AIRFLOW__CORE__PLUGINS_FOLDER
, alias:AIRFLOW_CORE_PLUGINS_FOLDER
- Default:
$MELTANO_PROJECT_ROOT/orchestrate/plugins
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set core.plugins_folder <core.plugins_folder>
export AIRFLOW__CORE__PLUGINS_FOLDER=<core.plugins_folder>
SQL Alchemy Connection (core.sql_alchemy_conn
)
#
-
Environment variable:
AIRFLOW__CORE__SQL_ALCHEMY_CONN
, alias:AIRFLOW_CORE_SQL_ALCHEMY_CONN
- Default:
sqlite:///$MELTANO_PROJECT_ROOT/.meltano/orchestrators/airflow/airflow.db
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set core.sql_alchemy_conn <core.sql_alchemy_conn>
export AIRFLOW__CORE__SQL_ALCHEMY_CONN=<core.sql_alchemy_conn>
Load Examples (core.load_examples
)
#
-
Environment variable:
AIRFLOW__CORE__LOAD_EXAMPLES
, alias:AIRFLOW_CORE_LOAD_EXAMPLES
- Default:
false
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set core.load_examples <core.load_examples>
export AIRFLOW__CORE__LOAD_EXAMPLES=<core.load_examples>
Pause DAGs at Creation (core.dags_are_paused_at_creation
)
#
-
Environment variable:
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION
, alias:AIRFLOW_CORE_DAGS_ARE_PAUSED_AT_CREATION
- Default:
false
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set core.dags_are_paused_at_creation <core.dags_are_paused_at_creation>
export AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=<core.dags_are_paused_at_creation>
Webserver Port (webserver.web_server_port
)
#
-
Environment variable:
AIRFLOW__WEBSERVER__WEB_SERVER_PORT
, alias:AIRFLOW_WEBSERVER_WEB_SERVER_PORT
- Default:
8080
How to use #
Manage this setting using
meltano config
or an
environment variable:
meltano config airflow set webserver.web_server_port <webserver.web_server_port>
export AIRFLOW__WEBSERVER__WEB_SERVER_PORT=<webserver.web_server_port>
Commands #
The airflow
orchestrator supports the following commands that can be used with
meltano invoke
:
create-admin
#
- Equivalent to:
airflow users create --username admin --firstname FIRST_NAME --lastname LAST_NAME --role Admin --email admin@example.org
Create an admin user.
How to use #
Run this command using
meltano invoke
:
meltano invoke airflow:create-admin [additional arguments...]
ui
#
- Equivalent to:
airflow webserver
Start the Airflow webserver.
How to use #
Run this command using
meltano invoke
:
meltano invoke airflow:ui [additional arguments...]
Looking for help? #
If you're having trouble getting the
airflow
orchestrator to work, look for an
existing issue in its repository, file a new issue,
or
join the Meltano Slack community
and ask for help in the #plugins-general
channel.
Found an issue on this page? #
This page is generated from a YAML file that you can contribute changes to. Edit it on GitHub!