Deploy a dbt project to Remote Execution Agents

Remote Execution Agents run tasks in your own Kubernetes infrastructure. Because Remote Execution mode doesn’t support astro dbt deploy, you must make your dbt project files available to agents through other methods.

This document covers two approaches for deploying dbt code to Remote Execution Agents, ordered by Astronomer’s recommendation.

Prerequisites

• A Remote Execution Deployment with at least one registered agent.
• A secrets backend and XCom backend configured for your agents.
• A dbt project in a Git repository.
• Cosmos installed in your agent client image for orchestrating dbt models.

Option 1: Include the dbt project as a Git submodule (recommended)

Use Git submodules to include an external dbt project repository inside your Dag repository. This approach keeps your dbt and Airflow code in separate repositories while combining them at deploy time.

This is the recommended approach because it:

• Preserves separation of ownership between dbt and Airflow teams.
• Pins the dbt project to a specific commit, giving you control over which version of dbt code runs.
• Works with GitDagBundle for automatic Dag versioning.

How it works

Your Dag repository includes the dbt project as a Git submodule. When agents fetch the Dag bundle from Git, the submodule contents are fetched alongside your Dags. Cosmos then reads the dbt project from the submodule folder to orchestrate dbt models as Airflow tasks.

Configure the Dag repository

$
git submodule add <dbt-repository-url> dbt/<project-name>

$
git submodule add https://github.com/your-org/jaffle-shop dbt/jaffle-shop

dag-repo/
├── .gitmodules
├── dags/
│   └── dbt_dag.py
└── dbt/
    └── jaffle-shop/       # Git submodule
        ├── models/
        ├── seeds/
        └── dbt_project.yml

1
from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
2
from pathlib import Path
3
from datetime import datetime
4
5
dbt_dag = DbtDag(
6
    project_config=ProjectConfig(
7
        dbt_project_path=Path("/usr/local/airflow/dags/dbt/jaffle-shop"),
8
    ),
9
    profile_config=ProfileConfig(
10
        profile_name="jaffle_shop",
11
        target_name="dev",
12
        profiles_yml_filepath=Path("/usr/local/airflow/dags/dbt/jaffle-shop/profiles.yml"),
13
    ),
14
    execution_config=ExecutionConfig(
15
        dbt_executable_path="/home/astro/dbt_venv/bin/dbt",
16
    ),
17
    schedule="@daily",
18
    start_date=datetime(2024, 1, 1),
19
    catchup=False,
20
    dag_id="dbt_jaffle_shop",
21
)

$
git add .gitmodules dbt/jaffle-shop dags/dbt_dag.py
$
git commit -m "Add dbt project as submodule with Cosmos DAG"
$
git push

Configure agents to fetch submodules

By default, GitDagBundle does not recursively fetch Git submodules. You must configure the git_conn_id connection to include submodule support and ensure the agent can authenticate to the submodule repository.

If the dbt submodule repository uses the same authentication as the Dag repository, configure the Dag bundle with the same git_conn_id. If the submodule is in a public repository, no additional authentication is required.

For private submodule repositories that require separate credentials, configure an additional Git connection in your secrets backend or values.yaml.

Update the dbt submodule

When the dbt team pushes new changes, update the submodule reference in your Dag repository:

$
git submodule update --remote dbt/jaffle-shop
$
git add dbt/jaffle-shop
$
git commit -m "Update jaffle-shop submodule to latest"
$
git push

Agents pick up the change on the next GitDagBundle refresh cycle.

For more information about working with Git submodules in an Astro context, see Use Git submodules with an Astro project.

Option 2: Include dbt code in the agent client image

Build the dbt project directly into your custom agent client image. This approach works well when dbt code changes infrequently and you want a self-contained image.

1
FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0
2
3
# Install dbt into a virtual environment
4
RUN python -m venv /home/astro/dbt_venv && \
5
    source /home/astro/dbt_venv/bin/activate && \
6
    pip install --no-cache-dir dbt-postgres && \
7
    deactivate
8
9
# Install Cosmos
10
RUN pip install astronomer-cosmos
11
12
# Copy dbt project into the image
13
COPY dbt/jaffle-shop /usr/local/airflow/dbt/jaffle-shop

$
docker build -f Dockerfile.client -t your-registry.example.com/custom-agent:1.0.0 .
$
docker push your-registry.example.com/custom-agent:1.0.0

1
workers:
2
  - name: default-worker
3
    image: your-registry.example.com/custom-agent:1.0.0
4
5
dagProcessor:
6
  image: your-registry.example.com/custom-agent:1.0.0
7
8
triggerer:
9
  image: your-registry.example.com/custom-agent:1.0.0

$
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

This approach requires rebuilding and redeploying the agent image every time dbt code changes. For teams that update dbt models frequently, consider Option 1 instead.

Compare approaches

Related documentation

• Remote Execution overview
• Configure Dag sources
• Deploy Remote Execution project
• Deploy dbt projects to Astro
• Use Git submodules with an Astro project
• Orchestrate dbt Core jobs with Airflow and Cosmos