Postgres is a free and open source relational database system. Integrating Postgres with Airflow allows you to interact with your Postgres database, run queries, ans load or export data from an Airflow DAG.
This guide provides the basic setup for creating a Postgres connection.
A connection from Airflow to Postgres requires the following information:
public)The method to retrieve these values will vary based which cloud provider you use to host Microsoft SQL Server. Refer to the following documents to for more information about retrieving these values:
For example, if you’re running Postgres in a Relational Data Store (RDS) in AWS, complete the following steps to retrieve these values:
public will be used.Astro users can also create connections using the Astro Environment Manager, which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See Create Airflow connections in the Astro UI.
Open your Astro project and add the following line to your requirements.txt file:
This will install the Postgres provider package, which makes the Postgres connection type available in Airflow.
Run astro dev restart to restart your local Airflow environment and apply your changes in requirements.txt.
In the Airflow UI for your local Airflow environment, go to Admin > Connections. Click + to add a new connection, then choose Postgres as the connection type.
Fill out the following connection fields using the information you retrieved from Get connection details:
Click Test. After the connection test succeeds, click Save.
Airflow uses the psycopg2 python library to connect to Postgres through the PostgresHook. You can also directly use the PostgresHook to create your own custom operators.