Scaling Airflow at Wix for Analytics and AI with Ethan Shalev

Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.

In this episode, we're joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.

Key Takeaways:

(00:00) Introduction.

(02:13) Wix structures data engineering across multiple product-focused organizations.

(03:40) Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.

(04:31) Migration creates an opportunity to remove long-standing legacy Airflow code.

(05:32) Internal playbooks and Cursor rules standardize and speed up DAG migrations.

(07:39) Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.

(09:16) Deferrable operators reduce scheduler congestion in large Airflow environments.

(12:54) AI-generated code still requires review and strong testing practices.

(14:52) Moving to managed Airflow reduces operational burden on internal platform teams.

(15:57) Improving multi-tenancy and UI personalization remains a key Airflow need.

Resources Mentioned:

Ethan Shalev: https://www.linkedin.com/in/eshalev/

Wix: https://www.linkedin.com/company/wix-com/ | https://www.wix.com/

Apache Airflow: https://airflow.apache.org/

Astronomer Astro Platform: https://www.astronomer.io/

Trino: https://trino.io/

Apache Iceberg: https://iceberg.apache.org/

Cursor: https://cursor.sh/

Airflow Summit: https://airflowsummit.org/

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow