Orchestrating the Future of Retail

Retailers unlock margin, loyalty, and resilience by orchestrating clean, timely data across their enterprise. To do this, they are converging on five strategic IT investment priorities spanning AI and advanced analytics through data platform modernization to back-office improvements. All are underpinned by rock-solid security defences.

In this guide we profile each investment. We'll identify the objectives, challenges, and required capabilities behind each before exploring how the Astro unified orchestration platform built on top of Apache Airflow® can best support them.

Why Airflow and Astro?

• Apache Airflow has grown to become the industry’s most widely used system for orchestrating data workflows, as well as being one of the world’s most active open source projects.
• Astro from Astronomer is the first and only multi-and hybrid-cloud unified data operations platform. Astro is the unified, enterprise-grade orchestration platform to build, run, and observe mission-critical data and AI pipelines.

Recent events underscore the urgent need: in 2025 alone, high-profile European retailers such as Marks & Spencer (M&S), The Co-operative Group (Co-op), Adidas, and Harrods suffered coordinated cyberattacks that disrupted online sales, shut down payment systems, and compromised customer data.

These incidents highlight that legacy, fragmented IT stacks are no longer just inefficient, they are business risks that can bring trading to a halt and damage trust. Retailers must modernize to improve security, reduce attack surfaces, and build operational resilience before attackers strike.

Improved security is also required for enterprises to grow. Retailers cannot scale AI, deliver seamless omnichannel journeys, or optimize supply chain execution on brittle, outdated infrastructure. A modern, secure, cloud-ready architecture is the foundation that enables both defense and innovation.

Capabilities for Modern Retail Architecture

To support modern retail operations with resilience and security, organizations need orchestration that unifies environments, enables secure execution, supports hybrid deployments, enforces access controls, and scales reliably even under attack or peak operational loads.

Remote Execution: Enabling Secure, Cloud-Native Data Orchestration

For retailers, managed cloud platforms often raise concerns about data security, customer privacy, and regulatory compliance. Critical workflows process sensitive customer information, pricing data, payment details, inventory positions, and supply chain events. Moving that data into a vendor’s infrastructure can violate PCI-DSS, GDPR, or internal security policies if the right controls aren’t in place.

Astro solves this with Remote Execution, the Airflow 3 architecture that separates orchestration from execution. Retailers get a fully managed Airflow control plane maintained, upgraded, and secured by Astronomer while all workflow execution stays inside their own cloud or on-premises environment and within their compliance boundary.

Figure 1: Stepping through Remote Execution’s architecture and traffic flow

Remote Execution uses a three-plane architecture:

• The control plane manages users and metadata but never sees your data.
• The orchestration plane schedules workflows in a single-tenant environment.
• The execution plane (fully yours) runs the tasks using your infra, secrets, and permissions.

Only outbound encrypted connections are used. There is no need for inbound firewall exceptions. Astro’s exclusive remote execution agents authenticate with your IAM role and policy and run jobs under customer-managed identities. This aligns with zero-trust principles and removes the need to trade security for operational efficiency.

Bottom line: Astro gives you the benefits of a managed orchestration platform, including agility, performance, reliability, and reduced ops burden, without customer data ever leaving your secured and approved environment. That’s what makes it deployable for sensitive workloads and data where conventional SaaS models fail.

You can learn more by downloading our whitepaper: Remote Execution: Powering Hybrid Orchestration Without Compromise.

Astro Private Cloud

For organizations that cannot adopt any managed services, Astro Private Cloud delivers enterprise-grade Airflow-as-a-Service entirely within your own environment. It runs exclusively on customer-managed infrastructure—across private cloud, on-premises, or fully air-gapped deployments—providing complete ownership over data, network boundaries, and security controls.

Astro Private Cloud consolidates fragmented Airflow usage into a centrally governed platform with isolated, multi-tenant deployments. A unified control plane enables teams to standardize orchestration, enforce security and governance policies, and manage multiple Airflow environments while individual teams operate independently within dedicated namespaces.

By combining centralized governance with full infrastructure control, Astro Private Cloud reduces operational overhead, strengthens security and compliance, and enables organizations to reliably scale orchestration across the enterprise.

Note: Astro Private Cloud does not include features specific to the hosted Astro service, such as the Astro IDE and Astro Observe.

Astro in Action

Data teams in retailers adopt Astro to eliminate the legacy schedulers that often cripple the ability to ship new data products and workflows. Moving from legacy orchestration systems such as AutoSys, Control-M, Informatica or Apache Oozie to Astro unlocks strategic and operational gains:

• Cut costs by up to 75%. Organizations moving to Astro typically realize major savings through reduced infrastructure, licensing, and operational overhead, freeing budget for innovation.
• Unblock agility and scale with cloud-native orchestration. As a modern orchestration platform, Astro gives teams the flexibility, resilience, and scalability needed to support fast-moving data and AI initiatives without the constraints of legacy tooling and manual overhead.
• Attract and retain top engineering talent. Airflow embodies code-first and open source philosophies. By using Airflow, data teams recruit top talent more easily and onboard faster while avoiding lock-in to niche or proprietary technology.

Commonly migrated workloads include ETL jobs, data warehouse loads and refreshes, report generation and distribution, batch file transfers (FTP/SFTP jobs), data validations and quality checks, time-triggered or event-triggered job dependencies across systems, and mainframe and SAP job coordination.

No matter what workload or legacy orchestration tool your organization is using, Astronomer’s Professional Services team can help. The company’s experts can build an operational framework to smoothly and safely migrate your workloads to Astro.

• A major European grocery chain runs a highly automated, on-premises data platform supporting 2,700 stores and thousands of analysts, but legacy orchestration tools could not meet zero-tolerance SLAs for mission-critical finance and forecasting workloads, and post-cyber attack security expectations made community-supported, manually operated Airflow untenable. Adopting Astro Private Cloud delivered SLA-backed enterprise support, hardened security aligned to CISO requirements, and standardized operations that eliminated manual overhead. The result was a 50% reduction in FTE time, resilient on-premises Airflow trusted for critical workloads, and 3,000 business analysts supported at scale.
• One of the world's largest retailers needed an orchestration platform capable of scaling to thousands of users and 1,200+ deployments, but legacy systems and self-managed Airflow created performance bottlenecks, operational drag, and slow team onboarding. Standardizing on Astronomer centralized enterprise-wide orchestration and delivered streamlined onboarding, unified workload management, and strengthened governance across the business. Today the retailer runs 30 million+ tasks per month on Astronomer, with an operational model built for resilience and long-term scale.

Embedding AI and automation across both operational workflows and customer-facing experiences aims to cut manual effort, boost accuracy, and scale personalization. Based on recent research, retail and consumer goods companies allocated an average of 3.3% of their total revenue to AI over the course of 2025.

When AI initiatives are executed well, store associates spend less time on repetitive tasks, customers enjoy smoother and more relevant experiences, and the business achieves faster time to value with fewer errors. Key use cases targeted by retailers include:

• Personalized recommendations that adapt offers and content to each shopper’s behavior in real time.
• Demand forecasting that predicts store- and SKU-level demand to optimize replenishment.
• Dynamic pricing that adjusts margins and sell-through based on demand, inventory, and competition.
• Store automation that uses computer vision and sensors to detect stockouts, shrinkage, and queue spikes.
• Service automation with chatbots and agents handling routine customer inquiries and order issues.

The Challenge: Fragmented Data and Legacy Systems

Retailers face disconnected data, inconsistent inventory accuracy, and outdated systems that can’t support real-time AI. Data quality issues slow model performance, legacy platforms block automation, and siloed channels prevent unified personalization. Compliance requirements add friction, and limited internal AI skills make scaling difficult.

The cost of inaction is clear: missed demand, margin erosion, longer cycle times, and customer experiences that lag behind digital native, AI-driven competitors.

Capabilities for AI-First Retail Operations and Customer Journeys

Operationalizing AI in retail requires more than accurate models. It demands orchestration that unifies channel, customer, and supply chain data. It enforces security and governance. Finally, it automates model lifecycles spanning training, inference, evaluation, and continuous improvement.

Airflow and Astro in Action

Airflow is already used by some of the most demanding AI companies and agentic workloads on the planet:

• OpenAI has standardized on Airflow across its business with over 7,000 pipelines spanning research, operations, and finance, all while providing a foundation for 10x growth. Read more.
• GitHub relies on Airflow to process billions of developer events per day, orchestrating the feedback loops used to continuously improve Copilot. Read more.

The retail industry is following suit:

• One of South America's fastest-growing e-commerce companies hit a breaking point as self-managed Airflow environments multiplied across teams, creating orchestration sprawl, inconsistent practices, and limited visibility into compute costs that slowed AI/ML delivery across restaurant operations and marketplace logistics. Moving to Astro unified orchestration under a single managed platform with centralized cost control and enterprise-grade reliability. The result was a 3x increase in task capacity, 8x growth in deployments, and 26+ teams operating with shared orchestration, governance, and cost transparency.
• DoorDash operates one of the largest Airflow deployments in the industry, serving data and ML engineers across thousands of pipelines. As scale grew, a single monolithic instance created scheduler pressure, slow Dag parsing, upgrade risk, and reliability issues that threatened the ML workflows powering their marketplace. The team solved this by building a custom unifying layer across tiered Airflow instances, enabling horizontal scaling, isolated upgrades, and consistent cross-instance ML pipeline management. Read more in the Airflow in Action blog post.

“Astronomer has been a great partner in helping us migrate our legacy schedulers to a more modern platform. The team has been very responsive and knowledgeable, and we're confident we can complete this project successfully together."
Enterprise Architect, premier luxury fashion retailer

Retailers are seeking to unlock a unified, real-time analytics foundation that delivers a complete view of customers, operations, and financial performance. As a result, teams access accurate, timely insights across channels, enabling better forecasting, faster decisions, and tighter margin control. To illustrate the priority of analytics to the industry, 70% of grocery retailers said they’re investing in data analytics platforms.

With these investments, analytics scale from descriptive dashboards to predictive and prescriptive intelligence, supported by governed data, automated pipelines, and self-service access for business users. Key use cases include

• Real-time performance dashboards that unify POS, e-commerce, and supply chain metrics for operational decision-making.
• Customer segmentation and LTV analytics that identify high-value shoppers and tailor engagement strategies.
• Merchandising analytics that reveal SKU performance, sell-through trends, and promotion effectiveness.
• Supply chain visibility analytics that identify bottlenecks and optimize inventory flows.
• Omnichannel journey analytics that track behavior and attribution across digital and in-store touchpoints.

Capabilities for Modern Retail Analytics

As noted with initiative 1, most retailers face siloed systems. They also struggle with inconsistent data definitions and slow batch reporting that can’t support real-time decisions. Data quality issues undermine trust in dashboards, and legacy BI tools lack predictive capabilities. As competition tightens and demand patterns shift faster, retailers can no longer operate with delayed or partial visibility.

The cost of inaction is lost margin, poor localization, and misaligned merchandising decisions

Airflow in Action

Kleinanzeigen, Germany’s largest online classifieds marketplace, needed to rebuild its data platform from the ground up while keeping live operations running for 30 million monthly users and 55 million listings. In under six months the team migrated their entire warehouse to Databricks, refactored 100+ pipelines, and standardized 600+ production tables using dbt orchestrated with Cosmos on Astro across over 2 PB of production data. The result was consistent deployment patterns, enforceable best practices, and a platform that shifted the team from infrastructure firefighting to delivering high-value marketplace insights. Read the case study to learn more.

Retailers are seeking to deliver consistent, personalized, real-time experiences across every touchpoint. Online and in-store interactions share a single view of customers, inventory, pricing, and promotions. AI-driven personalization, responsive fulfillment options, and unified omnichannel journeys drive loyalty, higher conversion, and stronger lifetime value.

79% of retailers expect to increase investments in CX and personalization with key use cases including:

• Real-time personalization across web, mobile, store kiosks, and email.
• Unified customer profiles built from loyalty, browsing, purchase, and service data.
• Omnichannel services such as BOPIS, curbside, ship-from-store, and cart continuity.
• Clienteling and associate tools backed by complete shopper histories.
• Personalized service automation using LLM-based agents and summarization.

Capabilities for Omnichannel, Data-Driven CX

Retailers are prevented from unlocking these use cases because CX systems still operate in silos: POS, e-commerce, loyalty, and service platforms don’t share data in real time. Legacy POS and EPOS systems capture limited or inaccurate data, creating inconsistencies that flow downstream into personalization engines and customer profiles. As a result, recommendations run on stale or incorrect inputs, omnichannel features break, and store associates lack context. Poor data quality and brittle integrations slow personalization and frustrate customers at every touchpoint.

Astro in Action

• Instacart partners with 1,500+ retailers across 85,000 stores to deliver the seamless shopping experiences millions of customers depend on, all underpinned by reliable, high-quality data. Fragmented legacy Airflow clusters with limited visibility and scalability issues threatened that experience. Consolidating onto a single centrally managed Airflow cluster enabled the team to orchestrate 2,200 pipelines and 16 million tasks per month at a 99.5% completion rate, giving retail partners the operational insights and analytics that drive better customer outcomes. Learn more.
• A global athletic footwear and apparel retailer suffered a major outage in its self-managed Airflow environment, taking essential data operations offline for nearly a week and exposing fundamental gaps in reliability and operational ownership across the pipelines feeding its business and customer systems. Astronomer’s Professional Services team executed a rapid migration of 350+ pipelines to Astro, restoring production workflows and eliminating single-point-of-failure risk. The result was a stable, managed Airflow foundation built for future automation and growth.

Figure 2: With the Astro platform, data teams work with a unified data stack to build, run, and observe all of their critical data pipelines across AI, app, and analytics workflows.

Retailers aim to run a responsive, data-driven supply chain and workforce operation with real-time visibility, accurate forecasts, automated replenishment, and optimized labor allocation. Inventory moves efficiently across stores, distribution centers, suppliers, and in-transit nodes, while fulfillment executes reliably and workforce schedules adapt to demand with minimal manual effort. 65% of retailers are reported to be investing in real-time inventory management to meet customer demand and reduce stockouts.

Capabilities for Intelligent Supply Chain and Workforce Operations

Supply chain data is fragmented across ERPs, WMS, TMS, supplier feeds, and store systems, creating delays and inaccuracies. Workforce planning relies on manual schedules and inconsistent data. Getting this right is critical as labour shortages remain one of the top challenges for retailers today.

Astro in Action

• One of the world's largest home furnishing retailers managed HR data with siloed tools, creating high TCO, inconsistent governance, and compliance exposure under the EU Pay Transparency Directive. Adopting Astro consolidated Payroll, Time, Talent, and Absence pipelines on a single platform with consistent RBAC and automated scalability. The result was tripled velocity for new data product delivery, reusable pipeline components that accelerated development, and reliable HR insights that keep the business ahead of evolving employment regulations.
• One of the UK's largest grocery chains struggled with dozens of fragmented Amazon MWAA instances causing performance issues, rising costs, and limited observability across hundreds of business units, a challenge accelerated by a series of cloud provider outages. Adopting Astro provided the enterprise-grade reliability, observability, and access controls needed to consolidate workflows, improve data quality, and simplify a complex multi-cloud strategy at scale.

Modernizing the data stack is no longer optional for retailers. It’s now the foundation for real-time demand forecasting, AI-driven personalization, unified omnichannel experiences, and modern supply chain execution. From eliminating the drag of legacy systems to scaling reliable, high-quality data pipelines, leaders are redefining orchestration as the control plane for merchandising, inventory, store operations, and analytics.

The most advanced organizations aren’t just wiring systems together. They’re consolidating, automating, and accelerating their data operations with platforms like Astro. The path forward is clear: unify orchestration, strengthen trust in your data, and enable every customer interaction and operational decision to be informed, responsive, and data-driven.

→ Build a trusted, future-ready data stack today
Run an Astro TCO analysis and get in touch with our experts today to get results faster.