Use TransformTransport to customize OpenLineage events

TransformTransport allows you to modify OpenLineage events just before they are sent, on a per-transport basis. This lets you send customized or transformed events to specific backends without changing the global OpenLineage configuration.

Use OpenLineage client version 1.38 or higher. Upgrading the OpenLineage client version is safe and independent of your Airflow version. Astronomer recommends always using the latest version of OpenLineage client.

Add the following to your requirements.txt to upgrade:

openlineage-python>=1.38.0

You can define your own transformer class to modify events as needed, or use one of the predefined transformers available for common use cases like renaming job namespaces.

If you use TransformTransport, it is important to note:

• TransformTransport only modifies the event right before sending.
• TransformTransport does not change the global OpenLineage configuration or variables such as the namespace. It means that the OpenLineage namespace remains unchanged globally.
• If you use global OpenLineage namespace elsewhere, such as passing it to Spark operators or OpenLineage macros, those references will still have the original namespace value. In such cases, you might need to manually update those references to ensure consistency or implement a similar approach for Spark integration.

This guide presents an example configuration that demonstrates how to use TransformTransport to customize OpenLineage events and send them to both Astro Observe and Atlan with different job namespaces. This approach can be adapted for other use cases or backends that require event transformation before delivery. See OpenLineage documentation for more examples.

Send OpenLineage events to Observe and Atlan with different namespaces

An example use case of TransformTransport for OpenLineage is you want to send OpenLineage events to both Astro Observe and Atlan, but need the OpenLineage namespace to be different only for Atlan.

This can be done using the predefined JobNamespaceReplaceTransformer transformer. This transformer lets you dynamically override the job namespace for specific backends using the TransformTransport interface.

Astro already configures OpenLineage transport for the default Astro Observe destination, so you don’t need to configure it again.

To add a secondary Atlan backend with a different job namespace, define the following environment variables:

# Transform applied only to Atlan transport
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TYPE=transform
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSFORMER_CLASS=openlineage.client.transport.transform.JobNamespaceReplaceTransformer
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSFORMER_PROPERTIES={"new_job_namespace": "<atlan-namespace-value>"}
# HTTP transport configuration for Atlan, to be used after transformation
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__TYPE=http
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__URL=https://<instance>.atlan.com/events/openlineage/airflow-astronomer/
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__AUTH={"type":"api_key", "apiKey":"<API_token>"}

Verify the setup

1. Trigger a DAG that emits OpenLineage events.
2. Confirm events appear in Astro Observe, the default backend.
3. Check your Atlan instance to confirm receipt of events with the updated namespace.
4. Check Airflow logs for any errors related to the OpenLineage transport layer.

If you do not overwrite the default value of OPENLINEAGE__TRANSPORT__CONTINUE_ON_FAILURE=true, a failed Atlan transport will not affect Astro delivery.