The Astro API is Astronomer’s REST API for managing resources on Astro, for example to create a Deployment. This guide provides the following best practices for general development, and specifically development with the Astro API, to ensure a safe and optimal experience:
This guide highlights the following Astro features:
Astronomer provides several tools to manage Astro resources. Each tool offers benefits for specific use cases:
The tool that suits you best depends on factors such as your technical experience, existing knowledge in your organization, and use of Astronomer. It’s common to use all available tools.
It’s a best practice to ensure your code handles unexpected situations properly, such as errors. Because Astro’s tooling generally distinguishes between client-side and server-side errors, you can automate a process for determining the cause of an error. All HTTP 4XX code indicate a client-side error and all HTTP 5XX status codes indicate a server-side error.
For example, Deployment names are unique within an Astronomer Workspace. If you create a second Deployment with the same name as an already existing Deployment, the Astro API returns an HTTP 400 error.
To ensure your code handles errors correctly, wrap requests in a try/except code block:
The statement, response.raise_for_status(), raises an exception on any HTTP response code that’s not 2XX, which are all non-successful HTTP response codes. In the except clause, you can handle this exception however you want.
In case of an error, the Astro API returns a reason in the response body, with the key message. For this specific example, it returns HTTP code 400. This code example prints the error reason. Without the reason, you can’t know why a request failed. In the example of a duplicated Deployment name, the logs show the following:
The complete error response structure is:
For traceability purposes, you can also include the requestId in your logs, which is an internal Astronomer identifier that Astronomer support can use to track down your request.
In automated scripts, such as CI/CD pipelines, you can query the Astro API with an API token. API tokens grant access to certain Astronomer resources, so it’s important to keep the token safe. Do not hardcode the token in code. Instead, store the token in a secret and expose it as an environment variable, ASTRO_API_TOKEN. Storing API tokens in a system dedicated for storing secret values, like GitHub Actions Secrets, ensures secret values are not visible to humans and only referenced by code when needed.
Additionally, a best security practice is the Principle of Least Privilege, where you grant only the permissions necessary to perform an action. This reduces the attack surface, or the number of ways a bad actor could cause damage, in case of a leaked API token.
Astronomer provides three levels of API tokens, from least to most privilege:
The Astro API documentation provides a convenient web interface to try out the API:
A graphical REST API client can be a helpful addition when developing with any REST API. Popular tools include Postman and Insomnia. The Astro API documentation provides downloads to the API specifications, which are YAML files that define the API structure, that you can load into your tool of choice. Graphical REST API clients often provide convenience features over web-based documentation including query history, value sharing with variables, the ability to define environments with different values, and chained requests, which lets you use results from one query in a follow-up query.
The Astro API limits requests in case the request rate passes certain thresholds, depending on the type of the request. When a request is rate limited, the API returns an HTTP 429 status code.
It’s a best practice to apply an exponential backoff strategy to not overload the server side for rate-limited requests. An exponential backoff strategy gradually increases the time between requests to allow for the server to recover and respond correctly, for example, by waiting 1, 2, 4, or 8 seconds between consecutive requests.
Consider the scenario of waiting for a Deployment to receive status HEALTHY after creation. Creating a Deployment can take a moment, so repeatedly requesting the status from the Astro API without any pause between requests can cause rate limiting.
The following example checks the HTTP response code and handle HTTP 429 separately from other response codes. While this script won’t hit any rate limit threshold on the Astro API since the code waits 5 seconds between requests, running multiple scripts simultaneously can quickly increase requests. When receiving an HTTP 429 response, it calculates a waiting period of 2 ** (timeout_attempts - 1) seconds and then increases the period depending on the attempt number.
In the previous example, the code defines custom logic for handling rate limits and exponential backoffs. While this works, Python’s requests library comes with several built-in utilities you can use to simplify the code. For example, you can avoid defining your own exponential backoff logic by using the Python requests library.
requests.session() creates a persistent session between requests and replaces requests.get(...) with session.get(...) to use the settings configured in the session. This way you don’t have to duplicate the same if e.response.status_code == 429 business logic for every request. The code example below configures the urllib3.Retry exponential backoff logic for HTTP status code 429 and up to 10 attempts. This means it waits 1, 2, 4, …, 128, 256, and 512 seconds in between the ten attempts.
You can set a default value for headers using Python’s request library as a way to simplify your code. While you can write headers={"Authorization": f"Bearer {astro_api_token}"} with every request, it’s cleaner to define this value once and then automatically apply it to every request using the session object:
For repeated requests to the Astro API, or any REST API, it’s a best practice to reuse connections. That means you (client) keep a connection open to the Astro API (server) for multiple requests, instead of opening and closing a connection for every request. This reduces latency and CPU usage. Reusing HTTP connections is also referred to as HTTP persistent connections or HTTP keep-alive, where keep-alive refers to the Keep-Alive header that used to be transmitted with the message.
Using Python’s requests library again, requests.session object reuses connections:
A single connection then handles the two GET requests, instead of recreating a connection. This becomes visible when configuring DEBUG logging:
In the logs, you can see the first example creates two connections. The second example uses requests.session, which reuses connections, so it only creates one connection.