Today's day-to-day business activities are fully intertwined with digital processes. The number of those digital processes and their implementation as workflows is increasing rapidly, not least because of the growing importance of machine learning applications. Nowadays, analyses and forecasts are only started manually in the prototype state; a productive system relies on automation. Here, the choice of the workflow management platform is a key factor for long-term success.
The challenge is that these digital processes must be centrally managed and organized. Especially for business-critical processes, reliable execution and flexibility in the workflow design are essential. In addition to pure execution, great importance is also attached to the monitoring and optimization of workflows and error management. Ideally, the processes are also designed in such a way that they can be easily scaled up.
Only if both the technical and professional side of the users is involved, acceptance and a sustainable integration of digital processes into the daily work routine can be achieved. The execution as workflows should therefore be as simple and comprehensible as possible.
Creating advanced workflows in Python
In Apache Airflow the workflows are created with the programming language Python. The entry hurdle is low. In a few minutes you can define even complex workflows with external dependencies to third party systems and conditional branches.
Schedule, execute and monitor workflows
The program-controlled planning, execution and monitoring of workflows runs smoothly thanks to the interaction of the components. Performance and availability can be adapted to even your most demanding requirements.
Best suited for Machine Learning
Here, your Machine Learning requirements are met in the best possible way. Even their complex workflows can be ideally orchestrated and managed using Apache Airflow. The different requirements regarding software and hardware can be easily implemented.
Robust orchestration of third-party systems
Already in the standard installation of Apache Airflow numerous integrations to common third party systems are included. This allows you to realize a robust connection in no time. Without risk: The connection data is stored encrypted in the backend.
Ideal for the Enterprise Context
The requirements of start-ups and large corporations are equally met by the excellent scalability. As a top level project of the Apache Software Foundation and with its origins at Airbnb, the economic deployment on a large scale was intended from the beginning.
A major advantage of Apache Airflow is the modern, comprehensive web interface. With role-based authentication, the interface gives you a quick overview or serves as a convenient access point for managing and monitoring workflows.
Flexibility by customization
The adaptability is given by numerous plugins, macros and individual classes. Since Airflow is completely based on Python, the platform is theoretically changeable up to the basics. Adapt Apache Airflow to your current needs at any time.
Truly scalable
Scaling with common systems like Celery, Kubernetes and Mesos is possible at just any time. In this context a lightweight containerization can be installed.
Completely free of charge
The workflow management platform is quickly available without license fees and with minimal installation effort. You can always use the latest versions to the full extent without any fees.
Benefit from a whole community
As the de facto standard for workflow management, the Airflow Community not only includes users, but the platform also benefits from dedicated developers from around the world. Current ideas and their implementation in code can be found online.
Agility by simplicity
The workflow definition is greatly accelerated by the implementation in Python and the workflows benefit from the flexibility offered. In the web interface with excellent usability, troubleshooting and changes to the workflows can be implemented quickly..
The new major release of Apache Airflow offers a modern user interface and new functions:
Automatic file transfer
Learn more about data literacy in our blog post and how it helps companies shape the social component of their data strategy.
In our new blog post, we recap the Apache Airflow Summit 2022 and tell you all about the latest developments and trending topics.
We show you a possible approach using GitLab CI/CD as a framework to create an application-specific operating framework for Apache Airflow.
In this blog article, we will introduce you to the approaches available for providing external data for self-service analytics by business users.
In our blogpost we will focus on which feature store fits your machine learning framework and what implementation options are available.
In our blogpost we will introduce the concept of the feature store and highlight the promises it holds.
In today's Blogpost, we will show you how to optimize the interaction of SAP HANA with Python by using the NextLytics Python SAP HANA Connector.
We will introduce you to the Airflow API and give you important tips regarding the secure configuration of Airflow and the usage of the endpoints.
Find out how important feature engineering is for successful machine learning models and learn about its possibilities and limitations.
The new Apache Airflow upgrade will soon be one year old. If you haven't considered upgrading before, now is a good time to do so.