Shaping Business Intelligence
About us
Our Methodology
NextTables
NextLytics Connectors
Data Science & Engineering
SAP Planning
Dashboarding
SAP Data Warehouse
Procurement Analytics
SAP HR Reporting & Analytics
Data Quality Management
Working at NextLytics
Senior
Junior
Students
Jobs
Automated digital workflows are key to efficient and successful business processes: Departments can focus on the important decisions and always have the latest reports at their disposal, analytical dashboards show up-to-date information, and prediction models are trained with the most fit data available. Availability and accountability of data are crucial for trust in data-driven decision making and planning. Trust the world’s most popular open source workflow orchestration platform to run your workflows like clockwork: Apache Airflow
Apache Airflow is the leading open source workflow orchestration framework. It runs the time-critical processes and data pipelines of hundreds of companies worldwide including pioneers of the data-centric business model like AirBnB, Pinterest, Spotify, or Zalando. Software-as-a-Service giants AWS and Google Cloud Platform offer Airflow as part of their regular service catalog and Astronomer even builds their whole product and company around the open source project.
In Apache Airflow the workflows are created with the programming language Python. The entry hurdle is low. In a few minutes you can define even complex workflows with external dependencies to third party systems and conditional branches.
The program-controlled planning, execution and monitoring of workflows runs smoothly thanks to the interaction of the components. Performance and availability can be adapted to even your most demanding requirements.
Here, your Machine Learning requirements are met in the best possible way. Even their complex workflows can be ideally orchestrated and managed using Apache Airflow. The different requirements regarding software and hardware can be easily implemented.
Already in the standard installation of Apache Airflow numerous integrations to common third party systems are included. This allows you to realize a robust connection in no time. Without risk: The connection data is stored encrypted in the backend.
The requirements of start-ups and large corporations are equally met by the excellent scalability. As a top level project of the Apache Software Foundation and with its origins at Airbnb, the economic deployment on a large scale was intended from the beginning.
Apache Hive
Kubernetes Engine
Amazon DynamoDB
Amazon S3
Amazon SageMaker
Databricks
Hadoop Distributed File System (HDFS)
Bigtable
Google Cloud Storage (GCS)
Google BigQuery
Google Cloud ML Engine
Azure Blob Storage
Azure Data Lake
...
Due to the large number of integrations to other systems, the transfer of data and files can be easily realized. So-called sensors are used to check start conditions such as the existence of a file at periodic intervals. For example, a CSV file can be loaded from the cloud service into a database. In this way, unnecessary manual work can be automated.
If there is no easy way for system integration per task operator (and the community has not provided a solution yet), the connection via APIs and HTTP Requests is still possible at any time. For example, a process chain in SAP BW can be started and tracked synchronously. Conversely, system integration through Airflows excellent API is possible without any problems.
Your data management will benefit the most from Apache Airflow. It has never been easier to combine different, structured data sources into one target table. Originally Airflow was developed according to the needs of Extract-Transform-Load (ETL) workflows and offers smart concepts like reusable workflow parts to build even complex workflows fast and robust.
Not only during the development of a Machine Learning application many processes which are ideally implemented as a workflow exist. The productive execution of a model can also be implemented as a workflow and thus provide, for example, current forecast data at fixed intervals. The data preparation and training of the current model version can be easily realized with Airflow.
/Logo%202023%20final%20dunkelgrau.png?width=221&height=97&name=Logo%202023%20final%20dunkelgrau.png)
