NextLytics
    Products
      IT-Services
        Business Analytics
          Career
            Blog

              Machine Learning workflows
              in Apache Airflow

              Digital workflow management is growing in importance

              Today's day-to-day business activities are fully intertwined with digital processes. The number of those digital processes and their implementation as workflows is increasing rapidly, not least because of the growing importance of machine learning applications. Nowadays, analyses and forecasts are only started manually in the prototype state; a productive system relies on automation. Here, the choice of the workflow management platform is a key factor for long-term success.

              The challenge is that these digital processes must be centrally managed and organized. Especially for business-critical processes, reliable execution and flexibility in the workflow design are essential. In addition to pure execution, great importance is also attached to the monitoring and optimization of workflows and error management. Ideally, the processes are also designed in such a way that they can be easily scaled up. 

              Only if both the technical and professional side of the users is involved, acceptance and a sustainable integration of digital processes into the daily work routine can be achieved. The execution as workflows should therefore be as simple and comprehensible as possible.

              Digital workflows with the open source platform Apache Airflow

              Workflows_1

              Creating advanced workflows in Python

              In Apache Airflow the workflows are created with the programming language Python. The entry hurdle is low. In a few minutes you can define even complex workflows with external dependencies to third party systems and conditional branches.
              Workflows_3

              Schedule, execute and monitor workflows

              The program-controlled planning, execution and monitoring of workflows runs smoothly thanks to the interaction of the components. Performance and availability can be adapted to even your most demanding requirements.
              SQL-Datenbank

              Best suited for Machine Learning

              Here, your Machine Learning requirements are met in the best possible way. Even their complex workflows can be ideally orchestrated and managed using Apache Airflow. The different requirements regarding software and hardware can be easily implemented.
              Sicherheit

              Robust orchestration of third-party systems

              Already in the standard installation of Apache Airflow numerous integrations to common third party systems are included. This allows you to realize a robust connection in no time. Without risk: The connection data is stored encrypted in the backend.
              SQL-Skalierung

              Ideal for the Enterprise Context

              The requirements of start-ups and large corporations are equally met by the excellent scalability. As a top level project of the Apache Software Foundation and with its origins at Airbnb, the economic deployment on a large scale was intended from the beginning.

              A glance at the comprehensive intuitive web interface

              A major advantage of Apache Airflow is the modern, comprehensive web interface. With role-based authentication, the interface gives you a quick overview or serves as a convenient access point for managing and monitoring workflows.

              The orchestration of third-party systems is realized through numerous existing integrations.

              • Apache Hive
              • Kubernetes Engine
              • Amazon DynamoDB
              • Amazon S3
              • Amazon SageMaker
              • Databricks
              • Hadoop Distributed File System (HDFS)
              • Bigtable
              • Google Cloud Storage (GCS)
              • Google BigQuery
              • Google Cloud ML Engine
              • Azure Blob Storage
              • Azure Data Lake
              • ...
              Orchestrierung
              The workflow management platform for your demands
              20_HG_R_Zahnrad

              Flexibility by customization

              The adaptability is given by numerous plugins, macros and individual classes. Since Airflow is completely based on Python, the platform is theoretically changeable up to the basics. Adapt Apache Airflow to your current needs at any time.

              Truly scalable

              Scaling with common systems like Celery, Kubernetes and Mesos is possible at just any time. In this context a lightweight containerization can be installed.
              HG_L_Skalierung_1
              20_HG_R_Kosten

              Completely free of charge

              The workflow management platform is quickly available without license fees and with minimal installation effort. You can always use the latest versions to the full extent without any fees.

              Benefit from a whole community

              As the de facto standard for workflow management, the Airflow Community not only includes users, but the platform also benefits from dedicated developers from around the world. Current ideas and their implementation in code can be found online.
              HG_L_Community
              HG_R_Userfriendly_1

              Agility by simplicity

              The workflow definition is greatly accelerated by the implementation in Python and the workflows benefit from the flexibility offered. In the web interface with excellent usability, troubleshooting and changes to the workflows can be implemented quickly..
              Best-in-class workflow management with Apache Airflow 2.0?  How do you manage your workflows with Apache Airflow? Which application scenarios are feasible in practice? With which features does the new major release react to the current challenges of workflow management?   Download Whitepaper  

              State-of-the-art workflow management with Apache Airflow 2.0

              The new major release of Apache Airflow offers a modern user interface and new functions:

              • Fully functional REST API with numerous endpoints for two-way integration of Airflow into different systems such as SAP BW
              • Functional definition of workflows to implement data pipelines for improved data exchange between tasks in the workflow using the TaskFlow API
              • Interval-based checking of an starting condition with Smart Sensors, which keep the workload of the workflow management system as low as possible
              • Increased usability in many areas (simplified Kubernetes operator, reusable task groups, automatic update of the web interface)

              Pipeline

              Apache Airflow in day-to-day business
              Icon_Datenübertragung
               

              Automatic file transfer

              Due to the large number of integrations to other systems, the transfer of data and files can be easily realized. So-called sensors are used to check start conditions such as the existence of a file at periodic intervals. For example, a CSV file can be loaded from the cloud service into a database. In this way, unnecessary manual work can be automated.
              Icon_Projekt
               

              Executing ETL workflows

              Your data management will benefit the most from Apache Airflow. It has never been easier to combine different, structured data sources into one target table. Originally Airflow was developed according to the needs of Extract-Transform-Load (ETL) workflows and offers smart concepts like reusable workflow parts to build even complex workflows fast and robust.
              Icon_Automatisierung
               

              Triggering of external process chains via API

              If there is no easy way for system integration per task operator (and the community has not provided a solution yet), the connection via APIs and HTTP Requests is still possible at any time. For example, a process chain in SAP BW can be started and tracked synchronously. Conversely, system integration through Airflows excellent API is possible without any problems.
              Icon_MachineLearning
               

              Implement Machine Learning process

              Not only during the development of a Machine Learning application many processes which are ideally implemented as a workflow exist. The productive execution of a model can also be implemented as a workflow and thus provide, for example, current forecast data at fixed intervals. The data preparation and training of the current model version can be easily realized with Airflow.
              Do you like to learn more about Machine Learning?
              In our blog you will find more interesting articles on this topic
              Major Release Apache Airflow 2.0

              Apache Airflow 2.0 - New features for your workflow management

              Apache Airflow 2.0 further optimizes workflow management. The major release includes TaskGroups, the Taskflow API, Smart Sensors and much more.

              Machine Learning customer segmentation rfm analysis

              Machine Learning in customer segmentation with RFM-analysis

              Customer analyses are essential to align marketing strategies. Read how Machine Learning helps with customer segmentation using RFM-analysis

              Crystal ball prediction Machine Learning NextLytics Blog

              Machine Learning & the Day after Corona - Does the Glass Ball need to be Reexamined?

              Corona has changed the day-to-day routine of companies. How does this affect your planning cycles and planning tools? Can machine learning models provide support here?

              Machine Learning in the SAP Ecosystem: SAP HANA Predictive Analysis Library (PAL)

              In-Database Machine Learning with SAP HANA Predictive Analysis Library

              Data is the new currency these days. With SAP HANA Predictive Analysis Library (PAL) the full potential of enterprise data can be exploited. Read more about this in our blog article!

              How to use TensorFlow with SAP HANA EML

              How to use TensorFlow with SAP HANA EML

              SAP HANA External Machine Learning (EML) as part of the SAP HANA Application Function Library (AFL) allows the execution of already trained TensorFlow models

              How to create forecasts with SAP Analytics Cloud for Planning

              SAP Analytics Cloud for Planning can be used to create time series forecasts in a story. We introduce the forecast function and refer also to a machine learning approach to improve forecasts

              Can SAP Analytics Cloud perform Machine Learning?

              In this article we take a look at predictive analytics functions (Machine Learning, Augmented Analytics and Smart Predict) in the context of SAP Analytics Cloud

              Machine Learning with SAP Predictive Analytics - possibilities and limitations

              Machine Learning with SAP Predictive Analytics - possibilities and limitations

              Machine Learning with SAP Predictive Analytics enables companies to forecast future events and make data-driven, informed decisions. In this article we will highlight the possibilities and limitations of this tool.

              New whitepaper: How to bring machine learning and SAP BW together

              Find out in our white paper how you can bring machine learning and SAP BW together to generate added value for your company.

              PyCon and PyData in Berlin

              PyCon DE / PyData 2019: News on Machine Learning and Data Analytics

              Our Recap of PyCon DE / PyData 2019. We can recommend the conference for all Python users in the data analytics environment and machine learning area.

              Do you have any questions or need support for your next AI project?   We will be happy to assist you in implementing or optimizing your AI-based application with our know-how and show you how Machine Learning can provide added value for you and your company.   Talk to us!