Ein generisches und hoch skalierbares Framework zur Automatisierung und Ausführung wissenschaftlicher Datenverarbeitungs- und Simulationsworkflows

Scientists and engineers designing and implementing complex system solutions use
computational workflows for simulations, analysis, and evaluations. Along with
growing system complexity, the complexity of these workflows also increases. However,
without integration tools, scientists and engineers are more concerned with
implementing additional interfaces to integrate software tools and model sets, which
hinders their original research or engineering aims. Therefore, efficient automation
and parallel computation of complex workflows are increasingly important in order
to perform computational science in many scientific fields like energy and environmental
informatics. When coupling heterogeneous models and other executables, a
wide variety of software infrastructure requirements must be considered to ensure
the compatibility of workflow components. The consistent utilization of advanced
computing capabilities and the implementation of sustainable software development
concepts that guarantee maximum efficiency and reusability are further issues that
scientists within research organizations must regularly meet. This thesis addresses
... mehr
these challenges by presenting a new generic, modular, and highly scalable
process operation framework for efficient coupling and automated execution of
computational scientific workflows.
Based on a microservice architecture utilizing container virtualization and orchestration,
the framework supports the flexible and efficient parallelization of computational
tasks on distributed cluster nodes. Using distributed message-oriented
middleware and different I/O adapters provides a scalable and high-performance
communication infrastructure for data exchange between executables, allowing the
computation of workflows without requiring the adjustment of executables or the
implementation of interfaces or adapters. The convenient user interface based on
Apache NiFi technology ensures the simplified specification, processing, controlling,
and evaluation of computational scientific workflows. Due to the framework’s high
scalability and extended flexibility, use cases benefitting from parallel execution
are parallelized, thereby significantly saving runtime and improving operational
efficiency, especially during complex tasks like iterative grid optimization.