After three years of dedicated research and collaborative efforts, the eFlows4HPC project has successfully delivered a comprehensive workflow platform and an additional set of services facilitating the seamless integration of High-Performance Computing (HPC) simulation and modeling with big data analytics and machine learning techniques. Significant advancements in defining complex workflows resulted in enhanced workflow efficiency, and increased overall development and deployment productivity. A variety of scientific and industrial domains, including geosciences, climate science, manufacturing, and other initiatives within the European HPC and scientific applications ecosystem, will benefit from this innovative technology, reflecting the wide-reaching and transformative impact of the project.
The two major project outcomes, High-Performance Computing Workflows as a Service (HPCWaaS) and the project software stack, are crucial in supporting the development, deployment and execution of complex workflows. These advancements play a key role in tackling challenges related to natural disasters, climate change and the optimization of manufacturing processes. HPCWaaS helps users and developers to manage complex scientific workflows by simplifying their lifecycle steps and widening the access to HPC for newcomers. Additionally, the project has released as open source the eFlows4HPC software stack, which is versatile and applicable across numerous scientific and industrial applications requiring HPC, AI and big data techniques.
Rosa M. Badia, eFlows4HPC coordinator, thinks that “the achieved results mark a significant milestone, demonstrating the versatility of the eFlows4HPC software stack in leveraging HPC resources. This translates to a notable decrease in time-to-solution for application developers that need to combine traditional HPC with AI or big data in a single workflow.” She thinks that “we are shaping the future of HPC workflows and empowering researchers to tackle emerging challenges that require complex workflows.”
eFlows4HPC consortium has been highly engaged with multiple initiatives in order to ensure broader adoption of the eFlows4HPC methodologies by other scientific and industrial communities. Joint collaboration activities have been organized with other European initiatives and HPC Centers of Excellence (CoEs), such as ESiWACE3 (climate and weather simulations) and PerMedCoE (personalized medicine) communities. The project has also supported ChEESE and ChEESE-2P by developing complex workflows for earthquake impact simulation. eFlows4HPC methodologies will also be used for the development of workflows in DT-GEO, that aims to build a digital twin for geophysical extremes, and the CAELESTIS project. Finally, the SKA Regional Centres, the regional support network for the international initiative providing square-kilometre array telescopes, showed interest in evaluating the HPCWaaS methodology.
The eFlows4HPC project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. It also received funding from MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957).