FAIR and Reproducible High-Throughput Workflows with AiiDA

ORAL

Abstract

The ever-growing availability of computational power and sustained development of computational methods have contributed much to recent scientific progress. This progress presents new challenges regarding the sheer amount of calculations and data to be managed. Next-generation exascale computing infrastructures will harden these challenges and require automated and scalable solutions. We have thus developed a comprehensive, robust, open source, high-throughput infrastructure AiiDA (http://aiida.net) dedicated to address the challenges in automated workflow management and data provenance storage. We discuss how AiiDA’s engine can now sustain throughputs of ~100'000 processes/hour, while automatically storing full data provenance graphs. These are stored in a database making data queryable, traversable, and directly enabling high-performance data analytics. Any simulation software can be interfaced to AiiDA via its open plugin repository. We demonstrate how AiiDA's workflow engine provides advanced automation and error handling features, allowing users to write modular workflows, interoperable between many different quantum or classical codes. We highlight how the resulting data can be disseminated on the Materials Cloud (http://materialscloud.org) in fully FAIR-compliant mode.

–

Presenters

  • Sebastiaan Huber

    • Ecole Polytechnique Federale de Lausanne

Authors

  • Sebastiaan Huber

    • Ecole Polytechnique Federale de Lausanne

  • Nicola Marzari

    • Ecole Polytechnique Federale de Lausanne
    • Theory and Simulation of Materials (THEOS), and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), Ecole Polytechnique Federale de Lausanne
    • École Polytechnique Fédérale de Lausanne
    • Theory and Simulation of Materials (THEOS), and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), Ecole Polytechnique Federale de Lausanne,
    • Theory and Simulation of Materials (THEOS), Faculté des Sciences et Techniques de l’Ingénieur, École Polytechnique Fédérale de Lausanne
    • THEOS, EPFL
    • École Polytechnique Fédérale de Lausanne (EPFL)
    • Theory and Simulation of Materials (THEOS) and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne (E
    • Theory and Simulation of Materials (THEOS), and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), EPFL, CH-1015 Lausanne, Switzerland
    • Theory and simulation of materials (THEOS), National Centre for Computational Design and Discovery of Novel Materials (MARVEL), EPFL
    • Materials Engineering, EPFL
    • Theory and Simulations of Materials (THEOS), and National Center for Computational Design and Discovery of Novel Materials (MARVEL), Ecole Polytechnique Federale de Lausanne

  • Giovanni Pizzi

    • Ecole Polytechnique Federale de Lausanne
    • Theory and Simulation of Materials (THEOS), and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), Ecole Polytechnique Federale de Lausanne,
    • Theory and Simulation of Materials (THEOS), Faculté des Sciences et Techniques de l’Ingénieur, École Polytechnique Fédérale de Lausanne
    • THEOS, EPFL