High-Performance Single-Particle Imaging Reconstruction on Pre-Exascale Computing Platforms

ORAL  · Invited

Abstract

Single-particle imaging (SPI) reconstruction is the analysis of X-ray diffraction patterns from a light source directed at biological molecules. SPI uses intense X-ray free electron laser (FEL) pulses to obtain a continuous diffraction pattern from single molecules in a serial manner--one molecule at a time The analysis is computationally intensive, requiring diffraction patterns to be sorted into conformations and oriented in 3D for model building and refinement. SPI is also data-intensive, requiring potentially hundreds of thousands of diffraction patterns in order to produce a high-resolution result.

The ExaFEL project is an Exascale Computing Project with the goal of performing SPI reconstruction in quasi-real time where the analysis keeps up with experimental data rates. Upgraded detectors and light sources such as the Linac Coherent Light Source (LCLS) are delivering SPI experimental data at ever higher velocities and larger volumes.

This talk will overview the capabilities of SpiniFEL, the high-performance analytics code for SPI reconstruction developed by the ExaFEL team. SpiniFEL implements the multi-tiered iterative phasing (M-TIP) algorithm for SPI reconstruction. SpiniFEL has been accelerated via GPU offloading as well as through parallel and distributed programming models. It can run using the Message Passing Interface (MPI) or via the task-based Legion Runtime System. SpiniFEL runs on the pre-exascale machines Summit at the Oak Ridge Leadership Computing Facility (OLCF) and Perlmutter at the National Energy Research Scientific Computing Center (NERSC).

We will present results of SpiniFEL runs on pre-exascale computing platforms and outline how SpiniFEL SPI analysis fits into the larger vision of an inter-facility workflow that moves data from acquisition at an experimental facility to computation at a supercomputing center, so scientists can quickly determine structures and produce high quality results efficiently using scarce experimental resources.

*Office of Science and NNSA (grant No. 17-SC-20-SC; WBS 2.2.4.05 to ExaFEL); Office of Science (grant No. DE-AC02-05CH11231 to ASCR and BES programs).

Publication: Shih, Yu-hsuan, Garrett Wright, Joakim And'en, Johannes Blaschke and Alex H. Barnett. "cuFINUFFT: a load-balanced GPU library for general-purpose nonuniform FFTs." 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) (2021): 688-697.

Kommera, Pranay Reddy, Vinay Ramakrishnaiah, Christine Sweeney, Jeffrey Donatelli, and Petrus H. Zwart. GPU-accelerated multitiered iterative phasing algorithm for fluctuation X-ray scattering. J. Appl. Cryst. (2021). 54, 1179–1188.

Chang, Hsing-Yin, Elliott Slaughter, Seema Mirchandaney, Jeffrey Donatelli, and Chun Hong Yoon. Scaling and Acceleration of Three-Dimensional Structure Determination for Single-Particle Imaging Experiments with SpiniFEL. The 4th Annual
Parallel Applications Workshop, Alternatives To MPI+X. November 2021.

Presenters

  • Christine Sweeney

    • Los Alamos National Laboratory

Authors

  • Christine Sweeney

    • Los Alamos National Laboratory
  • Christine Sweeney

    • Los Alamos National Laboratory
  • Johannes Blaschke

    • Lawrence Berkeley National Laboratory
  • Hsing-Yin Chang

    • SLAC National Accelerator Laboratory
  • Jeffrey Donatelli

    • Lawrence Berkeley National Laboratory
  • Antoine Dujardin

    • SLAC National Accelerator Laboratory
  • Seema Mirchandaney

    • SLAC National Accelerator Laboratory
  • Ariana Peck

    • SLAC National Accelerator Laboratory
  • Amedeo Perazzo

    • SLAC National Accelerator Laboratory
  • Vinay Ramakrishnaiah

    • Los Alamos National Laboratory
  • Elliott Slaughter

    • SLAC National Accelerator Laboratory
  • Monarin Uervirojnangkoorn

    • SLAC National Accelerator Laboratory
  • Chun Hong Yoon

    • SLAC National Accelerator Laboratory
  • Petrus H Zwart

    • Lawrence Berkeley National Laboratory