Presentation
Managing Python in HPC Environments
Author/Presenters
Event Type
Workshop
W
Parallel Application Frameworks
Reproducibility
Scientific Computing
TimeMonday, November 12th11am - 11:30am
LocationD220
DescriptionPython has seen a rapid adoption in the weather and climate modeling science communities. This swift rise has taken HPC system administrators by surprise, leading to inadequate support. These trends, like those in other sciences, led to the development and widespread adoption of user managed binary distributions. An example being Anaconda in 2012, which comes with security risks. We present a system for mirroring Anaconda Python that embeds PyRats, a dependency analyzer and logger descended from Blais’ Snakefood into the Anaconda installers. We show that Anaconda performance, reliability, security, and availability can be improved dramatically and enable timely integration into automated test environments. The dependency logging yields insights into which packages users rely on most. It can help prioritize optimization efforts such as building packages for the CPU families used in a given HPC environment. We also discuss related work, including a complimentary automated Python provisioning effort by Oak Ridge National Laboratory that they call PythonEnv-noaa.
Archive
