Nebulaworks engineered a cloud hosted High Performance Compute (HPC) environment to accelerate R&D

Challenge

Nebulaworks partnered with a client to revolutionize their High-Performance Compute (HPC) environment in AWS. Faced with the need for a scalable, multi-tenant High-Performance Computing (HPC) environment for X-ray Crystallography analysis, Cryogenic Electron Microscopy, and Computational Chemistry analysis, the client sought Nebulaworks' expertise to integrate cutting-edge AWS services and open-source technologies, including CryoSPARC, to enhance their analysis and research capabilities. Prior to engaging Nebulaworks, they lacked a dependable and scalable platform deployed by Infrastructure-As-Code (IAC) to support their research and development efforts.

Solution

In order to accomodate the specific needs of the business stakeholders, Nebulaworks embedded a Product Engineering team to work directly with the Line of Business (LOB) to understand the mission and vision for the HPC system that needed to be implemented. Using a prescriptive process for engineering Product, Discovery sessions were critical to gather the state of the existing solution, and determine how to design a scalable environment that supported various teams. A Product Charter was developed as a result of Discovery that encapsulated the higher level mission. Objectives and Key Results (OKRs) were established from the Charter to guide deliverables on a quarterly basis. The Product Engineering team developed Epics and User Stories to begin implementing the system.

Equipped with a clear understanding of deliverables, Nebulaworks designed and implemented the custom HPC platform leveraging AWS’s robust cloud services. This solution utilized AWS S3 for data management, Amazon FSx for fast data access, AWS RDS for secure database storage for finished analysis, Amazon EC2’s ParallelCluster for scalable computing resources, AWS ECS for hosting the Data API, and AWS CloudFront for global delivery of the system’s frontend component. CryoSPARC, used for Cryogenic Electron Microscopy analysis, further empowered the teams scientific computing capabilities. Using a cloud based IAC approach, CryoSPARC was deployed in a repeatable way, enabling reproducibility and repeatabiliy for a critical system component. In addition to CryoSPARC, PipeDream was deployed using IAC to enable X-ray Crystallography analysis. The Nebulaworks team leveraged Git for version control, trunk-based development for stable releases, and a Product Engineering approach to ensure the solution precisely addressed the client’s specific challenges.

Why Nebulaworks

Nebulaworks stands out for its deep experience in leveraging AWS technologies to accelerate R&D in various sectors. Our flexible, customer-centric approach allows us to tailor HPC solutions to each client’s unique tooling, process requirements, and mission. Our embedded engineering teams, adept in agile methodologies, work closely with client stakeholders to build solutions that effectively solve their problems, ensuring a seamless transition from concept to operational workload.

Outcomes

The bespoke HPC solution drastically reduced the time from concept to operational workload, enabling the client to accelerate their analysis and research discoveries. By simplifying the integration of complex software and adopting agile methodologies, Nebulaworks not only delivered a robust HPC environment but also fostered a seamless, efficient research process. This partnership highlighted the power of combining expert knowledge with cutting-edge technology to advance scientific research.

The engagement resulted in a state-of-the-art a solution that significantly accelerated two main components to accelerate research. First, the deployment of the CryoSPARC tool in a repeatable way to perform Cryogenic Electron Microscopy analysis. Second the deployment and utilization of PipeDream for X-ray Crystallography analysis. The solution not only met the rigorous demands of scientific research but was also scalable, secure, and integrated with the latest open-source technologies.

In conclusion, this case study showcases Nebulaworks' expertise in tailoring flexible solutions to meet specific research needs, significantly reducing operational workloads and accelerating the path to groundbreaking discoveries and analysis.