Following an overview of services offered and a discussion of data security and governance, attendees were given the opportunity to spin up a small functioning HPC cluster on the EC2 platform, demonstrating the simplicity with which this can be done.
Whilst HPC has historically been at the forefront of developments in computing technology, adapting to the Cloud presents some new and unique challenges. In the first of N8 CIR's technical workshops, Phil Edwards and Scott Eberhardt of Amazon Web Services (AWS) were invited to demonstrate some of the latest developments intended to aid this process. The event was well attended by invited researchers and systems specialists from across the N8 member organisations.
Before getting into the detail, much emphasis was placed on the importance of information security at AWS and the variety of options available for the protection and governance of data. This is of particular importance given the highly sensitive and confidential nature of much research data.
As reflected by the renaming of N8 HPC to N8 CIR, computationally intensive research extends beyond traditional HPC workloads to encompass such realms as data analytics, high throughput computing (HTC). With regard to the latter of these, Christopher Paul from Research IT, University of Manchester, demonstrated bursting HTCondor into AWS using the recently developed Annex functionality. This offers the potential to deal with high demand for resources, crucially when the spot pricing of the required cloud instances falls within a predetermined acceptable range.
Attendees were then guided through the process of spinning up a virtual cluster using AWS's open source ParallelCluster tool, demonstrating the ease and speed with which an environment for running HPC workloads can be set up. Through the use of templates, virtual clusters can easily be tailored to the requirements of the workload. This stands in contrast to traditional workload scheduling, where jobs must be effectively "slotted in" to fit the available resources.
Finally, recent and future developments aimed at the HPC sector were presented. Some of the highlights included AWS FSx for Lustre, improvements to network latency for comms-heavy workloads, and a number of new extremely high memory instance types.
Interested parties are invited to contact their N8 CIR TMG representative for further information. It is intended that repeat or similar events, including other major Cloud platforms, will be arranged throughout the N8 member institutions in the coming months. The exchange and discussion of experiences using these resources is also welcomed and indeed encouraged.