"CloudHPC" - Harnessing Cloud Computing to Power Up HPC Applications

07 August 2017

Project coordinator:

Luciano Paschoal Gaspary

Institute of Informatics, Federal University of Rio Grande do Sul (UFRGS)


Project partners:

Vladimir Korkhov

Department of Computer Modeling and Multiprocessor Systems, St. Petersburg State University (SPbSU)


Wang Xingce

Computer Science Department, College of Information Science and Technology, Beijing Normal University (BNU)


Funding agencies:

CNPq (Brazil), MOST (China)


Handling massive amounts of data is commonplace for most modern scientific, engineering, and business applications. As these applications need to target several big data related challenges while delivering expected results in a timely manner, they frequently pose large computing power requirements. In this context, High Performance Computing (HPC) becomes a key factor for speeding up data processing, while also enabling faster time to market, lower capital expenditures, and higher valued innovation. To this end, HPC solutions have traditionally taken advantage of cluster and datacenter infrastructures for running applications having those computing power requirements. In addition, practitioners have also been leveraging cloud computing resources for meeting HPC demands when available resources do not suffice. In fact, the pay-per-use cost model and resource elasticity makes cloud computing an interesting environment for HPC, which can be provided with instant availability and flexible scaling of resources, among other recognized advantages.

In spite of the benefits of using cloud computing for HPC, a current approach has been the allocation of physical infrastructures in dedicated mode for fast HPC provisioning. Although convenient, it frequently leads to underutilized resources, i.e., an application may not fully utilize provided CPU and/or network resources. It also prevents dealing adequately with those applications whose resource demands grow beyond available capacity. Traditional virtualization technologies can help solving the problem but the overhead of both a) bootstrapping a virtual infrastructure for each application and b) sharing physical resources among several virtual instances might be significant. Boosting available physical resources by using cloud computing, in turn, has been hampered because of limited support for shifting HPC applications to the cloud. These issues hinder the wide adoption of cloud computing by the HPC community, thus becoming paramount to understand how one can perform smooth and effective migration of (parts of) HPC applications to the cloud.

In this research project, our main goal is to explore opportunities for symbiotic, coordinated use of HPC and cloud computing infrastructures. The idea is, starting from an HPC infrastructure such as the ones in place in BRICS member countries, to identify cloud computing aspects that could be explored to provide a far-reaching, flexible (e.g., in terms of adaptation to fluctuating demands), secure and efficient environment for running HPC. We will focus on devising models, algorithms and mechanisms to help users in having access to a suitable execution platform (built on top of open source-based developments) for running their applications, in those scenarios in which users' own infrastructure is missing or lacks required resources.

As an integral axis of the project, we will also investigate how different big data, science-related applications can be modeled so as to perform adequately when executed on top of the proposed hybrid environment.