A Cluster Scheduler for Real-time GOCI-II Data Processing in Heterogeneous Computing Systems

Abstract

In June 2017, we conducted a Critical Design Review (CDR) of GOCI-II Ground Segment (G2GS) system and are currently developing G2GS system in line with the launch of the GOCI-II in 2019. As the performance improvement of the GOCI-II sensor, the amount of data to be processed has increased gradually, such as 4 times the spatial resolution, 2 times the L2 product, and 1.5 times the observation band. Due to this increase, we had to utilize both parallel and distributed processing techniques for processing GOCI-II L2 products in real time. First, parallel processing is applied to utilize optimal hardware resources (i.e. Multi-core CPU, GPU, and Xeon-Phi) for each L2 product application. Then, heterogeneous computing systems have been built so that these applications can be distributed and processed across multiple computing resources. In order to efficiently operate the resources of the heterogeneous computing systems, there is need for a scheduler that can load-balance all available resources. In this study, we implemented a scheduler to allocate tasks considering the load of all computing resources and the priority of each L2 application. By using optimal hardware resources for each algorithm and using more resources at the same time, the utilization rate of the entire system can be increased. the amount of data to be processed has increased gradually, such as 4 times the spatial resolution, 2 times the L2 product, and 1.5 times the observation band. Due to this increase, we had to utilize both parallel and distributed processing techniques for processing GOCI-II L2 products in real time. First, parallel processing is applied to utilize optimal hardware resources (i.e. Multi-core CPU, GPU, and Xeon-Phi) for each L2 product application. Then, heterogeneous computing systems have been built so that these applications can be distributed and processed across multiple computing resources. In order to efficiently operate the resources of the heterogeneous computing systems, there is need for a scheduler that can load-balance all available resources. In this study, we implemented a scheduler to allocate tasks considering the load of all computing resources and the priority of each L2 application. By using optimal hardware resources for each algorithm and using more resources at the same time, the utilization rate of the entire system can be increased.