Matthieu Dreher, Swann Perarnau, T. Peterka, K. Iskra, P. Beckman
Implementing an in situ workflow involves several challenges related to data placement, task scheduling, efficient communications, scalability, and reliability. Most of the current implementations provide reasonably performant solutions to these issues by focusing on high-performance communications and low-overhead execution models at the cost of reliability and flexibility. One of the key design choices in such infrastructures is between providing a single-program, integrated environment or a multiple-program, connected environment, both solutions having their own strengths and weaknesses. While these approaches might be appropriate for current production systems, the expected characteristics of exascale machines will shift current priorities. After a survey of the trade-offs and challenges of integrated and connected in situ workflow solutions available today, we discuss in this paper how exascale systems will impact those designs. In particular, we identify missing features of current system-level software required for the evolution of in situ workflows toward exascale and how system software innovations from the Argo Exascale Computing Project can help address those challenges.
{"title":"In Situ Workflows at Exascale: System Software to the Rescue","authors":"Matthieu Dreher, Swann Perarnau, T. Peterka, K. Iskra, P. Beckman","doi":"10.1145/3144769.3144774","DOIUrl":"https://doi.org/10.1145/3144769.3144774","url":null,"abstract":"Implementing an in situ workflow involves several challenges related to data placement, task scheduling, efficient communications, scalability, and reliability. Most of the current implementations provide reasonably performant solutions to these issues by focusing on high-performance communications and low-overhead execution models at the cost of reliability and flexibility. One of the key design choices in such infrastructures is between providing a single-program, integrated environment or a multiple-program, connected environment, both solutions having their own strengths and weaknesses. While these approaches might be appropriate for current production systems, the expected characteristics of exascale machines will shift current priorities. After a survey of the trade-offs and challenges of integrated and connected in situ workflow solutions available today, we discuss in this paper how exascale systems will impact those designs. In particular, we identify missing features of current system-level software required for the evolution of in situ workflows toward exascale and how system software innovations from the Argo Exascale Computing Project can help address those challenges.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"85 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124677212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
D. Thompson, S. Jourdain, A. Bauer, Berk Geveci, Robert Maynard, Ranga Raju Vatsavai, P. O’leary
Summarization and compression at current and future scales requires a framework for developing and benchmarking algorithms. We present a framework created by integrating existing, production-ready projects and provide timings of two particular algorithms that serve as exemplars for summarization: a wavelet-based data reduction filter and a generator for creating image-like databases of extracted features (isocontours in this case). Both support browser-based, post-hoc, interactive visualization of the summary for decision-making. A study of their weak-scaling on a distributed multi-GPU system is included.
{"title":"In Situ Summarization with VTK-m","authors":"D. Thompson, S. Jourdain, A. Bauer, Berk Geveci, Robert Maynard, Ranga Raju Vatsavai, P. O’leary","doi":"10.1145/3144769.3144777","DOIUrl":"https://doi.org/10.1145/3144769.3144777","url":null,"abstract":"Summarization and compression at current and future scales requires a framework for developing and benchmarking algorithms. We present a framework created by integrating existing, production-ready projects and provide timings of two particular algorithms that serve as exemplars for summarization: a wavelet-based data reduction filter and a generator for creating image-like databases of extracted features (isocontours in this case). Both support browser-based, post-hoc, interactive visualization of the summary for decision-making. A study of their weak-scaling on a distributed multi-GPU system is included.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124422433","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Heirich, Elliott Slaughter, Manolis Papadakis, Wonchan Lee, T. Biedert, A. Aiken
This short paper describes an experimental prototype of in situ visualization in a task-based parallel programming framework. A set of reusable visualization tasks were composed with an existing simulation. The visualization tasks include a local OpenGL renderer, a parallel image compositor, and a display task. These tasks were added to an existing fluid-particle-radiation simulation and weak scaling tests were run on up to 512 nodes of the Piz Daint supercomputer. Benchmarks showed that the visualization components scaled and did not reduce the simulation throughput. The compositor latency increased logarithmically with increasing node count.
{"title":"In situ visualization with task-based parallelism","authors":"A. Heirich, Elliott Slaughter, Manolis Papadakis, Wonchan Lee, T. Biedert, A. Aiken","doi":"10.1145/3144769.3144771","DOIUrl":"https://doi.org/10.1145/3144769.3144771","url":null,"abstract":"This short paper describes an experimental prototype of in situ visualization in a task-based parallel programming framework. A set of reusable visualization tasks were composed with an existing simulation. The visualization tasks include a local OpenGL renderer, a parallel image compositor, and a display task. These tasks were added to an existing fluid-particle-radiation simulation and weak scaling tests were run on up to 512 nodes of the Piz Daint supercomputer. Benchmarks showed that the visualization components scaled and did not reduce the simulation throughput. The compositor latency increased logarithmically with increasing node count.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"95 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127099680","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The ultimate goal for radiation transport is to perform full-core reactor modelling and simulation. Advances in computational simulation bring this goal close to reality and the newest Monte Carlo transport codes have begun to shift to using accelerators that have become a stalwart in the supercomputing and HPC space. Within the reactor modelling and simulation community, Monte Carlo transport simulations are considered the gold standard for simulation. Through the use of "combinatorial geometry" (constructive solid geomtry), complex models can be used with fewer approximation compromises while at the same time scale to run on some of the largest supercomputers in the world. Unfortunately, the state-of-the-art for "combinatorial geometry" visualization is to decompose the geometry into a mesh. This approach could require a significant amount of memory which is antithetical to in situ visualization. To address this issue, we introduce a ray caster for visualizing combinatorial geometry in radiation transport code. By only using the accelerators for the radiation transport code and leaving the CPU cores idle, there is an opportunity to conduct on node in situ visualization with the idle CPU cores, something domain experts have up to this point been unable to do. By utilizing VTK-m, the visualization can be run on the CPU as this particular application demands, but also run on any architecture that is supported by VTK-m, enabling future re-use across different platforms.
{"title":"In Situ Visualization of Radiation Transport Geometry","authors":"Mark Kim, Tom J. Evans, S. Klasky, D. Pugmire","doi":"10.1145/3144769.3144770","DOIUrl":"https://doi.org/10.1145/3144769.3144770","url":null,"abstract":"The ultimate goal for radiation transport is to perform full-core reactor modelling and simulation. Advances in computational simulation bring this goal close to reality and the newest Monte Carlo transport codes have begun to shift to using accelerators that have become a stalwart in the supercomputing and HPC space. Within the reactor modelling and simulation community, Monte Carlo transport simulations are considered the gold standard for simulation. Through the use of \"combinatorial geometry\" (constructive solid geomtry), complex models can be used with fewer approximation compromises while at the same time scale to run on some of the largest supercomputers in the world. Unfortunately, the state-of-the-art for \"combinatorial geometry\" visualization is to decompose the geometry into a mesh. This approach could require a significant amount of memory which is antithetical to in situ visualization. To address this issue, we introduce a ray caster for visualizing combinatorial geometry in radiation transport code. By only using the accelerators for the radiation transport code and leaving the CPU cores idle, there is an opportunity to conduct on node in situ visualization with the idle CPU cores, something domain experts have up to this point been unable to do. By utilizing VTK-m, the visualization can be run on the CPU as this particular application demands, but also run on any architecture that is supported by VTK-m, enabling future re-use across different platforms.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"176 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124262954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Preeti Malakar, Christopher Knight, T. Munson, V. Vishwanath, M. Papka
Analysis of scientific simulation data enables scientists to glean insight from simulations. In situ analysis, which can be simultaneously executed with the simulation, mitigates I/O bottlenecks and can accelerate discovery of new phenomena. However, in typical modes of operation, this requires either stalling simulation during analysis phase or transferring data for analysis. We study the scalability challenges of time- and space-shared modes of analyzing large-scale molecular dynamics simulations. We also propose topology-aware mapping for simulation and analysis. We demonstrate the benefits of our approach using LAMMPS code on two supercomputers.
{"title":"Scalable In situ Analysis of Molecular Dynamics Simulations","authors":"Preeti Malakar, Christopher Knight, T. Munson, V. Vishwanath, M. Papka","doi":"10.1145/3144769.3144772","DOIUrl":"https://doi.org/10.1145/3144769.3144772","url":null,"abstract":"Analysis of scientific simulation data enables scientists to glean insight from simulations. In situ analysis, which can be simultaneously executed with the simulation, mitigates I/O bottlenecks and can accelerate discovery of new phenomena. However, in typical modes of operation, this requires either stalling simulation during analysis phase or transferring data for analysis. We study the scalability challenges of time- and space-shared modes of analyzing large-scale molecular dynamics simulations. We also propose topology-aware mapping for simulation and analysis. We demonstrate the benefits of our approach using LAMMPS code on two supercomputers.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134238707","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Matthew Larsen, J. Ahrens, Utkarsh Ayachit, E. Brugger, H. Childs, Berk Geveci, C. Harrison
This paper introduces ALPINE, a flyweight in situ infrastructure. The infrastructure is designed for leading-edge supercomputers, and has support for both distributed-memory and shared-memory parallelism. It can take advantage of computing power on both conventional CPU architectures and on many-core architectures such as NVIDIA GPUs or the Intel Xeon Phi. Further, it has a flexible design that supports for integration of new visualization and analysis routines and libraries. The paper describes ALPINE's interface choices and architecture, and also reports on initial experiments performed using the infrastructure.
{"title":"The ALPINE In Situ Infrastructure: Ascending from the Ashes of Strawman","authors":"Matthew Larsen, J. Ahrens, Utkarsh Ayachit, E. Brugger, H. Childs, Berk Geveci, C. Harrison","doi":"10.1145/3144769.3144778","DOIUrl":"https://doi.org/10.1145/3144769.3144778","url":null,"abstract":"This paper introduces ALPINE, a flyweight in situ infrastructure. The infrastructure is designed for leading-edge supercomputers, and has support for both distributed-memory and shared-memory parallelism. It can take advantage of computing power on both conventional CPU architectures and on many-core architectures such as NVIDIA GPUs or the Intel Xeon Phi. Further, it has a flexible design that supports for integration of new visualization and analysis routines and libraries. The paper describes ALPINE's interface choices and architecture, and also reports on initial experiments performed using the infrastructure.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123385829","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Max Zeyen, J. Ahrens, H. Hagen, K. Heitmann, S. Habib
In cosmological simulations, trillions of particles are handled and several terabytes of particle data are generated in each time step. Transferring this data directly from memory to disk in an uncompressed way results in a massive load on I/O and storage systems. Hence, one goal of domain scientists is to compress the data before storing it to disk while minimizing the loss of information. In this in situ scenario, the available time for the compression of one time step is limited. Therefore, the evaluation of compression techniques has shifted from only focusing on compression rates to including throughput and scalability. This study aims to evaluate and compare state-of-the-art compression techniques applied to particle data. For the investigated compression techniques, quantitative performance indicators such as compression rates, throughput, scalability, and reconstruction errors are measured. Based on these factors, this study offers a comprehensive analysis of the individual techniques and discusses their applicability for in situ compression. Based on this study, future challenges and directions in the compression of cosmological particle data are identified.
{"title":"Cosmological Particle Data Compression in Practice","authors":"Max Zeyen, J. Ahrens, H. Hagen, K. Heitmann, S. Habib","doi":"10.1145/3144769.3144776","DOIUrl":"https://doi.org/10.1145/3144769.3144776","url":null,"abstract":"In cosmological simulations, trillions of particles are handled and several terabytes of particle data are generated in each time step. Transferring this data directly from memory to disk in an uncompressed way results in a massive load on I/O and storage systems. Hence, one goal of domain scientists is to compress the data before storing it to disk while minimizing the loss of information. In this in situ scenario, the available time for the compression of one time step is limited. Therefore, the evaluation of compression techniques has shifted from only focusing on compression rates to including throughput and scalability. This study aims to evaluate and compare state-of-the-art compression techniques applied to particle data. For the investigated compression techniques, quantitative performance indicators such as compression rates, throughput, scalability, and reconstruction errors are measured. Based on these factors, this study offers a comprehensive analysis of the individual techniques and discusses their applicability for in situ compression. Based on this study, future challenges and directions in the compression of cosmological particle data are identified.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125446838","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Philippe P. Pébaÿ, G. Borghesi, H. Kolla, Janine Bennett, Sean Treichler
We present the current status of our work towards a scalable, asynchronous many-task, in situ statistical analysis engine using the Legion runtime system, expanding upon earlier work, that was limited to a prototype implementation with a proxy mini-application as a surrogate for a full-scale scientific simulation code. In contrast, we have more recently integrated our in situ analysis engines with S3D, a full-size scientific application, and conducted numerical tests therewith on the largest computational platform currently available for DOE science applications. The goal of this article is thus to describe the SPMD-Legion methodology we devised in this context, and compare the data aggregation technique deployed herein to the approach taken within our previous work.
{"title":"A Novel Shard-Based Approach for Asynchronous Many-Task Models for In Situ Analysis","authors":"Philippe P. Pébaÿ, G. Borghesi, H. Kolla, Janine Bennett, Sean Treichler","doi":"10.1145/3144769.3144775","DOIUrl":"https://doi.org/10.1145/3144769.3144775","url":null,"abstract":"We present the current status of our work towards a scalable, asynchronous many-task, in situ statistical analysis engine using the Legion runtime system, expanding upon earlier work, that was limited to a prototype implementation with a proxy mini-application as a surrogate for a full-scale scientific simulation code. In contrast, we have more recently integrated our in situ analysis engines with S3D, a full-size scientific application, and conducted numerical tests therewith on the largest computational platform currently available for DOE science applications. The goal of this article is thus to describe the SPMD-Legion methodology we devised in this context, and compare the data aggregation technique deployed herein to the approach taken within our previous work.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123609888","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In situ compression is a compromise between traditional post hoc and emerging in situ visualization and analysis. While the merits and limitations of various compressor options have been well studied, their performance impacts on scientific simulations are less clear, especially on large scale supercomputer systems. This study fills in this gap by performing in situ compression experiments on a leading supercomputer system. More specifically, we measured the computational and I/O impacts of a lossy wavelet compressor and analyzed the results with respect to various in situ processing concerns. We believe this study provides a better understanding of in situ compression as well as new evidence supporting its viability, in particular for wavelets.
{"title":"Performance Impacts of In Situ Wavelet Compression on Scientific Simulations","authors":"Shaomeng Li, Matthew Larsen, J. Clyne, H. Childs","doi":"10.1145/3144769.3144773","DOIUrl":"https://doi.org/10.1145/3144769.3144773","url":null,"abstract":"In situ compression is a compromise between traditional post hoc and emerging in situ visualization and analysis. While the merits and limitations of various compressor options have been well studied, their performance impacts on scientific simulations are less clear, especially on large scale supercomputer systems. This study fills in this gap by performing in situ compression experiments on a leading supercomputer system. More specifically, we measured the computational and I/O impacts of a lossy wavelet compressor and analyzed the results with respect to various in situ processing concerns. We believe this study provides a better understanding of in situ compression as well as new evidence supporting its viability, in particular for wavelets.","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"121 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127248632","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","authors":"","doi":"10.1145/3144769","DOIUrl":"https://doi.org/10.1145/3144769","url":null,"abstract":"","PeriodicalId":107517,"journal":{"name":"Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116372828","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：