AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

IF 3.3 3区 物理与天体物理 Q2 ASTRONOMY & ASTROPHYSICS Publications of the Astronomical Society of the Pacific Pub Date : 2024-07-11 DOI:10.1088/1538-3873/ad5b8a
Xiaoying Liu, Yingbo Liu, Lei Yang, Shichao Wu, Rong Jiang and Yongyuan Xiang
{"title":"AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression","authors":"Xiaoying Liu, Yingbo Liu, Lei Yang, Shichao Wu, Rong Jiang and Yongyuan Xiang","doi":"10.1088/1538-3873/ad5b8a","DOIUrl":null,"url":null,"abstract":"Effective data compression technology is essential for addressing data storage and transmission needs, especially given the escalating volume and complexity of data generated by contemporary astronomy. In this study, we propose utilizing deep learning-based lossless compression techniques to improve compression efficiency. We begin with a qualitative and quantitative analysis of the temporal and spatial redundancy in solar observation data. Based on this analysis, we introduce a novel deep learning-based framework called AstroDLLC for the lossless compression of astronomical solar images. AstroDLLC first segments high-resolution images into blocks to ensure that deep learning model training does not rely on high-computation power devices. It then addresses the non-normality of the partitioned data through simple reversible computational methods. Finally, it utilizes Bit-swap to train deep learning models that capture redundant features across multiple image frames, thereby enhancing compression efficiency. Comprehensive evaluations using data from the New Vacuum Solar Telescope reveal that AstroDLLC achieves a maximum compression ratio of 3.00 per image, surpassing Gzip, RICE, and other lossless technologies. The performance of AstroDLLC underscores its potential to address data compression challenges in astronomy.","PeriodicalId":20820,"journal":{"name":"Publications of the Astronomical Society of the Pacific","volume":"117 1","pages":""},"PeriodicalIF":3.3000,"publicationDate":"2024-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Publications of the Astronomical Society of the Pacific","FirstCategoryId":"101","ListUrlMain":"https://doi.org/10.1088/1538-3873/ad5b8a","RegionNum":3,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ASTRONOMY & ASTROPHYSICS","Score":null,"Total":0}
引用次数: 0

Abstract

Effective data compression technology is essential for addressing data storage and transmission needs, especially given the escalating volume and complexity of data generated by contemporary astronomy. In this study, we propose utilizing deep learning-based lossless compression techniques to improve compression efficiency. We begin with a qualitative and quantitative analysis of the temporal and spatial redundancy in solar observation data. Based on this analysis, we introduce a novel deep learning-based framework called AstroDLLC for the lossless compression of astronomical solar images. AstroDLLC first segments high-resolution images into blocks to ensure that deep learning model training does not rely on high-computation power devices. It then addresses the non-normality of the partitioned data through simple reversible computational methods. Finally, it utilizes Bit-swap to train deep learning models that capture redundant features across multiple image frames, thereby enhancing compression efficiency. Comprehensive evaluations using data from the New Vacuum Solar Telescope reveal that AstroDLLC achieves a maximum compression ratio of 3.00 per image, surpassing Gzip, RICE, and other lossless technologies. The performance of AstroDLLC underscores its potential to address data compression challenges in astronomy.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
AstroDLLC:通过基于深度学习的无损压缩,有效降低海量太阳观测数据的存储和传输成本
有效的数据压缩技术对于满足数据存储和传输需求至关重要,尤其是考虑到当代天文学产生的数据量和复杂性不断攀升。在本研究中,我们建议利用基于深度学习的无损压缩技术来提高压缩效率。我们首先对太阳观测数据的时空冗余进行了定性和定量分析。在这一分析的基础上,我们介绍了一种基于深度学习的新型框架,称为 AstroDLLC,用于无损压缩天文太阳图像。AstroDLLC 首先将高分辨率图像分割成块,以确保深度学习模型的训练不依赖于高计算能力的设备。然后,它通过简单的可逆计算方法解决了分割数据的非正态性问题。最后,它利用比特交换(Bit-swap)来训练深度学习模型,以捕捉多个图像帧的冗余特征,从而提高压缩效率。利用新真空太阳望远镜的数据进行的综合评估显示,AstroDLLC 每幅图像的最大压缩率达到 3.00,超过了 Gzip、RICE 和其他无损压缩技术。AstroDLLC 的性能凸显了它在解决天文学数据压缩难题方面的潜力。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Publications of the Astronomical Society of the Pacific
Publications of the Astronomical Society of the Pacific 地学天文-天文与天体物理
CiteScore
6.70
自引率
5.70%
发文量
103
审稿时长
4-8 weeks
期刊介绍: The Publications of the Astronomical Society of the Pacific (PASP), the technical journal of the Astronomical Society of the Pacific (ASP), has been published regularly since 1889, and is an integral part of the ASP''s mission to advance the science of astronomy and disseminate astronomical information. The journal provides an outlet for astronomical results of a scientific nature and serves to keep readers in touch with current astronomical research. It contains refereed research and instrumentation articles, invited and contributed reviews, tutorials, and dissertation summaries.
期刊最新文献
The Valuable Long-period Cluster Cepheid KQ Scorpii and other Calibration Candidates A New Parameterization for Finding Solutions for Microlensing Exoplanet Light Curves Multi-amplifier Sensing Charge-coupled Devices for Next Generation Spectroscopy Ejecta Masses in Type Ia Supernovae—Implications for the Progenitor and the Explosion Scenario* * Based in part on observations obtained with the Hobby-Eberly Telescope (HET), which is a joint project of the University of Texas at Austin, the Pennsylvania State University, Ludwig-Maximillians-Universitaet Muenchen, and Georg-August Universitaet Goettingen. The HET is named in honor of its principal benefactors, William P. Hobby and Robert E. Eberly. Physical Properties of Embedded Clusters in ATLASGAL Clumps with H ii Regions
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1