{"title":"Modular Data Analytics","authors":"Diomidis Spinellis","doi":"10.1109/ms.2024.3409988","DOIUrl":null,"url":null,"abstract":"Relational online analytical processing (ROLAP) often involves complex queries with many intermediate steps, which can be expensive to run on large datasets. Modularizing SQL queries improves readability, testability, and incremental execution. While SQL common table expressions (CTEs) and views help in modularization, they have availability, performance, or development limitations. To address these, simple-rolap, an open-source framework, automates dependency analysis and orchestrates the execution of modular queries using GNU make. It uses a main database containing the infrequently modified base data and a secondary database for caching intermediate results. Simple-rolap ensures efficient, maintainable ROLAP queries by splitting complex queries into simple ones that create intermediate tables or report results, which can be unit-tested. It uses the make tool to manage dependencies and execution order, enhancing workflow control, testing, visualization, and troubleshooting.","PeriodicalId":55018,"journal":{"name":"IEEE Software","volume":"373 1","pages":""},"PeriodicalIF":3.3000,"publicationDate":"2024-08-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Software","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1109/ms.2024.3409988","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}
引用次数: 0
Abstract
Relational online analytical processing (ROLAP) often involves complex queries with many intermediate steps, which can be expensive to run on large datasets. Modularizing SQL queries improves readability, testability, and incremental execution. While SQL common table expressions (CTEs) and views help in modularization, they have availability, performance, or development limitations. To address these, simple-rolap, an open-source framework, automates dependency analysis and orchestrates the execution of modular queries using GNU make. It uses a main database containing the infrequently modified base data and a secondary database for caching intermediate results. Simple-rolap ensures efficient, maintainable ROLAP queries by splitting complex queries into simple ones that create intermediate tables or report results, which can be unit-tested. It uses the make tool to manage dependencies and execution order, enhancing workflow control, testing, visualization, and troubleshooting.
关系型联机分析处理（ROLAP）通常涉及带有许多中间步骤的复杂查询，在大型数据集上运行时成本很高。将 SQL 查询模块化可提高可读性、可测试性和增量执行能力。虽然 SQL 共用表表达式（CTE）和视图有助于模块化，但它们在可用性、性能或开发方面有局限性。为了解决这些问题，开源框架 simple-rolap 使用 GNU make 自动进行依赖性分析并协调模块化查询的执行。它使用一个包含不常修改的基础数据的主数据库和一个用于缓存中间结果的辅助数据库。Simple-rolap 可将复杂的查询拆分成简单的查询，创建中间表或报告结果，并对其进行单元测试，从而确保 ROLAP 查询的高效性和可维护性。它使用 make 工具管理依赖关系和执行顺序，增强了工作流程控制、测试、可视化和故障排除能力。
期刊介绍：
IEEE Software delivers reliable, useful, leading-edge software development information to keep engineers and managers abreast of rapid technology change. Its mission is to build the community of leading software practitioners. The authority on translating software theory into practice, this magazine positions itself between pure research and pure practice, transferring ideas, methods, and experiences among researchers and engineers. Peerreviewed articles and columns by seasoned practitioners illuminate all aspects of the industry, including process improvement, project management, development tools, software maintenance, Web applications and opportunities, testing, and usability. The magazine''s readers specify, design, document, test, maintain, purchase, engineer, sell, teach, research, and manage the production of software or systems that include software. IEEE Software welcomes articles describing how software is developed in specific companies, laboratories, and university environments as well as articles describing new tools, current trends, and past projects'' limitations and failures as well as successes. Sample topics include geographically distributed development; software architectures; program and system debugging and testing; the education of software professionals; requirements, design, development, testing, and management methodologies; performance measurement and evaluation; standards; program and system reliability, security, and verification; programming environments; languages and language-related issues; Web-based development; usability; and software-related social and legal issues.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
