BenchCouncil Transactions on Benchmarks, Standards and Evaluations最新文献_第3页

Tensor databases empower AI for science: A case study on retrosynthetic analysis 张量数据库为科学赋予AI力量：一个关于反合成分析的案例研究

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2025-03-01 DOI: 10.1016/j.tbench.2025.100216

Xueya Zhang , Guoxin Kang , Boyang Xiao , Jianfeng Zhan

Retrosynthetic analysis is highly significant in chemistry, biology, and materials science, providing essential support for the rational design, synthesis, and optimization of compounds across diverse Artificial Intelligence for Science (AI4S) applications. Retrosynthetic analysis focuses on exploring pathways from products to reactants, and this is typically conducted using deep learning-based generative models. However, existing retrosynthetic analysis often overlooks how reaction conditions significantly impact chemical reactions. This causes existing work to lack unified models that can provide full-cycle services for retrosynthetic analysis, and also greatly limits the overall prediction accuracy of retrosynthetic analysis. These two issues cause users to depend on various independent models and tools, leading to high labor time and cost overhead.

To solve these issues, we define the boundary conditions of chemical reactions based on the Evaluatology theory and propose BigTensorDB, the first tensor database which integrates storage, prediction generation, search, and analysis functions. BigTensorDB designs the tensor schema for efficiently storing all the key information related to chemical reactions, including reaction conditions. BigTensorDB supports a full-cycle retrosynthetic analysis pipeline. It begins with predicting generation reaction paths, searching for approximate real reactions based on the tensor schema, and concludes with feasibility analysis, which enhances the interpretability of prediction results. BigTensorDB can effectively reduce usage costs and improve efficiency for users during the full-cycle retrosynthetic analysis process. Meanwhile, it provides a potential solution to the low accuracy issue, encouraging researchers to focus on improving full-cycle accuracy.

反合成分析在化学、生物学和材料科学中具有重要意义，为各种人工智能科学（AI4S）应用中化合物的合理设计、合成和优化提供重要支持。反合成分析侧重于探索从产物到反应物的途径，这通常使用基于深度学习的生成模型进行。然而，现有的反合成分析往往忽略了反应条件对化学反应的重要影响。这导致现有工作缺乏能够为逆合成分析提供全周期服务的统一模型，也极大地限制了逆合成分析的整体预测精度。这两个问题导致用户依赖于各种独立的模型和工具，从而导致较高的劳动时间和成本开销。为了解决这些问题，我们基于Evaluatology理论定义了化学反应的边界条件，并提出了首个集存储、预测生成、搜索和分析功能于一体的张量数据库BigTensorDB。BigTensorDB设计了张量模式，用于高效存储与化学反应相关的所有关键信息，包括反应条件。BigTensorDB支持全周期的反合成分析管道。从预测生成反应路径开始，基于张量模式寻找近似真实反应，最后进行可行性分析，增强了预测结果的可解释性。BigTensorDB可以有效降低用户在全周期反合成分析过程中的使用成本，提高效率。同时，它为低精度问题提供了一个潜在的解决方案，鼓励研究人员将重点放在提高全周期精度上。

{"title":"Tensor databases empower AI for science: A case study on retrosynthetic analysis","authors":"Xueya Zhang , Guoxin Kang , Boyang Xiao , Jianfeng Zhan","doi":"10.1016/j.tbench.2025.100216","DOIUrl":"10.1016/j.tbench.2025.100216","url":null,"abstract":"<div><div>Retrosynthetic analysis is highly significant in chemistry, biology, and materials science, providing essential support for the rational design, synthesis, and optimization of compounds across diverse Artificial Intelligence for Science (AI4S) applications. Retrosynthetic analysis focuses on exploring pathways from products to reactants, and this is typically conducted using deep learning-based generative models. However, existing retrosynthetic analysis often overlooks how reaction conditions significantly impact chemical reactions. This causes existing work to lack unified models that can provide full-cycle services for retrosynthetic analysis, and also greatly limits the overall prediction accuracy of retrosynthetic analysis. These two issues cause users to depend on various independent models and tools, leading to high labor time and cost overhead.</div><div>To solve these issues, we define the boundary conditions of chemical reactions based on the Evaluatology theory and propose BigTensorDB, the first tensor database which integrates storage, prediction generation, search, and analysis functions. BigTensorDB designs the tensor schema for efficiently storing all the key information related to chemical reactions, including reaction conditions. BigTensorDB supports a full-cycle retrosynthetic analysis pipeline. It begins with predicting generation reaction paths, searching for approximate real reactions based on the tensor schema, and concludes with feasibility analysis, which enhances the interpretability of prediction results. BigTensorDB can effectively reduce usage costs and improve efficiency for users during the full-cycle retrosynthetic analysis process. Meanwhile, it provides a potential solution to the low accuracy issue, encouraging researchers to focus on improving full-cycle accuracy.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"5 1","pages":"Article 100216"},"PeriodicalIF":0.0,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144307785","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluatology’s perspective on AI evaluation in critical scenarios: From tail quality to landscape 评估学在关键场景下的人工智能评估视角：从尾部质量到景观

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2025-03-01 DOI: 10.1016/j.tbench.2025.100203

Zhengxin Yang

Tail Quality, as a metric for evaluating AI inference performance in critical scenarios, reveals the extreme behaviors of AI inference systems in real-world applications, offering significant practical value. However, its adoption has been limited due to the lack of systematic theoretical support. To address this issue, this paper analyzes AI inference system evaluation activities from the perspective of Evaluatology, bridging the gap between theory and practice. Specifically, we begin by constructing a rigorous, consistent, and comprehensive evaluation system for AI inference systems, with a focus on defining the evaluation subject and evaluation conditions. We then refine the Quality@Time-Threshold (Q@T) statistical evaluation framework by formalizing these components, thereby enhancing its theoretical rigor and applicability. By integrating the principles of Evaluatology, we extend Q@T to incorporate stakeholder considerations, ensuring its adaptability to varying time tolerance. Through refining the Q@T evaluation framework and embedding it within Evaluatology, we provide a robust theoretical foundation that enhances the accuracy and reliability of AI system evaluations, making the approach both scientifically rigorous and practically reliable. Experimental results further validate the effectiveness of this refined framework, confirming its scientific rigor and practical applicability. The theoretical analysis presented in this paper provides valuable guidance for researchers aiming to apply Evaluatology in practice.

尾质量作为评估人工智能在关键场景下推理性能的指标，揭示了人工智能推理系统在现实应用中的极端行为，具有重要的实用价值。然而，由于缺乏系统的理论支持，其采用受到了限制。针对这一问题，本文从评价学的角度分析了人工智能推理系统的评价活动，弥合了理论与实践之间的差距。具体而言，我们首先构建了一个严谨、一致、全面的AI推理系统评估体系，重点是定义评估主体和评估条件。然后，我们通过形式化这些组件来完善Quality@Time-Threshold （Q@T）统计评估框架，从而增强其理论严谨性和适用性。通过整合Evaluatology的原则，我们扩展Q@T以纳入利益相关者的考虑，确保其对不同时间公差的适应性。通过完善Q@T评估框架并将其嵌入到Evaluatology中，我们提供了一个强大的理论基础，提高了人工智能系统评估的准确性和可靠性，使该方法在科学上严谨，在实践中可靠。实验结果进一步验证了该改进框架的有效性，验证了其科学严谨性和实际适用性。本文的理论分析对旨在将评价学应用于实践的研究人员具有重要的指导意义。

{"title":"Evaluatology’s perspective on AI evaluation in critical scenarios: From tail quality to landscape","authors":"Zhengxin Yang","doi":"10.1016/j.tbench.2025.100203","DOIUrl":"10.1016/j.tbench.2025.100203","url":null,"abstract":"<div><div>Tail Quality, as a metric for evaluating AI inference performance in critical scenarios, reveals the extreme behaviors of AI inference systems in real-world applications, offering significant practical value. However, its adoption has been limited due to the lack of systematic theoretical support. To address this issue, this paper analyzes AI inference system evaluation activities from the perspective of Evaluatology, bridging the gap between theory and practice. Specifically, we begin by constructing a rigorous, consistent, and comprehensive evaluation system for AI inference systems, with a focus on defining the evaluation subject and evaluation conditions. We then refine the Quality@Time-Threshold (Q@T) statistical evaluation framework by formalizing these components, thereby enhancing its theoretical rigor and applicability. By integrating the principles of Evaluatology, we extend Q@T to incorporate stakeholder considerations, ensuring its adaptability to varying time tolerance. Through refining the Q@T evaluation framework and embedding it within Evaluatology, we provide a robust theoretical foundation that enhances the accuracy and reliability of AI system evaluations, making the approach both scientifically rigorous and practically reliable. Experimental results further validate the effectiveness of this refined framework, confirming its scientific rigor and practical applicability. The theoretical analysis presented in this paper provides valuable guidance for researchers aiming to apply Evaluatology in practice.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"5 1","pages":"Article 100203"},"PeriodicalIF":0.0,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144168926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Ethical and regulatory challenges in machine learning-based healthcare systems: A review of implementation barriers and future directions 基于机器学习的医疗保健系统中的伦理和监管挑战：对实施障碍和未来方向的回顾

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2025-03-01 DOI: 10.1016/j.tbench.2025.100215

Shehu Mohammed, Neha Malhotra

Machine learning significantly enhances clinical decision-making quality, directly impacting patient care with early diagnosis, personalized treatment, and predictive analytics. Nonetheless, the increasing proliferation of such ML applications in practice raises potential ethical and regulatory obstacles that may prevent their widespread adoption in healthcare. Key issues concern patient data privacy, algorithmic bias, absence of transparency, and ambiguous legal liability. Fortunately, regulations like the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), and the FDA AI/ML guidance have raised important ways of addressing things like fairness, explainability, legal compliance, etc.; however, the landscape is far from risk-free. AI liability is another one of the gray areas approaching black, worrying about who is liable for an AI medical error — the developers, the physicians, or the institutions. The study reviews ethical risks and potential opportunities, as well as regulatory frameworks and emerging challenges in AI-driven healthcare. It proposes solutions to reduce bias, improve transparency, and enhance legal accountability. This research addresses these challenges to support the safe, fair, and effective deployment of ML-based systems in clinical practice, guaranteeing that patients can trust, regulators can approve, and healthcare can use them.

机器学习显著提高了临床决策质量，通过早期诊断、个性化治疗和预测分析直接影响患者护理。尽管如此，在实践中，这种ML应用程序的日益普及引发了潜在的伦理和监管障碍，可能会阻止它们在医疗保健领域的广泛采用。关键问题涉及患者数据隐私、算法偏见、缺乏透明度和模糊的法律责任。幸运的是，《通用数据保护条例》（GDPR）、《健康保险可移植性和责任法案》（HIPAA）和FDA人工智能/机器学习指南等法规提出了解决公平、可解释性、法律合规性等问题的重要方法；然而，前景远非没有风险。人工智能的责任是另一个接近黑色的灰色地带，人们担心谁应该为人工智能的医疗错误负责——是开发人员、医生还是机构。该研究回顾了人工智能驱动的医疗保健领域的伦理风险和潜在机遇，以及监管框架和新出现的挑战。报告提出了减少偏见、提高透明度和加强法律问责制的解决方案。本研究解决了这些挑战，以支持在临床实践中安全、公平和有效地部署基于ml的系统，确保患者可以信任，监管机构可以批准，医疗保健可以使用它们。

{"title":"Ethical and regulatory challenges in machine learning-based healthcare systems: A review of implementation barriers and future directions","authors":"Shehu Mohammed, Neha Malhotra","doi":"10.1016/j.tbench.2025.100215","DOIUrl":"10.1016/j.tbench.2025.100215","url":null,"abstract":"<div><div>Machine learning significantly enhances clinical decision-making quality, directly impacting patient care with early diagnosis, personalized treatment, and predictive analytics. Nonetheless, the increasing proliferation of such ML applications in practice raises potential ethical and regulatory obstacles that may prevent their widespread adoption in healthcare. Key issues concern patient data privacy, algorithmic bias, absence of transparency, and ambiguous legal liability. Fortunately, regulations like the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), and the FDA AI/ML guidance have raised important ways of addressing things like fairness, explainability, legal compliance, etc.; however, the landscape is far from risk-free. AI liability is another one of the gray areas approaching black, worrying about who is liable for an AI medical error — the developers, the physicians, or the institutions. The study reviews ethical risks and potential opportunities, as well as regulatory frameworks and emerging challenges in AI-driven healthcare. It proposes solutions to reduce bias, improve transparency, and enhance legal accountability. This research addresses these challenges to support the safe, fair, and effective deployment of ML-based systems in clinical practice, guaranteeing that patients can trust, regulators can approve, and healthcare can use them.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"5 1","pages":"Article 100215"},"PeriodicalIF":0.0,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144271090","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

AICB: A benchmark for evaluating the communication subsystem of LLM training clusters AICB：一个评估LLM训练集群通信子系统的基准

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2025-03-01 DOI: 10.1016/j.tbench.2025.100212

Xinyue Li, Heyang Zhou, Qingxu Li, Sen Zhang, Gang Lu

AICB (Artificial Intelligence Communication Benchmark) is a benchmark for evaluating the communication subsystem of GPU clusters, which includes representative workloads in the fields of Large Language Model (LLM) training. Guided by the theories and methodologies of Evaluatology, we simplified the real-workload LLM training systems through AICB that maintain good representativeness and usability. AICB bridges the gap between application benchmarks and microbenchmarks in the scope of LLM training. In addition, we constructed a new GPU-free evaluation system that helps researchers evaluate the communication system of the LLM training systems. To help the urgent demand on this evaluation subject, we open-source AICB and make it available at https://github.com/aliyun/aicb.

AICB （Artificial Intelligence Communication Benchmark）是评估GPU集群通信子系统性能的基准，包含了大型语言模型（LLM）训练领域的代表性工作负载。在评估学理论和方法的指导下，我们通过AICB简化了实际工作量的法学硕士培训系统，保持了良好的代表性和可用性。AICB在LLM培训范围内弥合了应用基准和微基准之间的差距。此外，我们构建了一个新的无gpu评估系统，帮助研究人员对LLM培训系统的通信系统进行评估。为了满足对这一评估主题的迫切需求，我们将AICB开源，并在https://github.com/aliyun/aicb上提供。

引用次数: 0

Predicting the number of call center incoming calls using deep learning 使用深度学习预测呼叫中心呼入的数量

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2025-03-01 DOI: 10.1016/j.tbench.2025.100213

Armaghan Nikfar , Javad Mohammadzadeh

One of the main problems in call centers is the call queue. This can lead to long waiting times for customers, increased frustration and call abandonment. The important role that predictive analytics plays in optimizing call center operations is increasingly recognized. Advanced models can be trained by training datasets such as the number of calls that have occurred throughout history, and by estimating how religious and public holidays have affected the weight of hours and the number of calls, and this study provides an analysis of 4 years. Call center data from Shatel, an Internet service provider. Predictive deep learning models, specifically the Bidirectional Short-Term Memory Model (BLSTM), were used to predict the number of incoming calls, predict the number of calls to centers, and prevent call queues with an accuracy of 90.56.

呼叫中心的主要问题之一是呼叫队列。这可能会导致客户等待时间过长，增加挫折感和放弃电话。预测分析在优化呼叫中心运营中的重要作用日益得到认可。高级模型可以通过训练数据集进行训练，例如历史上发生的电话数量，以及通过估计宗教和公共假日如何影响小时数和电话数量的权重，本研究提供了4年的分析。来自互联网服务提供商Shatel的呼叫中心数据。预测深度学习模型，特别是双向短期记忆模型（Bidirectional short - Memory Model， BLSTM），被用于预测呼入数量，预测呼叫中心的数量，并防止呼叫队列，准确率为90.56。

引用次数: 0

Open Source Evaluatology: An evaluation framework and methodology for open source ecosystems based on evaluatology 开源评估学：基于评估学的开源生态系统评估框架和方法

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2024-12-01 DOI: 10.1016/j.tbench.2025.100190

Fanyu Han , Shengyu Zhao , Wei Wang , Aoying Zhou , Weining Qian , Xuan Zhou , Jiaheng Peng , Lan You , Yang Chen , Xiaoya Xia , Yenan Tang , Liyun Yang , Chunqi Tian

The open-source ecosystem, as an important component of the modern software industry, has increasingly attracted attention from both academia and industry regarding its evaluation. However, current open-source evaluation methods face several issues, such as inconsistent evaluation standards, lack of theoretical support in the evaluation process, and poor comparability of evaluation results. Guided by the foundational theories of evaluatology, this paper proposes a new interdisciplinary research field, Open Source Evaluatology, and constructs an evaluation theoretical framework and methodology for open-source ecosystems. The main contributions of this paper include: (1) Based on the five axioms of evaluation theory, a theoretical system for Open Source Evaluatology is developed, and the basic concepts, evaluation dimensions, and evaluation standards for the open-source ecosystem are proposed; (2) An evaluation conditions (EC) framework is designed, encompassing five levels: problem definition, task instances, algorithm mechanisms, implementation examples, and supporting systems. A combined evaluation model (EM) based on statistical metrics and network metrics is also introduced; (3) Experimental validation using the GitHub dataset shows that the proposed evaluation framework effectively assesses various features of open-source projects, developers, and communities, and has been verified in multiple practical application scenarios. The research demonstrates that Open Source Evaluatology provides a standardized theoretical guide and methodological support for open-source ecosystem evaluation, which can be widely applied in various scenarios, such as open-source project selection, developer evaluation, and community management, and plays a significant role in promoting the healthy and sustainable development of open-source ecosystems.

开源生态系统作为现代软件产业的重要组成部分，其评价日益受到学术界和业界的关注。然而，目前的开源评价方法面临着评价标准不统一、评价过程缺乏理论支持、评价结果可比性差等问题。在评估学基础理论的指导下，提出了一个新的跨学科研究领域——开源评估学，构建了开源生态系统评估的理论框架和方法。本文的主要贡献有：(1)基于评估理论的五大公理，构建了开源评估学的理论体系，提出了开源生态系统的基本概念、评估维度和评估标准；(2)设计了评估条件（EC）框架，包括问题定义、任务实例、算法机制、实现示例和支持系统五个层次。介绍了一种基于统计度量和网络度量的综合评价模型；(3)基于GitHub数据集的实验验证表明，本文提出的评估框架有效地评估了开源项目、开发者和社区的各种特征，并在多个实际应用场景中得到了验证。研究表明，开源评估学为开源生态系统评估提供了标准化的理论指导和方法支持，可广泛应用于开源项目选择、开发者评估、社区管理等多种场景，对促进开源生态系统健康可持续发展具有重要作用。

{"title":"Open Source Evaluatology: An evaluation framework and methodology for open source ecosystems based on evaluatology","authors":"Fanyu Han , Shengyu Zhao , Wei Wang , Aoying Zhou , Weining Qian , Xuan Zhou , Jiaheng Peng , Lan You , Yang Chen , Xiaoya Xia , Yenan Tang , Liyun Yang , Chunqi Tian","doi":"10.1016/j.tbench.2025.100190","DOIUrl":"10.1016/j.tbench.2025.100190","url":null,"abstract":"<div><div>The open-source ecosystem, as an important component of the modern software industry, has increasingly attracted attention from both academia and industry regarding its evaluation. However, current open-source evaluation methods face several issues, such as inconsistent evaluation standards, lack of theoretical support in the evaluation process, and poor comparability of evaluation results. Guided by the foundational theories of evaluatology, this paper proposes a new interdisciplinary research field, Open Source Evaluatology, and constructs an evaluation theoretical framework and methodology for open-source ecosystems. The main contributions of this paper include: (1) Based on the five axioms of evaluation theory, a theoretical system for Open Source Evaluatology is developed, and the basic concepts, evaluation dimensions, and evaluation standards for the open-source ecosystem are proposed; (2) An evaluation conditions (EC) framework is designed, encompassing five levels: problem definition, task instances, algorithm mechanisms, implementation examples, and supporting systems. A combined evaluation model (EM) based on statistical metrics and network metrics is also introduced; (3) Experimental validation using the GitHub dataset shows that the proposed evaluation framework effectively assesses various features of open-source projects, developers, and communities, and has been verified in multiple practical application scenarios. The research demonstrates that Open Source Evaluatology provides a standardized theoretical guide and methodological support for open-source ecosystem evaluation, which can be widely applied in various scenarios, such as open-source project selection, developer evaluation, and community management, and plays a significant role in promoting the healthy and sustainable development of open-source ecosystems.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"4 4","pages":"Article 100190"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145871577","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

COADBench: A benchmark for revealing the relationship between AI models and clinical outcomes COADBench：揭示人工智能模型与临床结果之间关系的基准

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2024-12-01 DOI: 10.1016/j.tbench.2025.100198

Jiyue Xie , Wenjing Liu , Li Ma , Caiqin Yao , Qi Liang , Suqin Tang , Yunyou Huang

Alzheimer’s disease (AD), due to its irreversible nature and the severe social burden it causes, has garnered significant attention from AI researchers. Numerous auxiliary diagnostic models have been developed with the aim of improving AD diagnostic services and thereby reducing the social burden. However, due to a lack of validation regarding the clinical value of these models, no AD diagnostic model has been widely accepted by clinicians or officially approved for use in enhancing AD diagnostic services. The clinical value of traditional medical devices is validated through rigorous randomized controlled trials to prove their impact on clinical outcomes. In contrast, current AD diagnostic models are only validated based on their accuracy, and the relationship between these models and patient outcomes remains unknown. This gap has hindered the acceptance and clinical use of AD diagnostic models by healthcare professionals. To address this issue, we introduce the COADBench, a benchmark centered on clinical outcomes for evaluating the clinical value of AD diagnostic models. COADBench curated subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database who have at least two cognitive score records (the most commonly used clinical endpoint in AD clinical trials) from different follow-up visits. To the best of our knowledge, for the first time, it links the cognitive scores of subjects with model performance, using patient cognitive scores as clinical outcomes after intervention to evaluate the models. Through the benchmarking of current mainstream AD diagnostic algorithms using COADBench, we find that there was no significant correlation between the subjects’ cognitive improvement and the model’s performance, which means that the current performance evaluation criteria of mainstream AD diagnostic algorithms are not combined with clinical value.

阿尔茨海默病（AD）因其不可逆性和严重的社会负担，引起了人工智能研究人员的极大关注。为了改善阿尔茨海默病的诊断服务，从而减轻社会负担，已经开发了许多辅助诊断模型。然而，由于缺乏对这些模型的临床价值的验证，没有一个AD诊断模型被临床医生广泛接受或正式批准用于增强AD诊断服务。传统医疗器械的临床价值是通过严格的随机对照试验来验证的，以证明它们对临床结果的影响。相比之下，目前的AD诊断模型仅基于其准确性进行验证，这些模型与患者预后之间的关系尚不清楚。这一差距阻碍了医疗保健专业人员对AD诊断模型的接受和临床使用。为了解决这个问题，我们引入了COADBench，这是一个以临床结果为中心的基准，用于评估AD诊断模型的临床价值。COADBench从阿尔茨海默病神经影像学倡议（ADNI）数据库中挑选了至少有两个不同随访的认知评分记录（阿尔茨海默病临床试验中最常用的临床终点）的受试者。据我们所知，这是第一次将受试者的认知得分与模型性能联系起来，使用患者的认知得分作为干预后评估模型的临床结果。通过使用COADBench对当前主流AD诊断算法进行对标，我们发现被试的认知改善与模型的性能之间没有显著的相关性，这意味着目前主流AD诊断算法的性能评价标准没有与临床价值相结合。

{"title":"COADBench: A benchmark for revealing the relationship between AI models and clinical outcomes","authors":"Jiyue Xie , Wenjing Liu , Li Ma , Caiqin Yao , Qi Liang , Suqin Tang , Yunyou Huang","doi":"10.1016/j.tbench.2025.100198","DOIUrl":"10.1016/j.tbench.2025.100198","url":null,"abstract":"<div><div>Alzheimer’s disease (AD), due to its irreversible nature and the severe social burden it causes, has garnered significant attention from AI researchers. Numerous auxiliary diagnostic models have been developed with the aim of improving AD diagnostic services and thereby reducing the social burden. However, due to a lack of validation regarding the clinical value of these models, no AD diagnostic model has been widely accepted by clinicians or officially approved for use in enhancing AD diagnostic services. The clinical value of traditional medical devices is validated through rigorous randomized controlled trials to prove their impact on clinical outcomes. In contrast, current AD diagnostic models are only validated based on their accuracy, and the relationship between these models and patient outcomes remains unknown. This gap has hindered the acceptance and clinical use of AD diagnostic models by healthcare professionals. To address this issue, we introduce the COADBench, a benchmark centered on clinical outcomes for evaluating the clinical value of AD diagnostic models. COADBench curated subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database who have at least two cognitive score records (the most commonly used clinical endpoint in AD clinical trials) from different follow-up visits. To the best of our knowledge, for the first time, it links the cognitive scores of subjects with model performance, using patient cognitive scores as clinical outcomes after intervention to evaluate the models. Through the benchmarking of current mainstream AD diagnostic algorithms using COADBench, we find that there was no significant correlation between the subjects’ cognitive improvement and the model’s performance, which means that the current performance evaluation criteria of mainstream AD diagnostic algorithms are not combined with clinical value.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"4 4","pages":"Article 100198"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145871523","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating long-term usage patterns of open source datasets: A citation network approach 评估开源数据集的长期使用模式：引文网络方法

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2024-12-01 DOI: 10.1016/j.tbench.2025.100199

Jiaheng Peng, Fanyu Han, Wei Wang

The evaluation of datasets serves as a fundamental basis for tasks in evaluatology. Evaluating the usage patterns of datasets has a significant impact on the selection of appropriate datasets. Many renowned Open Source datasets are well-established and have not been updated for many years, yet they continue to be widely used by a large number of researchers. Due to this characteristic, conventional Open Source metrics (e.g., number of stars, issues, and activity) are insufficient for evaluating the long-term usage patterns based on log activity data from their GitHub repositories.

Researchers often encounter significant challenges in selecting appropriate datasets due to the lack of insight into how these datasets are being utilized. To address this challenge, this paper proposes establishing a connection between Open Source datasets and the citation networks of their corresponding academic papers. By mining the citation network of the corresponding academic paper, we can obtain rich graph-structured information, such as citation times, authors, and more. Utilizing this information, we can evaluate the long-term usage patterns of the associated Open Source dataset.

Furthermore, this paper conducts extensive experiments based on five major dataset categories (Texts, Images, Videos, Audio, Medical) to demonstrate that the proposed method effectively evaluates the long-term usage patterns of Open Source datasets. Additionally, the insights gained from the experimental results can serve as a valuable reference for future researchers in selecting appropriate datasets for their work.

数据集的评估是评估任务的基本基础。评估数据集的使用模式对选择合适的数据集具有重要影响。许多著名的开源数据集已经建立良好，并且多年没有更新，但它们仍然被大量研究人员广泛使用。由于这个特点，传统的开源指标（例如，星星数量、问题和活动）不足以评估基于GitHub存储库的日志活动数据的长期使用模式。由于缺乏对这些数据集如何被利用的了解，研究人员在选择适当的数据集时经常遇到重大挑战。为了应对这一挑战，本文建议在开源数据集与其相应学术论文的引文网络之间建立连接。通过挖掘相应学术论文的引文网络，我们可以获得丰富的图结构信息，如引文次数、作者等。利用这些信息，我们可以评估相关开源数据集的长期使用模式。此外，本文基于五个主要数据集类别（文本、图像、视频、音频、医疗）进行了大量实验，以证明所提出的方法有效地评估了开源数据集的长期使用模式。此外，从实验结果中获得的见解可以为未来的研究人员选择合适的数据集提供有价值的参考。

{"title":"Evaluating long-term usage patterns of open source datasets: A citation network approach","authors":"Jiaheng Peng, Fanyu Han, Wei Wang","doi":"10.1016/j.tbench.2025.100199","DOIUrl":"10.1016/j.tbench.2025.100199","url":null,"abstract":"<div><div>The evaluation of datasets serves as a fundamental basis for tasks in evaluatology. Evaluating the usage patterns of datasets has a significant impact on the selection of appropriate datasets. Many renowned Open Source datasets are well-established and have not been updated for many years, yet they continue to be widely used by a large number of researchers. Due to this characteristic, conventional Open Source metrics (e.g., number of stars, issues, and activity) are insufficient for evaluating the long-term usage patterns based on log activity data from their GitHub repositories.</div><div>Researchers often encounter significant challenges in selecting appropriate datasets due to the lack of insight into how these datasets are being utilized. To address this challenge, this paper proposes establishing a connection between Open Source datasets and the citation networks of their corresponding academic papers. By mining the citation network of the corresponding academic paper, we can obtain rich graph-structured information, such as citation times, authors, and more. Utilizing this information, we can evaluate the long-term usage patterns of the associated Open Source dataset.</div><div>Furthermore, this paper conducts extensive experiments based on five major dataset categories (Texts, Images, Videos, Audio, Medical) to demonstrate that the proposed method effectively evaluates the long-term usage patterns of Open Source datasets. Additionally, the insights gained from the experimental results can serve as a valuable reference for future researchers in selecting appropriate datasets for their work.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"4 4","pages":"Article 100199"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145871525","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Patrick Star: A comprehensive benchmark for multi-modal image editing Patrick Star：多模态图像编辑的综合基准

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2024-12-01 DOI: 10.1016/j.tbench.2025.100201

Di Cheng , ZhengXin Yang , ChunJie Luo , Chen Zheng , YingJie Shi

Generative image editing enhances and automates traditional image designing methods. However, there is a significant imbalance in existing research, where the development of sketch-guided and example-guided image editing has not been sufficiently explored compared to text-guided image editing, despite the former being equally important in real-world applications. The leading cause of this phenomenon is the severe lack of corresponding benchmark datasets. To address this issue, this paper proposes a comprehensive and unified benchmark dataset, Patrick Star, which consists of approximately 500 test images, to promote balanced development in this field across multi-task and multi-modal settings. First, theoretical analysis grounded in Evaluatology highlights the importance of establishing a balanced benchmark dataset to advance research in image editing. Building on this theoretical foundation, the dataset’s construction methodology is explained in detail, ensuring it addresses critical gaps in existing studies. Next, statistical analyses are conducted to verify the dataset’s usability and diversity. Finally, comparative experiments underscore the dataset’s potential as a comprehensive benchmark, demonstrating its capacity to support balanced development in image editing.

生成图像编辑是对传统图像设计方法的改进和自动化。然而，在现有的研究中存在着明显的不平衡，尽管草图引导和示例引导图像编辑在现实应用中同样重要，但与文本引导图像编辑相比，草图引导和示例引导图像编辑的发展尚未得到充分的探索。造成这种现象的主要原因是严重缺乏相应的基准数据集。为了解决这一问题，本文提出了一个全面统一的基准数据集Patrick Star，该数据集由大约500个测试图像组成，以促进该领域在多任务和多模式设置下的平衡发展。首先，基于评估学的理论分析强调了建立平衡的基准数据集对推进图像编辑研究的重要性。在此理论基础上，详细解释了数据集的构建方法，确保它解决了现有研究中的关键空白。然后进行统计分析，验证数据集的可用性和多样性。最后，对比实验强调了数据集作为综合基准的潜力，展示了其支持图像编辑平衡发展的能力。

{"title":"Patrick Star: A comprehensive benchmark for multi-modal image editing","authors":"Di Cheng , ZhengXin Yang , ChunJie Luo , Chen Zheng , YingJie Shi","doi":"10.1016/j.tbench.2025.100201","DOIUrl":"10.1016/j.tbench.2025.100201","url":null,"abstract":"<div><div>Generative image editing enhances and automates traditional image designing methods. However, there is a significant imbalance in existing research, where the development of sketch-guided and example-guided image editing has not been sufficiently explored compared to text-guided image editing, despite the former being equally important in real-world applications. The leading cause of this phenomenon is the severe lack of corresponding benchmark datasets. To address this issue, this paper proposes a comprehensive and unified benchmark dataset, Patrick Star, which consists of approximately 500 test images, to promote balanced development in this field across multi-task and multi-modal settings. First, theoretical analysis grounded in Evaluatology highlights the importance of establishing a balanced benchmark dataset to advance research in image editing. Building on this theoretical foundation, the dataset’s construction methodology is explained in detail, ensuring it addresses critical gaps in existing studies. Next, statistical analyses are conducted to verify the dataset’s usability and diversity. Finally, comparative experiments underscore the dataset’s potential as a comprehensive benchmark, demonstrating its capacity to support balanced development in image editing.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"4 4","pages":"Article 100201"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145871576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Advanced Deep Learning Models for Improving Movie Rating Predictions: A Benchmarking Study 用于提高电影评分预测的高级深度学习模型：基准研究

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Pub Date : 2024-12-01 DOI: 10.1016/j.tbench.2025.100200

Manisha Valera , Dr. Rahul Mehta

Predicting movie ratings very precisely has become a vital aspect of personalized recommendation systems, which requires robust and high-performing models. for evaluating the effectiveness in predicting movie ratings, this study conducts a comprehensive performance analysis of various deep learning architectures, which includes BiLSTM, CNN + LSTM, CNN + GRU, CNN + Attention, CNN, VAE, Simple RNN, GRU + Attention, Transformer Encoder, FNN and ResNet. Here each model’s performance is evaluated on movie reviews’ dataset, enhanced with sentiment scores and user ratings, by using a range of evaluation metrics: Mean Absolute Error (MAE), R² score, Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Explained Variance. Here the results highlight distinct strengths and weaknesses among the models, in which VAE model consistently delivering superior accuracy, whereas attention-based models prove prominent improvements in interpretability and generalization. This analysis offers important insights into choosing models for movie recommendation systems, which also highlights the balance between prediction accuracy and computational efficiency. The discoveries from this study serve as a benchmark for future developments in movie rating prediction, supporting the researchers and practitioners in augmenting recommendation system performance.

非常精确地预测电影评分已经成为个性化推荐系统的一个重要方面，这需要强大和高性能的模型。为了评估预测电影评分的有效性，本研究对各种深度学习架构进行了全面的性能分析，包括BiLSTM、CNN + LSTM、CNN + GRU、CNN + Attention、CNN、VAE、Simple RNN、GRU + Attention、Transformer Encoder、FNN和ResNet。在这里，每个模型的表现都是在电影评论的数据集上进行评估的，通过使用一系列评估指标：平均绝对误差（MAE）、R²分数、均方误差（MSE）、均方根误差（RMSE）和解释方差（Explained Variance），用情感分数和用户评级来增强。这里的结果突出了模型之间不同的优点和缺点，其中VAE模型始终提供优越的准确性，而基于注意力的模型在可解释性和泛化方面证明了显著的改进。这一分析为电影推荐系统选择模型提供了重要的见解，也强调了预测精度和计算效率之间的平衡。本研究的发现可以作为电影评分预测未来发展的基准，支持研究人员和从业者增强推荐系统的性能。

{"title":"Advanced Deep Learning Models for Improving Movie Rating Predictions: A Benchmarking Study","authors":"Manisha Valera , Dr. Rahul Mehta","doi":"10.1016/j.tbench.2025.100200","DOIUrl":"10.1016/j.tbench.2025.100200","url":null,"abstract":"<div><div>Predicting movie ratings very precisely has become a vital aspect of personalized recommendation systems, which requires robust and high-performing models. for evaluating the effectiveness in predicting movie ratings, this study conducts a comprehensive performance analysis of various deep learning architectures, which includes BiLSTM, CNN + LSTM, CNN + GRU, CNN + Attention, CNN, VAE, Simple RNN, GRU + Attention, Transformer Encoder, FNN and ResNet. Here each model’s performance is evaluated on movie reviews’ dataset, enhanced with sentiment scores and user ratings, by using a range of evaluation metrics: Mean Absolute Error (MAE), R² score, Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Explained Variance. Here the results highlight distinct strengths and weaknesses among the models, in which VAE model consistently delivering superior accuracy, whereas attention-based models prove prominent improvements in interpretability and generalization. This analysis offers important insights into choosing models for movie recommendation systems, which also highlights the balance between prediction accuracy and computational efficiency. The discoveries from this study serve as a benchmark for future developments in movie rating prediction, supporting the researchers and practitioners in augmenting recommendation system performance.</div></div>","PeriodicalId":100155,"journal":{"name":"BenchCouncil Transactions on Benchmarks, Standards and Evaluations","volume":"4 4","pages":"Article 100200"},"PeriodicalIF":0.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145871526","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0